[00:04:34] PROBLEM - SSH on amslvs2 is CRITICAL: Server answer: [00:05:19] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:06:13] RECOVERY - SSH on ssl3001 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:07:34] RECOVERY - SSH on amslvs2 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:11:55] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [00:21:03] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [00:21:22] PROBLEM - SSH on nescio is CRITICAL: Server answer: [00:27:04] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:29:46] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:31:25] PROBLEM - SSH on nescio is CRITICAL: Server answer: [00:32:18] PROBLEM - SSH on ssl3001 is CRITICAL: Server answer: [00:33:40] RECOVERY - SSH on ssl3001 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:36:57] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [00:38:00] PROBLEM - SSH on ssl3001 is CRITICAL: Server answer: [00:38:27] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:45:12] RECOVERY - SSH on ssl3001 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [01:32:28] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [01:41:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 246 seconds [01:42:49] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 266 seconds [01:49:16] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 657s [01:55:25] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [01:57:57] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [01:58:34] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 16 seconds [03:23:55] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [03:33:57] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [03:38:00] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [03:38:00] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [03:38:00] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [04:14:55] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [05:50:01] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [06:09:57] morning [06:12:58] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [06:25:40] moin [06:26:41] not yet fixed i think: 11 18:03:32 < jeremyb> spence is up to 86 hrs since last puppet run... [06:52:30] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [07:04:24] wtf, why does ganglia have """Wikimedia Grid > Miscellaneous pmtpa > en.wikipedia.org""" ? [07:04:42] and there's a 10.0.0.34 in there too [07:05:58] hrmmm, and virt0 seems to be missing entirely from ganglia [07:07:30] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [07:10:43] morning [07:13:01] hey [07:19:17] oh, i forgot to mention: amslvs2, ssl3001, and nescio (all ams) flapped more than a few times each last night. maybe worth a look [07:22:30] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [07:34:48] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [07:37:30] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [08:59:42] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.054 second response time [09:06:27] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [09:06:27] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [09:36:25] New patchset: ArielGlenn; "ms10 to public hostname" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19206 [09:37:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19206 [09:37:53] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19206 [10:01:37] New patchset: ArielGlenn; "add ms10 to list of media rsyncers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19208 [10:02:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19208 [10:02:18] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19208 [10:12:27] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [10:49:16] !log starting initial rsync from ms8 to ms10 (should take approximately forever) [10:49:27] Logged the message, Master [10:57:24] hm? [10:58:44] apergos: what's the plan there? [10:59:07] right now we rsync to eqiad [10:59:34] anyways the idea is to have two hosts, one in eqiad and one in tampa, with full copies of the original media [10:59:48] aren't we going to use swift instead of ms*? [10:59:52] one as primary and one in case the primary dies... used to feed media mirrors outside [10:59:55] (which we do already) [10:59:57] like this week? [11:00:08] yep [11:00:34] these hosts will have a flat copy exposed from rsyncing to the outside [11:01:03] they will get it by polling upload.wm.o (which will mostly be the squids I guess, but sometimes it will fall through to swift) for new/changed media [11:01:27] *for rsyncing to the outside [11:01:57] I have a script already in place for it, which I'll be testing once the switchover is done and seems stable-ish [11:02:00] polling how? [11:02:12] http gets [11:02:15] standard stuff [11:02:32] we'll have lists of changed/new media from, I guess I get that out of the db [11:02:38] idea is to run that once a day [11:03:05] much better than trying to ask swift for a list of everything in all its containers [11:03:15] also back end independent [11:03:53] anyways the silly thing right now is that we copy from tampa to equiad and then our mirrors get from there [11:05:06] it would have been a nice way to stress-test the backup fiber cable :p [11:05:13] no it woudln't :-D [11:05:22] hihi [11:05:41] in fact we do our daily copy over that wire, which I quietly let hang that day [11:05:53] given we were limping along on one link [11:08:04] !log apt: remove php-wikidiff2/1.1.0-2 from lucid-wikimedia/universe (lucid-wikimedia/main has 1.1.2-1) [11:08:13] Logged the message, Master [11:08:33] !log apt: remove wikidiff2/0.0.1wm1 (obsolete), copysrc php-wikidiff2 {lucid,precise}-wikimedia [11:08:42] Logged the message, Master [11:08:47] hmmm [11:58:57] !log apt: include php5 5.3.10-1ubuntu3.2+wmf1 @ precise-wikimedia [11:59:05] Logged the message, Master [12:04:06] !log apt: include php-memcached 2.0.1-6~wmf+lucid2 @ lucid-wikimedia [12:04:14] Logged the message, Master [12:11:37] apergos can't fill that one fiber anyway [12:11:48] no matter how much he tries to stress test with those server copies ;) [12:12:32] * apergos files that away as a challenge for later :-P [12:13:36] later? that's lame [12:13:42] in 10 years you'll fill it with ease [12:13:44] i'm talking NOW [12:22:15] later before the end of *this* year [12:24:09] !log Restored SnapMirror relationship between nas1-a and nas1001-a, 70 days lagged, transfer initiated [12:24:18] Logged the message, Master [12:27:02] just 70 days? I'm impressed [12:27:09] hehe [12:27:15] do you know anything about the "labs" aggregate? [12:27:21] I think ryan did that for testing once, not entirely sure [12:27:24] first time I'm hearing of this [12:27:25] it can't be in production use ;) [12:27:43] well, considering the netapps didn't work until what? yesterday? [12:27:47] no, it can't be in production :) [12:27:51] they've been up and down [12:28:06] until friday you mean ;) [12:28:15] yeah [12:28:53] btw, the ontap web interface is not completely useless [12:29:06] any reason to use it over the cli? [12:29:15] somtimes it's easier [12:29:18] sometimes it's not [12:29:33] I used to use both, depending on the task [12:30:54] snapmirror back in sync [12:31:11] how do I login to the nas? via root pwd? [12:31:13] not that we need it for anything [12:31:14] yes [12:32:38] ah of course, "too many users logged in" [12:32:44] hehe [12:32:50] didn't get fixed in 8 I see [12:34:53] I thought we were going to use snapmirror? [12:35:05] yeah, but not the existing test volume [12:35:06] or you mean it's useless now that it has no useful data? [12:35:11] aha [12:35:12] i just copied images onto it for testing [12:35:19] and I wanted to see if it could recover after all the failures ;) [12:35:21] it did [12:35:23] it's back in sync [12:35:41] :-) [12:36:05] nas{1,1001}-{a,b} [12:36:06] nice. [12:37:08] mark: did you see this? 13 07:19:17 < jeremyb> oh, i forgot to mention: amslvs2, ssl3001, and nescio (all ams) flapped more than a few times each last night. maybe worth a look [12:37:45] hmm, perhaps connectivity problems [12:38:25] oh, just ssh [12:38:31] some ssh login scanner i'm sure [12:38:36] hah [12:39:20] paravoid: what do you think about ontap cluster mode? [12:40:51] mark: also, i noticed (before now) that the ARIN /22 had new contacts but now I see that the ams /24 has old contacts. maybe want to update there too [12:41:44] mark: never used it, it's an 8+ feature [12:41:47] can we even use it? [12:41:58] if we restrict to non-block-level access, I think so [12:42:06] they used to have completely separate product lines, the OnTAP and OnTAP GX [12:42:09] still so [12:42:15] well, we can use either [12:42:19] but they're completely different OSs [12:42:22] and with OnTAP 8 I heard they merged the two [12:42:43] I think the GX needed separate hardware [12:42:44] at least, you have to install a different image and there's no conversion [12:43:16] * paravoid googles it [12:43:37] dammit mark, why can't you ask all the easy questions that I have the answers for. [12:45:40] ONTAP isn’t a Unified OS, it’s a brand. They could call StorageGrid or Engenio ONTAP and it’ll be about as Unified as Cluster-Mode and 7-Mode. There is no unification here beyond the block diagram they’ll draw on your white board and that diagram is not connected to reality but to brand marketing. [12:45:45] *sigh* [12:45:56] i guess we'll just use 7-mode [12:47:37] !log Setup NTP on all NetApps [12:47:46] Logged the message, Master [12:59:54] !log Doing NDU of IOM3 shelf firmware on nas1-a [13:00:03] Logged the message, Master [13:05:18] !log Doing NDU of IOM3 shelf firmware on nas1001-a [13:05:26] Logged the message, Master [13:08:52] hi w [13:08:57] arrived ok? [13:10:57] paravoid: so the new ontap version we run now does support ipv6 quite well it seems [13:11:07] yay [13:11:11] 8.0 didn't afaik [13:11:20] indeed [13:15:23] i hate the current network setup of the netapps, but they don't support anything fancy [13:15:33] i'm considering running a RIP instance on our routers, just for the netapps :-( [13:16:58] nah, looks like that also can't do what I want [13:17:29] what do you want? [13:17:47] not both controllers down whenever we upgrade row B access switches [13:18:18] i'd want to connect each controller to a different row stack, but then they can't take over eachother's ips anymore [13:18:23] since they're different subnets [13:19:05] I lack information about our network topology... [13:19:10] do we have that documented anywhere? [13:19:17] not really [13:19:21] basically, the routers just route [13:19:32] every eqiad row has its own subnets [13:19:42] and every row consists of an ex4200 stack [13:19:51] so no l2 across rows atm [13:20:05] every stack connects to the router directly? [13:20:06] mostly to avoid sTP [13:20:08] yes [13:20:28] if the netapps spoke OSPF or so, i'd consider hooking them up to the routers directly [13:20:29] local cross-connect? :) [13:20:31] ew? :) [13:20:37] how's that ew [13:20:44] that's very elegant [13:20:59] you probably didn't understand what I proposed [13:21:11] no [13:21:12] local switching on the mx [13:21:20] don't want local switching on the mx [13:21:26] because "ew" [13:21:34] yeah mostly [13:21:43] yeah, that's what I figured [13:21:47] if we would do it, then for a special netapp only subnet [13:21:53] I said it myself to avoid making it sound like a real proposal :P [13:21:57] hehe [13:22:31] we COULD use vlans on the netapp [13:22:40] and connect it to both row A and row B vlans or something [13:22:56] how would we do failover then? [13:23:03] how would we access it from the hosts that is? [13:23:03] nah still sucks [13:23:13] via the row A VLAN or the row B VLAN? [13:23:20] or you mean do RIP over the two VLANs? [13:23:26] yeah [13:23:31] but not sure it really does that [13:23:34] it's not very configurable [13:23:40] i believe it has rip just to get routes from the network [13:23:49] even if it does, I'd expect it to be so uncommon to be completely buggy [13:23:53] yeah [13:24:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [13:24:31] yay for ethernet topologies [13:25:07] good morning you two! (nycer here) you sound busy, but I could use some help troubleshooting boot pxe for the analytics dells if you've got a sec sometime [13:25:18] we could hook them up to the routers directly and only bridge those ports between [13:25:22] but it would eat up 4 MX ports [13:25:33] are these 10g? [13:25:39] the MX ports yes [13:25:44] so we'd have to buy 10G cards for the netapps too [13:25:45] the netapps? [13:25:48] I believe it doesn't have them yet [13:25:51] only for internal use [13:25:54] (afaik) [13:25:57] internal use? [13:26:02] it says this: [13:26:02] slot 0: Dual 10G Ethernet Controller T320E-SFP/KR [13:26:02] c0a MAC Address: 00:a0:98:14:6f:ac (auto-unknown-cfg_down) [13:26:03] c0b MAC Address: 00:a0:98:14:6f:ad (auto-10g_kr-fd-up) [13:26:08] but I believe that's internal to the chassis [13:26:13] I don't think so [13:26:13] for between the two controllers or so? dunno [13:26:27] the internal link between the two controllers is infiniband iirc [13:26:36] well it's worth investigating [13:26:53] ottomata: so what's the status? [13:27:50] paravoid: it has private IPs bound to those controllers [13:27:54] I think it's using them for something [13:28:02] they're UP [13:28:24] one of the two is UP, both have private IPs bound [13:29:03] mark, the status is [13:29:07] as far as I know [13:29:32] i think last i heard he booted to the installer fine (via pxe) and then the installer itself was unable to get a lease. and tcpdump on brewster showed no request showing up [13:29:43] they are able to reach brewster/dhcp during initial boot, but once they start to go through installer configs (at the auto configure networking step) they don't seem to be able to reach them anymore [13:29:54] i'm not really sure how this step works, and notpeter and I are a little stumped [13:29:55] ok [13:29:56] also installer's shell said a different MAC than the BIOS? [13:29:59] which machine should I try? [13:30:04] oh it did? [13:30:08] analytics1011 [13:30:14] ottomata: i think so [13:30:34] what kind of machines are these? [13:30:34] slot 0: Dual 10G Ethernet Controller T320E-SFP/KR [13:30:34] c0a MAC Address: 00:a0:98:14:6f:ac (auto-unknown-cfg_down) [13:30:37] whoops [13:30:44] mark@fenari:~$ ssh root@analytics1011.mgmt [13:30:44] ssh: connect to host analytics1011.mgmt port 22: Connection refused [13:32:27] hello? [13:32:32] nakr [13:32:33] mark [13:32:34] http://wikitech.wikimedia.org/view/Dell_PowerEdge_C2100 [13:32:39] sorry, was finding link [13:32:41] C2100s? ew [13:34:04] 04:7d:7b:a5:e1:b2 [13:34:15] is what I got from BIOS [13:34:30] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [13:34:37] 2: eth0: mtu 1500 qdisc mq state DOWN qlen 1000 [13:34:38] link/ether 90:e2:ba:11:7f:40 brd ff:ff:ff:ff:ff:ff [13:34:46] clearly what linux thinks is eth0 is not connected [13:35:08] e1:b2 is not in the lot [13:36:29] hm [13:36:42] well that would be why it wouldn't work then... [13:36:45] is that nic1 or nic2? [13:37:09] all other macs are higher and incement by one [13:37:13] seems like it should be the first nic [13:37:44] weird, i'm going to bios boot 1012 and see what the MAC is there, to make sure i have it right [13:37:47] 1012 behaved the same way [13:38:17] open a ticket for rob in the eqiad queue to check on that [13:38:32] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [13:38:32] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [13:38:32] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [13:38:55] to check to see why MACs are reporting wrong (if they are?) [13:39:06] to check what the hell is up with the connections on those boxes [13:39:09] he can figure it out [13:39:23] ok [13:46:49] well well [13:46:54] there you go ottomata [13:47:33] paravoid: meh [13:47:52] perhaps i'll make one vlan which we bridge between the two MXes [13:48:08] hm [13:48:19] that goes through the EXes? [13:48:24] i'd prefer not [13:48:31] this has the potential of looping at some point in the future [13:48:35] i'd prefer to hookup the netapp controllers directly [13:48:43] would cost 2 MX ports [13:49:01] and it might save us from a loop [13:49:06] yeah [13:49:11] so, I agree with you [13:49:13] direct is better [13:49:16] just probably need to buy 10G NICs for the netapps [13:49:46] oh, I have an idea what those 10Gs might be [13:49:51] how do we do inter-site replication? [13:50:03] just over the normal network interfaces [13:50:23] it's LACP over two GigEs per controller [13:50:55] https://communities.netapp.com/thread/12200 [13:51:08] "So those 2 ports (c0a, c0b) would be for the cluster interconnect only. Makes sense for this to no longer be infiniband as I imagine its easier for manufacturing, and the bandwidth is suitable." [13:51:15] so, the IB that I knew got replaced by those [13:51:39] yeah [13:53:32] you can add up to 4x dual 10GbE adapters apparently [13:54:00] the filers are usually normal x86 boxes and they get PCIs [13:54:05] yes [13:54:12] at least the ones I've seen, clearly some things have changed :) [13:54:18] still so on ours [13:55:52] I wonder how much overpriced will those cards be :P [13:56:10] probably very [13:57:14] i wonder if you could just put some generic ones in ;) [13:57:29] moorning RobH! [13:57:32] probably not [13:57:34] it needs a driver of course [13:57:37] heya [13:57:52] could you help me figure out some issues with the auto install of the new dells? [13:57:55] we still haven't figured it out [13:58:15] it looks like maybe bios is reporting different MAC addresses (not sure about that, could be my fault, trying to figure that out now) [14:01:25] ottomata: what's the problem? [14:01:38] i just looked at it [14:01:48] all ethernet NICs on the system report disconnected [14:02:05] ah, this was to RobH [14:02:16] yes [14:03:11] oh they report disconnected? [14:03:30] sorry, i missed that when you said it before [14:03:33] crazy, ok [14:03:40] no.. [14:03:44] eth4 is connected [14:03:50] 6: eth4: mtu 1500 qdisc mq state UP qlen 1000 [14:03:51] link/ether 04:7d:7b:a5:e6:94 brd ff:ff:ff:ff:ff:ff [14:03:51] inet6 2620:0:861:106:67d:7bff:fea5:e694/64 scope global dynamic [14:03:51] valid_lft 2591999sec preferred_lft 604799sec [14:03:51] inet6 fe80::67d:7bff:fea5:e694/64 scope link [14:03:51] valid_lft forever preferred_lft forever [14:03:54] so that's the problem [14:04:16] hm [14:04:20] perhaps rob can see what available NICs there are [14:04:23] and probably use the other one [14:04:27] who bought these servers? [14:04:32] I don't recall signing off on these [14:04:38] so I just checked one of the macs I had on 1012 [14:04:40] and it was off [14:04:58] I may have pasted in the wrong place, I have 1012's mac on 1013 in linux-host-entries [14:05:13] so this might be my fault [14:05:26] i'm going to change 1012 and try to boot and see if it works [14:06:51] New patchset: Ottomata; "Fixing MAC address on analytics1012." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19214 [14:07:11] mark or paravoid, could one of you approve that? I can merge and apply [14:07:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19214 [14:07:32] I will [14:07:41] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19214 [14:07:43] too late :P [14:08:04] nope [14:08:09] I'm in sockpuppet :P [14:08:35] paravoid, are you merging on sockpuppet? [14:08:42] apergos: I merged your ms10 hosts.allow too [14:10:04] crap did I no do that? mybad [14:10:34] thanks [14:15:27] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [14:25:10] ok, mark, RobH, I just double checked on analytics1011 [14:25:15] bios reports e1:b2 as NIC1 [14:25:22] which is what I have in puppet [14:25:27] but linux doesn't [14:25:35] so as long as you can boot from what linux thinks is eth0 [14:25:38] that's probably easier to use [14:25:41] PXE boot i mean [14:25:59] or just disable the secondary nic in bios [14:26:08] so weird, why would they see different MACs? [14:26:13] its odd that ubuntu is detecting nic2 for nic1 [14:26:18] isn't the secondary NIC the mgmt interface [14:26:22] it used to happen on older super micro systems [14:26:22] naw, as far as I can tell, neither are correct [14:26:24] no [14:26:26] NIC2 is e1:b3 [14:26:34] the mgmt interface is not either of the primary nics [14:26:36] mark saw: [14:26:36] link/ether 04:7d:7b:a5:e6:94 brd [14:26:40] ah ok [14:27:00] not nic2 [14:27:04] nic5 [14:27:08] eth4 [14:27:09] nic5! [14:27:11] aye [14:27:11] ok [14:27:17] uhh, and those aren't in bios…ok [14:27:20] at least I don't see them [14:27:26] there are only two nics in the c2100 [14:27:32] ok, how do I find the proper MACs on each of these dells? [14:27:37] in the bios [14:27:45] for analytics1011 [14:27:50] the bios shows [14:27:53] * NIC1 Mac Address [04-7D-7B-A5-E1-B2] [14:27:54] * NIC2 Mac Address [04-7D-7B-A5-E1-B3] [14:27:59] ok [14:28:23] mark says he saw eth4 link/ether 04:7d:7b:a5:e6:94 brd in linux [14:28:29]