Interesting lil problem

Hey /r/networking.

I am writing here to see if anyone has some insight or a path to investigate further into a strange little issue.

We have some compliance in our environment and because of this leverage Solarwinds UDT. This lil issue essentially makes the UDT unreliable because if it is not functioning as intended... and providing false alerts, it does not serve it's purpose.

The issue I am running into... about once per day, sometimes twice, I will get an alert that a Rogue MAC address has been detected. The MAC address is vendorless and appears to be generated randomly. There doesn't appear to be a timestamp correlation to when this occurs, it just happens when it damn well pleases.

Fortunately, using SW UDT, I can see what switch this has a direct connection to. It is not always the same path through the network, but often passes or direct connects to a specific port. I mirrored out the port (well, the port that is most commonly flagged, it's not consistent) to an unused one & ran a dumpcap/shark on the wire for the past 24 hours and it came back with three separate hits across the span of 24 hours.

Filtering out the results is interesting (well, to me anyways, I'm not a super network guru-type).

The packet that this exists in appears only once in the span of roughly 3 hours. It is always a single source:dest mac set and both of them are vendorless. A strange occurrence that has happened once (that I've noticed since capturing) is the mac address exists as "concurre_00:00:34"

The protocols I've seen listed associated with the packet are: "0x0c78" and "0x748c" (so far anyway).

Somewhere in here, the managed switch must be registering this in it's arp table, or I'm guessing it wouldn't be detected? When I've http'd into the switch to view the arp table, it does not exist there either. It blips and disappears, almost as if it's relying on this for some internal function that I am unaware of.

I have no doubt it is something in the environment that is doing this and I'd like to remove it so that the UDT solution can function as intended. As it stands now, it's unreliable because of this.

I'm stumped. Any seen something similar to this or have a better path? Glad to answer anything, but can't always respond immediately.

Hope everyone had a nice holiday & I greatly appreciate your time.