Why do I have dancing MACs?

I've run into an interesting issue. Lately, there have been quite a few changes on our network. We migrated all our 6500s and 4500s to Cisco 9500 (cores) and Meraki MS390 (L3+edge). However, as of lately, we've noticed MAC addresses bouncing around ports on a switch. This was causing issues where we had port-security enabled. My question is where can I even start investigating?

 

We'll use this MAC as the example. (14b3.1f0d.0295). As you can see the first entry is for a MAC ending in 0f8f on port g0/18 disabling the port. After a few minutes (the last 3 groups of the log) show MAC 0295 coming in on 3 different ports (1 of them being port 18). I can confirm that nobody was logged in physically at the machines during these times or plugging/unplugging things in to the switch.

Sep 27 08:13:32: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/18, putting Gi0/18 in err-disable state Sep 27 08:13:33: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0f8f on port GigabitEthernet0/18. Sep 27 08:13:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to down Sep 27 08:13:35: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to down Sep 27 08:14:03: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Gi0/18 Sep 27 08:14:11: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to up Sep 27 08:14:12: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to up Sep 27 08:17:36: %LINK-3-UPDOWN: Interface GigabitEthernet0/13, changed state to down Sep 27 08:17:46: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/12, putting Gi0/12 in err-disable state Sep 27 08:17:46: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f10.7b57 on port GigabitEthernet0/12. Sep 27 08:17:47: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/12, changed state to down Sep 27 08:20:15: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/12, putting Gi0/12 in err-disable state Sep 27 08:20:15: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.34e5 on port GigabitEthernet0/12. Sep 27 08:20:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/12, changed state to down Sep 27 08:20:17: %LINK-3-UPDOWN: Interface GigabitEthernet0/12, changed state to down Sep 27 08:17:34: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/13, putting Gi0/13 in err-disable state Sep 27 08:17:34: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0295 on port GigabitEthernet0/13. Sep 27 08:17:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/13, changed state to down Sep 27 08:18:54: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/18, putting Gi0/18 in err-disable state Sep 27 08:18:54: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0295 on port GigabitEthernet0/18. Sep 27 08:18:55: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to down Sep 27 08:18:56: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to down Sep 27 08:19:24: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Gi0/18 Sep 27 08:19:33: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to up Sep 27 08:19:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to up Sep 27 08:19:42: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/9, putting Gi0/9 in err-disable state Sep 27 08:19:42: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0295 on port GigabitEthernet0/9. Sep 27 08:19:43: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to down Sep 27 08:19:44: %LINK-3-UPDOWN: Interface GigabitEthernet0/9, changed state to down Sep 27 08:20:12: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Gi0/9 

Mail Server Dead after Network Redesign

Hello, I am the network engineer at a small company and I just finished redesigning the network! Replacing switches and routers that were 20 years old. The only issue that remains is our mail server. We have a mail server running on freebsd that sits behind a barracuda spam firewall. Post redeseign, we are unable to recieve outside mail, but we can send it out. Anybody had a similar issue? Would really appreciate some insight.

Lacking IOS XR knowledge for this... Please help

Hi,

Our company just bough a pair of ASR9903's shipped with 7.1.3 and I tried upgrading them to 7.4.1 with some complications...

So here it goes...

These are these filesystems present:

RP/0/RP0/CPU0:ios#show filesystem
File Systems:

Size(b) Free(b) Type Flags Prefixes

3796250624 3788713984 harddisk rw harddiska:

4060278784 4050333696 flash-disk rw disk0:

20507914240 20159877120 flash-disk rw apphost:

23653642240 23584514048 harddisk rw harddisk:

0 0 network rw tftp:

0 0 network rw ftp:

1015304192 1012920320 flash rw /misc/config

3796250624 3788713984 harddisk rw harddiskb:

I tried upgrading according to the official procedure provided from Cisco and once i made the TAR-file, mounted it and began upgrading the router crashed and now is spamming me that /misc/config disk is full and needs urgent attention.

I go into that folder and see my .tar file there so i went ahead and tried to remove it:

RP/0/RP0/CPU0:ios#delete /misc/config/7.4.1-iso-and-rpms.tar

Mon Sep 27 13:38:04.494 GMT-2
delete : /misc/config/7.4.1-iso-and-rpms.tar : path outside filesystem disallowing access

​

I cannot remove the file that is clogging my /misc/config drive (and currently has 0bytes free)

After some troubleshooting i noticed i also cannot create folders on that specifik disk on the other (currently un-upgraded router) with the same error message)

What am I missing here? Why can't i modify any files hosted in /misc/config?
Is this possible to solve myself?

​

​

Help greatly appreciated.

OOB management port question

Hi,

Some access switches have a dedicated out-of-band management port. How does that port differ from using the serial console port or any other port on VLAN 1?

I'm looking at different switches and trying to find out what the advantages of a dedicated management port is.

Is it for instance possible to power off and on the switch from the management port? Similar to how out-of-band management works on a server.

Or is it a security issue - isolating data traffic from control traffic?

Are there any rule of thumb when you should have switches with a dedicated management port?

​

How does Miracast/WiFi-Direct work?

Sorry if this post is in the wrong sub and please remove it if it is. I was quite interested in Miracast/WiFi-Direct and making my own casting application on Linux. I have tried multiple applications and they all are quite bad frankly. Even windows has problems working with this. I was wondering how the technology works behind the scenes and whether there are any resources on it.

I use casting on a daily basis and having a reliable application to work with would be amazing. Building my own would help me learn how it works behind the scenes. I am familiar with Python and Rust and any resources in these languages would be much appreciated.

​

Thank you in advance

Excessive STP TCN flushes ACI from one single port bounce in classic L2

Hi!

Seen discussions in multiple communities on the internet regarding this issue.

We have serious issues when a single port bounces on a classic switch with RPVST connected to ACI via a vPC through Nexus 5K's.

Basically we have standard brownfield migration with one EBG pr classic VLAN and L3 enabled on the Bridge domain and HW proxy configured on L2 in the BD.

Classic L2 network has been a mess I inherited and portfast on ports were non existent. and I haven't found all the ports with misconfiguration and no spanning-tree portfast.

Once in a while such a port bounces and the result is that the bridge domain in ACI "dies" for a period of time, 60 seconds++. and we see a "storm" of Excessive STP TCN flushes in the logs. 4-6 warning logs pr leaf switch in the fabric for 60 seconds.

And god forbid I bounce a uplink trunk between switches in classic net with no VLAN acl, then basically all BD's dies.

This started to be a problem after we expanded ACI to a second DC using stretch fabric, and due to the chip shortage we had to wait 6 months for the fiber leaf's. hence we had to "gaffa teip" a Nexus 3K vPC pair with vPC L2 into ACI on this location to get enough 10 Gig ports.

After this, any TCN from classic net killed the BD in ACI for aprox 1 minute.

only difference is now we have 2 vPC's to classic net. There are no direct links between the classic net switches on the two DC's so no loop in classic net to blame it on.

Talked to other experienced Consultants about this issue and they say that the have never had problems due to TCN flushing in any of the other ACI fabrics they have set up for customers.

What should I be looking at to fix this issue? besides find ALL ports and configure portfast, Which is impossible on switch uplinks anyway :)

I'm starting to plan for a pure L3 link from classic net to ACI and kill STP with fire in ACI. But I am at a loss when I have to explain the current outages to C-level.

Microsoft Teams

Hi guys having an issue with Microsoft teams. Have two Internet breakouts, sitting in front of a Palo Alto Firewall. Not inspecting traffic, not going through proxy. we're using less than 50% of our bandwidth.

Yet we get intermittent breaks during Teams meetings. It's not long probably lasts between 5-10 seconds, but its got the executives in my office in a tizzy.

Same thing doesn't happen with webex or zoom, anyone faced anything like this before ?Or any advice on where to look?

Logical Link Control tuning.

Hello, Everybody. Please, tell me, how can I tune LLC on any switch, or it hardcoded into hardware? For example, I wan't to change type of service from LLC1 to LLC2. Is it possible?

Moronic Monday!

It's Monday, you've not yet had coffee and the week ahead is gonna suck. Let's open the floor for a weekly Stupid Questions Thread, so we can all ask those questions we're too embarrassed to ask!

Post your question - stupid or otherwise - here to get an answer. Anyone can post a question and the community as a whole is invited and encouraged to provide an answer. Serious answers are not expected.

Note: This post is created at 01:00 UTC. It may not be Monday where you are in the world, no need to comment on it.

phpIPAM scanning issue.

Hello everyone, I'm interning at a bank and my internship project revolves around implementing the phpIPAM solution in one of the banks branches. My issue is that as far as I know phpIPAM uses ICMP and SNMP protocols to scan for available IP adresses in subnets. But for security reasons , the firewall on some of tthe banks devices block ICMP and SNMP so theoretically if phpIPAM scans a device where the ping port is blocked it would return that the corresponding IP adress is free to use when it's actually not. Is there anyway of working around this issue??