Why do I have dancing MACs?

I've run into an interesting issue. Lately, there have been quite a few changes on our network. We migrated all our 6500s and 4500s to Cisco 9500 (cores) and Meraki MS390 (L3+edge). However, as of lately, we've noticed MAC addresses bouncing around ports on a switch. This was causing issues where we had port-security enabled. My question is where can I even start investigating?

 

We'll use this MAC as the example. (14b3.1f0d.0295). As you can see the first entry is for a MAC ending in 0f8f on port g0/18 disabling the port. After a few minutes (the last 3 groups of the log) show MAC 0295 coming in on 3 different ports (1 of them being port 18). I can confirm that nobody was logged in physically at the machines during these times or plugging/unplugging things in to the switch.

Sep 27 08:13:32: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/18, putting Gi0/18 in err-disable state Sep 27 08:13:33: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0f8f on port GigabitEthernet0/18. Sep 27 08:13:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to down Sep 27 08:13:35: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to down Sep 27 08:14:03: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Gi0/18 Sep 27 08:14:11: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to up Sep 27 08:14:12: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to up Sep 27 08:17:36: %LINK-3-UPDOWN: Interface GigabitEthernet0/13, changed state to down Sep 27 08:17:46: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/12, putting Gi0/12 in err-disable state Sep 27 08:17:46: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f10.7b57 on port GigabitEthernet0/12. Sep 27 08:17:47: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/12, changed state to down Sep 27 08:20:15: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/12, putting Gi0/12 in err-disable state Sep 27 08:20:15: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.34e5 on port GigabitEthernet0/12. Sep 27 08:20:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/12, changed state to down Sep 27 08:20:17: %LINK-3-UPDOWN: Interface GigabitEthernet0/12, changed state to down Sep 27 08:17:34: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/13, putting Gi0/13 in err-disable state Sep 27 08:17:34: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0295 on port GigabitEthernet0/13. Sep 27 08:17:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/13, changed state to down Sep 27 08:18:54: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/18, putting Gi0/18 in err-disable state Sep 27 08:18:54: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0295 on port GigabitEthernet0/18. Sep 27 08:18:55: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to down Sep 27 08:18:56: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to down Sep 27 08:19:24: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Gi0/18 Sep 27 08:19:33: %LINK-3-UPDOWN: Interface GigabitEthernet0/18, changed state to up Sep 27 08:19:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/18, changed state to up Sep 27 08:19:42: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi0/9, putting Gi0/9 in err-disable state Sep 27 08:19:42: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 14b3.1f0d.0295 on port GigabitEthernet0/9. Sep 27 08:19:43: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/9, changed state to down Sep 27 08:19:44: %LINK-3-UPDOWN: Interface GigabitEthernet0/9, changed state to down Sep 27 08:20:12: %PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Gi0/9