Hi everyone, I ran into an issue last night that has me questioning everything I think I know about packet forwarding.
I'm starting to wonder if there's something about forwarding UDP packets I don't know...
Network diagram (simplified for ease of discussion):
For background, this is for an FM radio transmitter sitting on top of a mountain.
I have a core switch at headquarters (Cisco Cat 4510R+E with Sup8e supervisor cards). Connected to it is an audio sending server. It sends high quality UDP audio packets to an audio receiving server connected to a remote switch (Cisco Cat 3650, running ipbase) sitting on top of a mountain. That receiving server feeds audio to an FM transmitter so you can have your drive-time FM entertainment while you're stuck in Los Angeles traffic.
The audio sending server at headquarters is on VLAN 1, and VLAN 1 is extended over a T1, where the remote switch is connected. So on VLAN 1, the server is 10.1.1.100, the core switch is 10.1.1.1, and the remote switch is 10.1.1.254. The server is set to use 10.1.1.1 as its gateway. The audio receiving server is at 10.200.1.100 (VLAN 200 off the remote switch).
Up until now, a simple static route has been in place on the core switch:
ip route 10.200.1.0 255.255.255.0 10.1.1.254
So last night, we fired up a new 100 meg layer 2 microwave link from headquarters to the mountain top. I also configured OSPF between the core switch and the remote switch. Removed the static routes, and made sure the RIB updated.
show ip route
on both switches showed me exactly what I wanted to see. The new microwave link is now the path on both sides.
Now, when I checked the port statistics, I was still seeing a bunch of packets going over the T1, and not as many as I would expect to be going over the microwave. I expected to see some traffic over the T1 because we use LAN extenders, so regular broadcast (ARP, DHCP requests, etc) traffic.
I did a remote packet capture on the remote switch - the port connected to the LAN extender. To my surprise, UDP audio traffic from the sending server to the receiving server was still taking the T1!
But TCP traffic (http, etc) was properly taking the microwave link (as verified by another packet capture).
In order to get UDP to take the correct path, we ended up rebooting both audio servers.
For the life of me, I can't figure out why updating the RIB, by enabling OSPF and removing the static route, didn't affect the path the UDP packets were taking.
Is there something different about UDP traffic vs. TCP traffic as related to forwarding?
Maybe some sort of forwarding cache that needed to be cleared that I 'm not aware of? Something deep in the TCP/IP stack I'm unaware of?
One other clue... A couple months ago, we the same issue with an audio receiving server at another transmitter site that's connected by a Meraki MX. In that case, the audio receiving server is getting UDP packets from the same audio sending server at headquarters, but instead of a LAN extender or microwave, those packets go through an ASA, into a Meraki MX, over the Internet, to a Meraki at the transmitter site. The Meraki at the transmitter site has a local Internet provider and a cellular modem as a backup.
When we tried to move that remote Meraki to use the cellular modem, the UDP packets from the audio sending server continued to flow over the local internet provider - Until we rebooted the audio sending server. At the time we thought it was just a Meraki bug.
Now I don't know what to think.
Anybody got any good insight?
Thanks!
No comments:
Post a Comment