Good afternoon everyone.I really hope the subject got your attention and I want to thank you in advance for your time. It will be a lengthy one ...
The reason of my post is that during an attempted migration my colleagues tried to perform (part of which I was not unfortunately and the information that I have is rather scarce) a network loop was observed for a period of at least 85 minutes. All the routing relies on OSPF and all the devices participating are running in area 0.0.0.0.
To start with (and in order to explain some weird behavior later on) I am going to share with you a simplified physical and L2 topology of the network
The devices we need to focus on are in the middle - all the Router and SW ones. Considering this physical/L2 topology, we have in reality the following routing domain:
Some highlights about the network:
- On the physical/logical diagram, consider that devices Router1, SW1, SW2 and Router 2 (put an imaginary line in the middle of both diagrams basically) are in the DC1. Right hand side is DC2.
- Inter-DC Layer2 is stretched between Switches SW1-4 as show:
- vPC to the Spine/Leaf (FabricPath) network
- OTV between FP in DC1 and DC2
- Intra-DC Layer2 (SW1<->SW2 & SW3<->SW4) goes through the FabricPath (which is where the STP root is)
- Router1,2,3 and 4 have some sort of a full mesh between each other, having Layer3 links (as shown on the diagram with green lines)
- Inter-DC Layer2 is stretched between Switches SW1-4 as show:
- All switches (SW1-4) form OSPF adjacencies on numerous VLANs (50+ - It's not ideal, I know...but it is what it is)
- Switches to Routers OSPF, albeit being dedicated L3 links, also run in BROADCAST network type (also not ideal... but bear with me).
- HSRPv1 configured between all SW1 devices. SW1 and SW2 are Active/Standby. SW3/SW4 are just listening and waiting to jump in if needed...
- As being part of area 0.0.0.0, routing for all the VLAN subnets is consider to be part of the OSPF area 0.
About the problem now:
Consider a network 10.0.0.0/24 running on VLAN203. All the Switches, being members of this VLAN, include this network in their OSPF router LSAs. SW2 is considered to be a DR for the segment and as such it generates the network LSA.
After lots of network disruptions (layer1, Layer2 and Layer3 ones) OSPF adjacencies between all devices flapped numerous of times. At some point the network settled in a rather weird condition. Here is the interesting part...
I was lucky to get an output from the sh ip ospf interface from during the problem and what I seen there was.... Interesting, to say at least..
- SW1 had OSPF adjacencies with SW3 and SW4, considering itself as a DR.
- SW2 had OSPF adjacencies with SW3 and SW4, considering itself as a DR.
- SW1 was unable to see OSPF hellos from SW2 (this was due to a vPC failure - but I won't bother you with all this. For now just consider that SW1 to SW2 communication was blocked). So now we have a very weird scenario where all the devices are member of the same VLAN, but two of them are unable to talk to each other. And to make things even worse, they became DRs for this Transit Network.
Unfortunately I don't have much more information than this, but what I saw in the logs (I am dying to be able to take a peak at OSPF database from this moment, but that's not available :( ) was as follow:
On Router1 show ip route was saying that for network 10.0.0.0/24 it must go to Router2 and the information there was from SW2:
Router1 Router1#sh ip route 10.0.0.100 Routing entry for 10.0.0.0/24 Known via "ospf 1", distance 110, metric 12, type intra area Last update from 192.168.0.22 on TenGigabitEthernet11/2, 00:16:13 ago Routing Descriptor Blocks: * 192.168.0.22, from 2.2.2.2, 00:16:13 ago, via TenGigabitEthernet11/2 Route metric is 12, traffic share count is 1
On Router2 however, we've got this:
Router2
Router2#sh ip route 10.0.0.100 Routing entry for 10.0.0.0/24 Known via "ospf 1", distance 110, metric 12, type intra area Last update from 192.168.0.21 on TenGigabitEthernet11/2, 00:16:13 ago Routing Descriptor Blocks: * 192.168.0.21, from 1.1.1.1, 00:17:45 ago, via TenGigabitEthernet11/2 Route metric is 12, traffic share count is 1
And the end result as you may have already imagined was this:
Traceroute from Router1 to 10.0.0.100 Router1#sh ip route 10.0.0.100 Type escape sequence to abort. Tracing the route to 10.0.0.100 1 Router2 (192.168.0.22) 4 msec 4 msec 8 msec 2 Router1(192.168.0.21) 4 msec 4 msec 0 msec
This just blows my mind... Considering the fact that adjacencies between SW1 <-> Router1 and SW2<->Router2 never experienced any Flaps or connectivity issues, I just cannot imagine a situation where this would happen.
I tried to simulate all this in a lab, but I only managed to "break it" so only Router1 will think that it has to send the traffic to Router2. But I never managed to break it that much so Router2 to send it back to Router1.
Now there must have been some DB discrepancies, considering the fact we have two routers in the same segment acting as DRs. It means they'd both generate Network LSAs for this Transit network and as such this information will be provided to all the OSPF adjacent devices.. But if SW1 says "I am DR and I have adjacencies with 3&4" - exactly the same thing will happen for SW2. But for what in the world Router1 (having direct connectivity to SW1) will prefer to send traffic for network residing on it's directly connected neighbour (SW1) to Router2 instead?! Makes no sense to me.. When I simulated this in a lab, I managed to break adjacencies (by simply blocking OSPF messages between SW1 and SW2) I witnessed a problem when SW3 was elected as DR by SW1. SW1 "said" to Router1 'here is my Router LSA. For this transit network use Network LSA 3.3.3.3 - which was not existing as SW3 wasn't a DR for the segment and didn't generate any Type 2s!). All that is so weird..
So .... That's the long story "short". I am happy to provide a bit more details (although like I mentioned they're not much) but I think the above should be enough to form a constructive discussion.
Hope the case is interesting enough to tease your brain (especially those of you who're now in their peak of knowledge, paving their way through, obtaining that #number!!). I remember a decade ago reading OSPF RFCs for "fun" and actually understanding and enjoying it! (dаmn!). Now as you can imagine most of this is gone, but not all ;-)
Thanks very much in advance for reading all this and I really hope it makes (some!) sense...
P.s. Same posted on Cisco's forum:
No comments:
Post a Comment