So I have some NX-OS route reflectors and I am having some issues with path advertisements to both eBGP and iBGP peers to these RRs. This is an EVPN fabric, but I don't believe the issue is constrained to the l2vpn evpn SAFI.
The long and short of it is that I have two pods, pod1 (AS 65145) and pod2 (AS 65146). In pod1, my spines are RRs but do not participate in EVPN otherwise. My leaves in pod1 and my RRs in pod2 are only receiving one path for this externally originated route.
For instance, here is the route object and paths for pod1 RRs:
pod1-spine01# sh bgp l2vpn evpn rd 65000:10001 0.0.0.0 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 65000:10001 BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 32342030 Paths: (4 available, best #1) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Multipath: eBGP iBGP Advertised path-id 1 Path type: internal, path is valid, is best path AS-Path: 65000 , path sourced external to AS 172.24.251.3 (metric 2) from 172.24.254.3 (172.24.254.3) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Path type: internal, path is valid, not best reason: Router Id, multipath AS-Path: 65000 , path sourced external to AS 172.24.251.4 (metric 2) from 172.24.254.4 (172.24.254.4) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a6.cabd.b1d7 Path type: external, path is valid, not best reason: AS Path AS-Path: 65146 65000 , path sourced external to AS 172.24.251.101 (metric 2) from 172.24.254.101 (172.24.254.101) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:bc26.c729.dce7 Path type: external, path is valid, not best reason: Router Id AS-Path: 65146 65000 , path sourced external to AS 172.24.251.102 (metric 2) from 172.24.254.102 (172.24.254.102) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:bc26.c729.e8c7 Path-id 1 advertised to peers: 172.24.254.4 172.24.254.5 172.24.254.6 172.24.254.7 172.24.254.8 172.24.254.9 172.24.254.10 172.24.254.11 172.24.254.12 172.24.254.13 172.24.254.14 172.24.254.15 172.24.254.16 172.24.254.17 172.24.254.18 172.24.254.19 172.24.254.20 172.24.254.21 172.24.254.22 172.24.254.23 172.24.254.24 172.24.254.25 172.24.254.26 172.24.254.27 172.24.254.28 172.24.254.29 172.24.254.30 172.24.254.31 172.24.254.32 172.24.254.35 172.24.254.36 172.24.254.37 172.24.254.38 172.24.254.39 172.24.254.40 172.24.254.101 172.24.254.102 172.24.254.201
You can see that it has all of the paths.
If we look at a leaf in pod1, here's what we see:
pod1-leaf01a# sh bgp l2vpn evpn rd 65000:10001 0.0.0.0 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 65000:10001 BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 11477502 Paths: (3 available, best #1) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Multipath: iBGP Advertised path-id 1 Path type: internal, path is valid, is best path Imported to 2 destination(s) AS-Path: 65000 , path sourced external to AS 172.24.251.3 (metric 3) from 172.24.254.1 (172.24.254.1) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Received path-id 1 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Originator: 172.24.254.3 Cluster list: 172.24.254.1 Path type: internal, path is valid, not best reason: Neighbor Address AS-Path: 65000 , path sourced external to AS 172.24.251.3 (metric 3) from 172.24.254.2 (172.24.254.2) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Originator: 172.24.254.3 Cluster list: 172.24.254.2 Path-id 1 not advertised to any peer Route Distinguisher: 65000:10001 (L3VNI 10001) BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 11477661 Paths: (1 available, best #1) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Multipath: iBGP Advertised path-id 1 Path type: internal, path is valid, is best path Imported from 65000:10001:[5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224 AS-Path: 65000 , path sourced external to AS 172.24.251.3 (metric 3) from 172.24.254.1 (172.24.254.1) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Received path-id 1 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Originator: 172.24.254.3 Cluster list: 172.24.254.1 Path-id 1 not advertised to any peer
Now, logic says that "maximum-paths" and "additional-paths" are probably the cause of this...but...practice says it's not:
pod1-spine01# sh run bgp version 7.0(3)I7(3) feature bgp router bgp 65145 router-id 172.24.254.1 address-family l2vpn evpn maximum-paths 2 maximum-paths ibgp 2 additional-paths send additional-paths receive neighbor 172.24.254.101 remote-as 65146 description Pod2 Route Reflector password 3 ! update-source loopback0 ebgp-multihop 10 address-family l2vpn evpn send-community send-community extended route-map rmUnderlay-EBGP-Nexthop-Unchanged out neighbor 172.24.254.102 remote-as 65146 description Pod2 Route Reflector password 3 ! update-source loopback0 ebgp-multihop 5 address-family l2vpn evpn send-community send-community extended route-map rmUnderlay-EBGP-Nexthop-Unchanged out neighbor 172.24.254.0/23 remote-as 65145 description Pod1 leaves password 3 ! update-source loopback0 address-family l2vpn evpn send-community send-community extended route-reflector-client
One other strange issue is that leaves in Pod2 is actually preferring pod1's route for this destination:
pod2-leaf01a# sh bgp l2vpn evpn rd 65000:10001 0.0.0.0 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 65000:10001 BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 384 Paths: (2 available, best #2) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Path type: internal, path is valid, not best reason: Router Id AS-Path: 65145 65000 , path sourced external to AS 172.24.251.3 (metric 4) from 172.24.254.102 (172.24.254.102) Origin IGP, MED not set, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Advertised path-id 1 Path type: internal, path is valid, is best path Imported to 2 destination(s) AS-Path: 65145 65000 , path sourced external to AS 172.24.251.3 (metric 4) from 172.24.254.101 (172.24.254.101) Origin IGP, MED not set, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Path-id 1 not advertised to any peer Route Distinguisher: 65000:10001 (L3VNI 10001) BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 385 Paths: (1 available, best #1) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Advertised path-id 1 Path type: internal, path is valid, is best path Imported from 65000:10001:[5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224 AS-Path: 65145 65000 , path sourced external to AS 172.24.251.3 (metric 4) from 172.24.254.101 (172.24.254.101) Origin IGP, MED not set, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Path-id 1 not advertised to any peer
This makes no sense to me, based on the BGP table of the two RRs in Pod2:
256111-border01a# sh bgp l2vpn evpn rd 65000:10001 0.0.0.0 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 65000:10001 BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 2773382 Paths: (3 available, best #2) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Path type: external, path is valid, not best reason: newer EBGP path AS-Path: 65145 65000 , path sourced external to AS 172.24.251.3 (metric 3) from 172.24.254.1 (172.24.254.1) Origin IGP, MED not set, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Advertised path-id 1 Path type: external, path is valid, is best path Imported to 2 destination(s) AS-Path: 65145 65000 , path sourced external to AS 172.24.251.3 (metric 3) from 172.24.254.2 (172.24.254.2) Origin IGP, MED not set, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Path-id 1 advertised to peers: 172.24.254.103 172.24.254.104 172.24.254.105 172.24.254.201 Route Distinguisher: 65000:10001 (L3VNI 10001) BGP routing table entry for [5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224, version 1948351 Paths: (2 available, best #2) Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn Path type: external, path is valid, not best reason: Locally originated Imported from 65000:10001:[5]:[0]:[0]:[0]:[0.0.0.0]:[0.0.0.0]/224 AS-Path: 65145 65000 , path sourced external to AS 172.24.251.3 (metric 3) from 172.24.254.2 (172.24.254.2) Origin IGP, MED not set, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:00a2.ee28.8411 Advertised path-id 1 Path type: local, path is valid, is best path AS-Path: 65000 , path sourced external to AS 172.24.251.101 (metric 0) from 0.0.0.0 (172.24.254.101) Origin IGP, MED 0, localpref 100, weight 0 Received label 10001 Extcommunity: RT:65000:10001 ENCAP:8 Router MAC:bc26.c729.dce7 Path-id 1 advertised to peers: 172.24.254.1 172.24.254.2 172.24.254.103 172.24.254.104 172.24.254.105 172.24.254.201
This looks the way I'd expect it to, as the RR in this case is also the EVPN border leaf, but you can see here that this RR is not receiving both paths from the other RRs.
Relevant configuration of BGP on the two sets of border nodes is the same:
pod1-border01a# sh run bgp router bgp 65145 router-id 172.24.254.3 log-neighbor-changes address-family l2vpn evpn advertise-pip neighbor 172.24.254.1 remote-as 65145 description Pod1 Route Reflector password 3 ! update-source loopback0 address-family l2vpn evpn send-community send-community extended neighbor 172.24.254.2 remote-as 65145 description Pod1 Route Reflector password 3 ! update-source loopback0 address-family l2vpn evpn send-community send-community extended vrf TENANT1 address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map rmFABRIC-REDIST-TENANT1-INTERFACES redistribute static route-map rmTENANT1-STATIC-to-BGP redistribute hmm route-map rmALL-TYPE-2-ROUTES maximum-paths 2 maximum-paths ibgp 2 neighbor 10.1.1.2 remote-as 65000 description Northbound Router Tenant1 VRF password 3 ! address-family ipv4 unicast send-community route-map rmEBGP-TAG-TYPE-2-ROUTES out soft-reconfiguration inbound neighbor 10.1.1.4 remote-as 65000 description Northbound Router Tenant1 VRF password 3 ! address-family ipv4 unicast send-community route-map rmEBGP-TAG-TYPE-2-ROUTES out soft-reconfiguration inbound
And for pod2 border:
256111-border01a# sh run bgp feature bgp router bgp 65146 router-id 172.24.254.101 address-family l2vpn evpn advertise-pip additional-paths send additional-paths receive neighbor 172.24.254.1 remote-as 65145 description Pod1 Route Reflector password 3 ! update-source loopback0 ebgp-multihop 10 address-family l2vpn evpn send-community send-community extended route-map rmUnderlay-EBGP-Nexthop-Unchanged out neighbor 172.24.254.2 remote-as 65145 description Pod1 Route Reflector password 3 ! update-source loopback0 ebgp-multihop 5 address-family l2vpn evpn send-community send-community extended route-map rmUnderlay-EBGP-Nexthop-Unchanged out neighbor 172.24.254.0/23 remote-as 65146 description Pod2 Datacenter Speakers password 3 ! update-source loopback0 address-family l2vpn evpn send-community send-community extended route-reflector-client vrf TENANT1 address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map rmFABRIC-REDIST-TENANT1-INTERFACES redistribute hmm route-map rmALL-TYPE-2-ROUTES maximum-paths 2 maximum-paths ibgp 2 neighbor 10.1.1.6 remote-as 65000 description Northbound Router Tenant1 VRF password 3 ! address-family ipv4 unicast send-community route-map rmEBGP-TAG-TYPE-2-ROUTES out soft-reconfiguration inbound neighbor 10.1.1.8 remote-as 65000 description Northbound Router Tenant1 VRF password 3 ! address-family ipv4 unicast send-community route-map rmEBGP-TAG-TYPE-2-ROUTES out soft-reconfiguration inbound
I don't believe that this is directly an EVPN-related problem. I believe it's something to do with route-reflector loop prevention of some kind, but I really have no idea what it could be at this point, since the low hanging fruit ("maximum-paths" and "additional-paths") don't seem to have an effect, and there really aren't many more knobs to turn. I do have a TAC case open, but so far they've been less than helpful.
Thanks in advance.
No comments:
Post a Comment