Hi reddit,
I have been tackling a strange issue for quite some time and I am about to pull my hair out of my skull. I have a very robust MPLS network with 3 hospitals on it that has virtually no issues. Recently we have been asked to extend a network to a hosted application in a datacenter in Chicago.
AT&T is sunsetting MPLS networks and could only bring up a new AVPN link since it was out of state so I have both the MPLS and AVPN routers connected to my core switch.
The issue seems to only be traffic to and from the datacenter in Chicago on the AVPN and one of the hospitals here in Oklahoma on the MPLS.
AT&T has looked at each circuit muitple times and cannot find any trouble with either circuit.
The only device outside of my control is a sonic wall firewall at the vendor end but they claim since testing between our hub site and their end is fine that the issue must be on my end. However AT&T claims that it's at the CPE in chicago.
I've added a shaping policy to the 100MB interface to shape to 95% average of 100MB but that did not seem to make much of a difference.
Site A - 1GB Connection to AT&T MPLS Cloud (gig1/26)
Site A - 100MB Connection to Site C (gig1/25)
Site B – 1GB Connection to MPLS Cloud (gig2/1)
Site C – 100MB Connection to (Site A)
Speeds from Site A to Site B = Full Gig
Speeds from Site A to Site C = Full 100MB
Speeds from Site B to Site C = only getting about 25MB (single iperf stream)
Speeds from Site C to Site B = also only getting about 25MB
Pings between Site A to Site B = constant
Pings between Site A and Site C = Constant
Pings between Site B and Site C = dropping 1 in every 25-30 pings
Pings between Site C and Site B = also dropping 1 in every 25-30 pings
Links between the MPLS Sites and the AVPN site are under less than 10% load so it does not appear to be a bandwidth issue on either link.
GigabitEthernet1/25 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 2cd0.2db6.73a0 (bia 2cd0.2db6.73a0)
Description: AVPN CONNECTION
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 9/255, rxload 21/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is on, output flow-control is on
Auto-MDIX on (operational: on)
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:45:09
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 8377000 bits/sec, 2818 packets/sec
5 minute output rate 3814000 bits/sec, 2996 packets/sec
7060638 packets input, 2349533345 bytes, 0 no buffer
Received 562 broadcasts (5 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
7784770 packets output, 1257602605 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
OMC-4507-Svr-P#show interfaces gigabitEthernet 1/26
GigabitEthernet1/26 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 2cd0.2db6.73a1 (bia 2cd0.2db6.73a1)
Description: MPLS CONNECTION
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 12/255, rxload 7/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is off, output flow-control is off
Auto-MDIX on (operational: on)
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:45:15
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 27460000 bits/sec, 9636 packets/sec
5 minute output rate 49993000 bits/sec, 10869 packets/sec
26999845 packets input, 12034585352 bytes, 0 no buffer
Received 1688 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
28217241 packets output, 14397891842 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
----
Has anyone dealt with a similar issue or have any advice?
No comments:
Post a Comment