Troubleshooting intermittent timeouts between an F5 and our Hyper-V Infrastructure

Hi everyone,

I've been stuck with this topic for a while and not sure where to go next. If anyone can point me in a direction I would be very grateful. I have very little experience in this type of "deep dive" network troubleshooting.

F5 support has told me the issue is "between the F5 and the VM", which doesn't help much since the connection goes F5 <-> Switch <-> Hyper-V Clusters. The switch is managed by our datacenter provider while we own both the F5 and the Hyper-V Clusters.

What happens is that we will have intermittent timeouts of connections to the F5 that I have yet been unable to reproduce reliably. At random intervals, accessing one of our virtual servers will time out. Sometimes this happens 10 times in a row, most of the time it will work fine immediately after. My next step would have been to see if this issue occurs on infrastructure other than our clusters. They are both configured exactly the same so the issue could be existent on them both.

The traffic in Wireshark looks like this:

On the server (10.0.0.58):

No. Time Source Destination Protocol Length Info

2381 14:10:57.92 10.0.0.58 10.0.0.43 TCP 66 54236 → 25 [SYN, ECN, CWR] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

2744 14:11:00.92 10.0.0.58 10.0.0.43 TCP 66 [TCP Retransmission] 54236 → 25 [SYN, ECN, CWR] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

3394 14:11:06.93 10.0.0.58 10.0.0.43 TCP 66 [TCP Retransmission] 54236 → 25 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

On the F5 VS (10.0.0.43):

No. Time Source Destination Protocol Length Info

7736 14:10:57.93 10.0.0.58 10.0.0.43 TCP 66 54236 → 25 [SYN, ECN, CWR] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

7738 14:10:57.93 10.0.0.43 10.0.0.58 TCP 62 25 → 54236 [SYN, ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 SACK_PERM=1

8560 14:11:00.93 10.0.0.43 10.0.0.58 TCP 62 [TCP Retransmission] 25 → 54236 [SYN, ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 SACK_PERM=1

8562 14:11:00.93 10.0.0.58 10.0.0.43 TCP 66 [TCP Retransmission] 54236 → 25 [SYN, ECN, CWR] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

10983 14:11:06.93 10.0.0.43 10.0.0.58 TCP 62 [TCP Retransmission] 25 → 54236 [SYN, ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 SACK_PERM=1

10985 14:11:06.94 10.0.0.58 10.0.0.43 TCP 66 [TCP Retransmission] 54236 → 25 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

17059 14:11:18.93 10.0.0.43 10.0.0.58 TCP 62 [TCP Retransmission] 25 → 54236 [SYN, ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 SACK_PERM=1