TCP Offload Woes

Hi everyone,

I'm running into an issue with how to handle TCP offload within a "modern" infrastructure design that I'm working with.

The goal was to come up with a flexible network infra to run latency sensitive bare metal services with the potential to integrate containerized applications at some point. We ended up going with a spine-leaf design using eBGP between all spine/leaf/servers, with the servers peering w/ 2x spine switches. The servers would be reached via a /32 IP address assigned to a dummy interface which would be advertised into the fabric. If one of the switches would fail, the BGP connection dies right away since the interface on the server appears down and traffic picks up routing over the redundant link. We were looking at a zero trust network model with the majority of the firewalling done on the edge servers themselves.

Enter poor communication. The plan right now is to use Mellanox cards w/ TCP offload for all application traffic using LD_PRELOAD. This poses a problem for our design as the dummy interface is a virtual interface created and handled entirely by the kernel and as far as I understand cannot have it's traffic offloaded to a network card. This also poses an issue w/ our firewall design as all of the TCP sockets won't be available for the OS to inspect/filter.

Has anyone else ran into an issue like this in the past? I'm hoping that I have a misunderstanding of how things are working and I'm overlooking something, but am thinking it's time to start to redesign things...