How does a load-balancer handle too many connections, and how in case of websockets/SSE?

This isn't about whether one will ever need it. This is a general question, the answer of which I've been looking for as a curious CS grad.

Here's what I've learned from my research I have been doing for few days, after which I have by questions lined up.

​

PART 1 is about general load balancing with increasing number of connections

PART 2 is specific to persistent connections as websockets and Server-sent events

My concerns are in-particular to PART 2, so you can directly jump to that if you find it long.

​

PART 1 - general load balancing

Nginx is one of the many load-balancers available and is widely used. It can help in following ways:

- Small traffic, replicated servers for fault tolerance can be load balanced with servers referred by their IP address.

- Large traffic, multi-node/multi-server deployment where nginx handles the network requests and individual servers handle the CPU/DB operations.

​

However, there is a limitation on the number of connections which can be managed by the machine running the load balancer.

- RAM - this answer states that about 16GB RAM is required per million connections.

- Number of ports available - which can be handled by introducing virtual interfaces as the number of ports limit is 65535 per interface.

​

Still, this cannot handle large number of connections.

Another strategy is to have DNS based load-balancing which can region-wise distribute the traffic with the requirement of servers to be in sync and downtime as DNS cache update takes time.

Solutions like this one handle about a million clients on a single node, but I'm more concerned about a distributed system with enormous traffic.

What is the strategy services spanning multiple data-centers use to be available.?

​

PART 2 - for persistent connections

Persistent connections as SSE and websockets have redundant network operation on the load-balancer side as well as on the individual server. So I can think of two strategies

1. Get entire traffic through the load balancer (however this seems to be redundant on part of network I/O on proxy and server).
2. Pick a server for init request. The server returns it's canonical URL (say www1.domain.co) and then the client connects directly to the server, without nginx in between. This avoids duplicate network ops but removes location transparency.

How would you recommend to solve this problem?