Ubiquiti: Routing in Wonderland…

TLDR Version:

What in the world is Ubiquiti doing?! Why are we empowering a company that gives a giant middle finger to accepted networking and security standards (not to mention their customers)?

Central Point Here: Why does Ubiquiti listen to every gateway IP (of every network including WAN) as a VIP on every defined network?

Putting aside that this flies in the face of a handful of accepted networking standards and practices, this has the potential of causing several routing and security problems within the environment. Why on earth would we do this?

Most critically, why does deploying firewall rules restricting cross-network communication not stop this behavior? I can tell you why… They define each IP as a VIP on each interface… WTF?!

Full Form:

I have been spending the last week “working” with UI Support to try to untangle a mess around Port Forwarding / Firewall Rules and Routing that has left my head spinning. We have our network built upon the Unify Dream Machine (UDM) Pro along with several other pieces Ubiquiti hardware including Switches and Access Points. While there are several things I initially appreciate about the Unify Platform (unsolicited Letterkenny reference…), unfortunately, this deployment has never worked as expected since day one.

Of all the other issues (or frustrations) this latest issue has got me pounding my head against my desk screaming “Why!....” to the networking gods in the heavens. As it turns out, simple networking concepts such as Port Forwarding are never as straight forward as they seem once you enter the Twilight – I mean Ubiquiti Zone… As it turns out, Unify takes a rather drastic routing shortcut that has the potential to mess with how the network fundamentally operates: Allowing the gateway to listen on every assigned IP address as a VIP on every defined network regardless of the network / vlan that IP address is associated with on the gateway (even WAN).

What does this mean? In simple terms assuming you have two defined networks on Unifi (192.168.1.0/24 and 192.168.2.0/24), the Unifi gateways (USG or UDM / UDM Pro) will listen for traffic destined to EITHER x.x.1.1 OR x.x.2.1 on both networks – this occurs even when firewall rules are defined specifically to deny inter-network traffic. This is to say that while a host at 192.168.1.100 (Alice) cannot speak to host 192.168.2.100 (Bob), Alice CAN speak to 192.168.2.1 and conversely Bob can also speak to 192.168.1.1 – all of this happening even in environments where networks 192.168.1.0/24 and 192.168.2.0/24 are not able to communicate and should not know of the existence of each other. Okay, so this may not be a total nightmare… unless:

1. You have very large and complex networks with complex routing requirements.
2. You have non-Unifi networks defined (sub-networks) lower in the stack that leverage the same IP space as another network in the Unifi platform that by all intents and purposes should not be routable from the network you are operating on… (not a very clean or expected setup, but I have seen this happen on occasion).
3. MOST CRITICALLY: When you take into account that this same routing shortcut applies to traffic destine for the gateway’s WAN IP address. Yes, that is right, sending traffic to 12.23.34.45 (WAN IP) from Alice (192.168.1.100) goes through exactly one hop as the Unifi system is designed to listen on every port (every network) for traffic to ANY of its IP Address (regardless of the network that IP address is attached to). It is simply mind boggling that this traffic is not first routing out and then back in.

In the environment in which I am having so many issues, we have the need to make available critical services that exist on the 192.168.3.0/24 network. These services are behind UDP 10520 through 10530 (and yes, the ISP is allowing this traffic) – call this the “Happy” service. Because of the nature of these services and the availability requirements, the Happy services must be accessible at the company’s FQDN (company.com) which, of course, ties back to 12.23.34.45 as the company’s static IP. Port Forwarding rules have been created and enabled to allow such traffic through; however, it is critical to understand that these Port Forwarding rules only apply to the WAN interfaces (WAN / WAN2). This then creates an environment in which, for devices internal to the network, they are not able to route to the Happy service on company.com / 12.23.34.45 as all interfaces (even LAN) are listening on this (WAN) IP address. As the traffic is incoming to 12.23.34.45 on the LAN interface, the Port Forwarding rules are never applied and therefore the traffic cannot route.

Now some may be asking, and somewhat rightfully so, why not just setup the internal (on-network) devices to use the private IP addresses of the Happy service. This could be a great work around, except, these services must be accessible seamlessly to the user as they move between being on or off the corporate network. This is not to even mention the argument that this is not how routing is supposed to work.

This is not the only instance in which I ran into a particular function or feature within Unifi that didn’t appear to work properly just to find out that Ubiquiti “doesn’t do things that way…” Since the early days of deploying this equipment it became clear to me how Ubiquiti seems to take a dump on accepted networking standards and employ dangerous shortcuts in routing, firewalling, security, and network management. My message to Ubiquiti: let’s grow up, stop taking shortcuts, and actually provide the features and functionality you promise in a manner that aligns with accepted networking standards – please, just have a sit down with the folks over at IEEE.

This would all be far less concerning and aggravating if Ubiquiti support ever really gave a damn and treated their customers with the respect they deserve. Every support experience I have had with Ubiquiti has either ended in “oh, well with how Unify works you can’t do that” or being belittled and talked down to as if I know absolutely nothing about even the simple functions of breathing. This would be like calling up Nike and explaining that my shoes are missing their laces and then being taken through a frustrating three-hour process where they try to explain how to tie my shoes (“is the computer plugged in sir?”). This is not the problem!

To be fair, in a recent conversation with a Manager in their support department, they acknowledged they have several support issues that they are trying to work through, but have never really addressed them. Great, we have reached the acceptance stage, but now is the time to FIX the problem.

Note: Today I work for a large cyberspace security vendor… we too have our support issues. But nothing any of my customers have ever reported (which is some very head scratching and aggravating shit), even comes close to the average support call I have with Ubiquiti.

Let’s be clear… I am not here to simply rail on Ubiquiti and the Unifi platform – well, in some way I am (writing this has actually been very therapeutic). I am here to spread the awareness of these issues so others working with the Unifi platform don’t fall into some of the same pits of despair I have. Believe me, I would much rather be an advocate for Ubiquiti – I mean what are the reasonable alternatives these days (providing similar functionality): Cisco Meraki, which is outlandishly expensive in hardware, support, and licensing costs. I am also hear to see if we can start to build a ground swell of support to pressure Ubiquiti into fixing some of these issues and to stop taking such shortcuts.

To frame this discussion (and responses) quickly, I have been in Networking and Cyberspace Security since 2001 (longer than Ubiquiti has been in existence). I have experience architecting and deploying networks of varying complexities and sensitivities from standard corporate networks to highly secure and even air-gapped networks. I have extensive experience working directly with a large range of networking and network security vendors and their products, including: Cisco (including Meraki & Linksys), Juniper, Extreme, Check Point, Fortinet, Palo Alto, NetGear, Dell, HPE (including Aruba & 3Com), and even TACLANE. I also have experience with Cisco ACI, VMware NSX, and with networking in all the major cloud vendors. I say all this in attempts to weed out knee jerk responses of “You don’t know what I’m talking about” – though, I am certain those will inevitably come.

I am curious to hear what insights other people here may have along with any other “oddities” where things don’t work as they otherwise should in the world of Ubiquiti.