Best way to share a serial device over LAN

I have a serial device with an RS232 port, and I'm trying to give shared access, over LAN, to three specific VMs.

I'm looking at a single RS232/RJ45 Serial-to-Ethernet device server. But from my understanding, only one VM will be able to connect to it.

So I'm wondering if anyone has had experience with this? Am I on the right track, or completely off in my approach?

​