Trying to setup tunnel to router for multiple sites

I think a VPN tunnel here is something that would do what I need, but am not sure.

So I have equipment that I manage that is located at customers sites throughout my state. Probably around 10 locations. The setup’s are very typical… typically

• ISP Modem

----Customers equipment (Printers, PC’s etc)

----Unmanaged switch dedicated to my equipment

So what I want/need to do is find a way to somehow “tunnel” in to the network from off-site so I can access whatever equipment and resources I need.

My thought was to somehow put some type of mini router to where the ISP’s modem/router provides a “wan” connection to the mini router, and then my unmanaged switch would hook into the “lan” side. I imaging using some type of mini-router with openwrt or ddwrt so that I can have it act as a vpn client. None of these locations have static ip’s, nor is any port forwarding an option, so I imagine I would need to host a VPN server which I could do on a VPS as I can give that a dedicated IP / URL.

Reason for no static ip’s or port forwarding being an option is we frequently have issues where customers change ISP’s or have the ISP’s come change out equipment and no one makes us aware ahead of time, which causes issues as the settings never get put back in right which causes us to not be able to connect, and in some cases where the customer works from off site, prevents them from connecting too.

These VPN connections would not handle heavy traffic. Some sites would only see a VPN connection once a week for about an hour while someone audits equipment. But we do need to have some basic user control. To where I could make an account called “Site1User” that would allow a vpn connection to Site-1 , same with site-2, so on and so forth, this way the customers can connect as well. And of course, one for myself to access any sites, although that isn’t a requirement, I can have a separate login for each site.

The ISP modems/routers we are dealing with aren’t anything commercial, typically it’s the equipment like you or I would have in our homes.

These locations are usually unmanned, and I am the one they come to with any network issues to begin with.

Of course I am open to other options, I am decent when it comes to basic networking stuff, but this isn’t anything I have done before.