Monday, August 26, 2019

Bad quality peering from US to Netherlands?

A customer of ours is running Windows DFS and Veeam backups (Baremetal with the Veeam Agent) from five locations around the globe to our datacenter in the Netherlands.

DFS is used to synchronize company-wide information with every branch, the DFS namespace is only 14GB in size.

Every branch location has a Read-Only Domain Controller which is also the local Fileserver (DFS), there is a business-type Internet connection with usually 100Mbps up/down speeds. We use ASA 5506-X's to setup a full-mesh IPsec overlay over each location. In addition each location has a IPsec tunnel to our datacenter in the Netherlands which houses their central off-site backup server.

The data that is sent in day-to-day operations over the Branch to Branch IPsec overlay is Active Directory related traffic, and the DFS delta's, there doesn't need to be a lot of bandwidth available for this purpose.

The most important thing however is the IPsec tunnel for the Veeam Backup. The job contains the entire local filesystem that is differentially backed up over night and fully backed up every month.

The size of the dataset is around 1.5TB per location.

We are currently experiencing issues with the backup from a branch location in Portland, Oregon. Previously the branch had a Comcast Business connection (I believe Starter Internet with only 50/5Mbps bandwidth). Since they have upgraded to a Allstream Business Fiber 100/100Mbps connection.

The full backup isn't able to finish in time before the 180 hour job runtime limit passes. The max bandwidth that we are able to achieve from the US to the Netherlands is about 500kB/s which equates to about 4Mbps, at this rate the backup would need 834 hours (1.5TB / 500kB / 3600sec = 833.333333333 hr) to finish. The minimum amount of average bandwidth we would need is around (1.5TB / 180hr / 3600sec * 8bits = 18.5185185 Mbps) 20Mb/s, we would think this is way less than should be available over the 100Mbps up/down Fiber Internet connection and thus should be achievable.

The datacenter upstream ISP has a full 1Gbps connection available and has more than enough bandwidth available when the backup runs.

Other branches (they are all in Europe) have no problems running the same type of backup job.

We have run multiple tests over the IPsec overlays and we are not able to achieve more than 25Mbps throughput from Portland, OR to Amsterdam, NL. Withouth the IPsec overlay we are not able to achieve more than 27Mbps throughput over the same path.

We have checked the traceroutes and the carrier's BGP Looking glass and can't see a uneccessary long path end-to-end. Our upstream ISP also cannot find an issue with the BGP path.

Could this be a bad peering issue? Are there other tests/things we could try?, we have contacted Allstream but their support is useless and the techs we spoke to are only able to troubleshoot last-mile issues (ISP handoff port issues).

TL;DR:

Customer has branches around the world connected with business ISP's. There is a off-site backup server in the Netherlands. The North-American branch connected via Allstream Business Fiber 100/100Mbps is experiencing end-to-end bandwidth issues that compromises their backup operations (nightly differential, monthly full (1.5TB). Other European branches have no issue running the backup job. Could this be a bad peering issue?



No comments:

Post a Comment