IT Problem - What makes 2 not like the other 12

Symptoms: At 2 sites of 12 The speed test at the firewall shows poor results that gets worse over time until the connection grinds to a halt.

Tests:
Rebooting everything simultaneously at the data rack clears the issue but it will return tomorrow or later today.
Rebooting the modem alone clears the issue but it will return tomorrow or later today.
Rebooting just the firewall has no effect.
Bandwidth graphs look pretty normal at control and affected sites.
At 1 affected site the Spectrum tech replaced the modem. No affect.
At the other affected site Spectrum did not as they said their tests didn't find a problem.
Running a speed test with and without the LAN plugged in has no effect.

Background:
11 (including both affected) sites recently had roughly 10 Fanvil X3U phones installed that connect to a central PBX (3CX)
12 sites are setup mostly identically setup and configured.
All 12 sites have Single cable Modem connection in a corporate environment.
All 12 sites have pfSense firewalls version 2.5.1.
12 sites are in an IPSEC mesh configuration.
All 12 sites have a 52 port PoE switch that powers a number of PoE devices.

6 of 12 sites have Spectrum (including the affected) with a speed
of 400/20 with 150/10 at the firewall under load.
1 of 12 is on 1000/1000 using a Comporium Fiber connection that connects the mesh to a Datacenter (Where the PBX is).
5 of 12 sites have Comcast with 500/35 speeds.