Huge latency increase caused by saturating 1 port synced @ 100mbit on unmanaged 6-port gig switch.

I ran into this weird issue in my home last night and was curious what exactly the cause is.

I have gigabit FTH in my condo, running pfsense on a white box mini pc with only 2 Ethernet ports one wan one lan. The pfsense lan port connects to a 6 port unmanaged dlink switch which connects to the punch down block for all the Ethernet ports in the condo. Guess they did a pretty poor job with the runs because some of rooms will only sync up @ 100mbit and other will run at gigabit.

Anyways I noticed that if I saturate a port synced at 100mbit my wan latency increases by 50-200 ms and really bad jitter. I can nearly saturate my gig connection on another port synced at gigabit speeds and it barely effects latency or jitter of my wan connection.

Would this be caused by the switch having a hard time buffering packet queues between different interfaces synced at different speeds? Can this be fixed by getting a better switch?