Question regarding buffebloat chart from dslreports and if I should look into it more?

I get an F rating for bufferbloat on dslreports. When I look at the chart after the test, the chart shows that while Idle it spikes to over 1000ms, download also spikes past 10000ms, and upload barely spikes at all.

I have heard that bufferbloat really only affects you jf other users are hogging up all of the bandwidth, but if the chart shows spikes while Idle, does that mean that it is possibly affecting me at all times? Or what does the idle portion actually mean?