Single band router very slow in less interference surrounding

Hi,

I have one single band router set at channel number 4, kept beside single band router working on channel number 11. In my surrounding there are 2-3 more routers which are also operating on 2.4 GHz but on channel 7,1,11. I am getting very slow speed on both the routers. They at max go till 40 Mbps on 100 Mbps connection. At least channel 4 router should have performed better. I have checked through the cable I get ~96 Mbps speed. Channel -11 router is isp router and channel 4 router I used to get more speed by reducing interference. Any scope of increasing speed ? I know 5Hz router is the way to go, but any scope for improvement in current case ? If I lack knowledge please correct me.