3750X - Restrict SSH access to non Management SVI's

Hi,

I cant seem to find any working examples of this elsewhere.

I have a 3750X core switch with the below SVI's:

VLAN 100 - 10.1.0.254/24

VLAN 101 - 10.1.1.254/24

VLAN 102 - 10.1.2.254/24

VLAN 103 - 10.1.3.254/24

Vlan 100 is the management vlan but out of the box I can ssh into any of the above IP's without any problem.

How can I get it so that when I SSH into 10.1.0.254 it works but when I ssh into the other addresses, it doesnt work?

I am already able to restrict based on source but am wanting to restrict on destination instead.