Restrict SSH Access ACL Issues

So I've been having trouble with what should be a fairly simple task, but ACLs have never really been my strong suit.

I've got a Cisco ISR 4331 and it's sitting in front of our firewall. I want to be able to SSH into it from the MGMT port, but obviously only from the internal network. It should definitely block SSH from the Internet.

I've tried to restrict SSH access to the MGMT IP using the below ACL, but when I apply it it starts blocking SSH traffic. When I remove the ACL I can access it again.

I've set it to log both on allow and deny, and I can see a log entry when I try to access from the public IP, but when I try to access the management IP it says connection refused and doesn't log anything.

I've tried both standard and extended ACLs and I can't get it to work. I've had to disable SSH until I can get this fixed up.

If anyone has any suggestions on how to get this to work. I just need to stop SSH access from the public internet.

Diagram: https://i.imgur.com/wsJjH3D.png

The ACL config:

ip access-list extended SSH-ACL permit tcp 10.0.0.0 0.255.255.255 any eq 22 log deny tcp any any eq 22 log exit line vty 0 15 login local transport input SSH exec-timeout 5 access-class SSH-ACL in exit