NAT instead of firewall

I feel like I might be sent to networking hell for asking this question. I understand that "obscurity isn't security" etc. I'd never implement this in an enterprise, but this is for my home network.

The scenario is that I have a Cisco 1941 on my 220Mbps home broadband connection. If I enable NAT + Zone Based Firewall then the CPU on my router tops out at something like 180Mbps.

If I use CBAC I get a little more, but still not 220Mbps.

I'm now running OK using reflexive ACLs like it's 1999, but wondering do I even need reflexive ACLs?

I'm PATing everything to my outside IP. The only way I can see someone is going to get in to my network from the outside is if they're directly connected to the outside (its cable so I imagine they could be) and then set something like a static route with my Internet IP as their next hop for my internal RFC1918 address range.

So, could I get away with just PATing everything to my WAN IP and then having an ACL ingress from the internet similar to the following:

deny from any to RFC1918 permit from any to any

The PAT would act similar a stateful firewall, where any packets destined to my public IP that don't have a live connection are dropped and the ACL in from the outside prevent anything addressed to my internal address range being forwarded.