Juniper SRX FBF and Dest NAT question

So i have been working on a configuration for an SRX240H2 with 2 ISP connections. I am using FBF to forward packets to the correct ISP, I have this portion working. SRC NAT is also working as intended. Problem I have is with Destination NAT not working as expected. I have looked at the packet flow from Juniper regarding how incoming packets are processed, so If I am correct, Destination NAT should be applied first, then it determines the route for the incoming packet. The default routing instance (I'm assuming is inet.0?) has the correct route for the incoming packet after destination NAT, but I am still unable to pass traffic. If I remove the interface from the Filter for FBF and route using default instance, Destination NAT works correctly. It is only when the interface has an input filter applied to change the routing-instance to a virtual-router in forwarding mode. Am I missing something simple with DEST NAT to get it working? These are for some simple services that just need port forwarding.

configuration looks something like this:

routing-options { interface-routes { rib-group inet IMPORT-PHY; } static { route 0.0.0.0/0 next-hop [ <ISP-A> <ISP-B> ]; route <internal routes> ... } rib-groups { IMPORT-PHY { import-rib [ inet.0 routing-table-ISP1.inet.0 routing-table-ISP2.inet.0 ]; } } } firewall { filter filter1 { term term1 { from { source-address <Some Range>; } then { routing-instance routing-table-ISP2; } } term default { then { routing-instance routing-table-ISP1; } } } } routing-instances { routing-table-ISP1 { instance-type forwarding; routing-options { static { route 0.0.0.0/0 { next-hop ISP-A; qualified-next-hop ISP-B { preference 100; } } } } } routing-table-ISP2 { instance-type forwarding; routing-options { static { route 0.0.0.0/0 { next-hop ISP-B; qualified-next-hop ISP-A { preference 100; } } } } } nat { destination { pool 1{ some internal IP/32 port XXXX; rule-set tmp { from interface ge-0/0/0.0; rule 1_tmp { match { source-address 0.0.0.0/0; destination-address External-IP-ON-ISP-A/32; destination-port { XXXX; } } then { destination-nat { pool { 1; } } } } 

I have tried changing the from field for the destination nat rule to the untrust zone which contains both WAN interfaces, and that has not made a difference. I think I am missing something small in the routing portion, but I am not sure. Any input would be useful, and I can post a full config somewhere if needed.

Thanks.