Firepower - File policy inspection for malware, do you enable on every access rule?

have tons of acls and majority we have file policy enabled to scan for potential malware from source to destination, ive read its best not to do this because of resources and only enable on rules with untrusted destinations and nothing internal to other internal destinations, this seems odd because other internal servers could get infected just as much as untrusted internet file servers..any thoughts?