How does Cisco IP SLA work?

A customer I work with has two buildings. Both have their own networks, cores, ISP, and ASA's. In each building we have IP SLA echo configured to reach out to something reachable at their ISP. When if the IP SLA fails to reach get a response from whatever it is pinging it will cause the core switch in that building to send it's default traffic to the core of the other building which will then send the traffic out it's own ASA to its ISP.

Over the weekend we had both ISPs go down which created a loop of building A sending traffic to building B and building B sending traffic to building A. I changed the weighted default route to send traffic from building A directly to the building B ASA instead of the core and this ended the loop and brought everything back up.

Immediately when I made the change the loop stopped and both SLA's on both core could reach their respective paths on their own ISP. The question is: Regardless of the loop shouldn't the SLA on each core switch from each building still be trying to get to its destination via the original default route to its own ASA and not via the weighted one to the other building? If it could then the loop would have corrected itself...correct?