What causes latency between providers?

We have two buildings, each with two internet providers, AT&T and Verizon (same two providers at both buildings).

Latency from building 1, ATT to building 2, ATT is ~2ms. Latency from building 1, Verizon to building 2, Verizon is ~2ms. However, once you cross providers, from building 1, ATT to building 2, Verizon, latency jumps to ~25ms.

Obviously, the peering between the two providers causes some added latency. Why/what causes this? Would you expect that kind of increase in latency when both buildings are in the same city?