Trying to Anycast DNS, getting "reply from unexpected source"

I know I'm probably missing something simple here.

I'm trying to anycast a few DNS servers. Routes are announced via BGP and are working/visible. The DNS isn't working though.

If I try and dig something @ the anycast IP:

;; reply from unexpected source: public_IP#53, expected anycast_IP#53

I'm not 100% sure what I'm missing here as it should be working