Is bufferbloat ever an issue for internet traffic on the inner router with double NAT?

Suppose I have the following situation:

clients <-> router A <- 500 Mb/s -> router B <- 50 Mb/s -> internet

where all links have symmetric speeds. Do I need to worry about bufferbloat at all on router A? If so, should I run CoDel with a 50 Mb/s limit or a 500 Mb/s limit?

Thanks.