What is the disadvantage of using a protocol design in which UDP and IPv4 are merged?

This being an address of 48 bits that includes a 32 bit IPv4 address and a 16 bit port number?