What software allows you to do routing as code?

I have a server with 2x 10gbit uplinks and i am looking to create a system to do some more advanced filtering (mainly for stopping basic UDP ddos attacks)

What i want to end up doing is: wait for a client to do a TCP handshake to port 30120, and then open up that port on UDP too for that specific client.

This Should lower the load on that machine during a ddos attack (given that all other udp ports are blocked. My provider should be able to handle TCP ones)

I can handle the coding part. But i was not able to find any frameworks that let you do this kind of routing without breaking far into the linux kernel.

I've looked at pfsense but that does not seem to have a easy way to add custom rule packages.