Need help identifying point of origin for traffic transiting edge router.

Hey all, I've been working on a bit of a problem over the last week and I'm at my wits end. I've written a bit of a novel below - so give it a read if you have the time and let me know your thoughts.

​

We have a port mirror set up on the outside interface of our edge router at each of our sites. This port mirror sends traffic to different monitoring servers. Our security team recently noticed some unusual traffic transiting the outside interface.

The traffic is of moderate volume (20-100mbps) and is comprised entirely of IP fragments. The source and destination are both public IPv4 addresses that do not belong to us, nor do they exist anywhere on our network.

To route this traffic, our router will just be using the 0.0.0.0/0 route outbound to our ISP.

So either our ISP is routing this traffic to us in error, or something on our network is trying to punch out to that destination address. Problem is, I can't even figure out if the traffic is inbound / outbound from our network.

The traffic is also present across multiple sites, these sites use the same router hardware and topology, but they have a different ISP. The traffic is the same at both sites, the only difference is the source address which is different (but still a public IPv4 address).

Furthermore, the destination IP address in both cases has the owner listed as "DoD Network Information Center" (suspicious ?!?!).

​

-----------------

​

Some background on our physical topology - Each router interfaces directly withb the ISP NTU. Each router has two downstream switches, each with directly connected firewalls and appliances.

​

E.g.:

Firewall <----> Switch <----> Router <----> ISP handoff <----> ISP

Routing topology is as follows:

We receive the 0.0.0.0/0 route via BGP from the ISP at each site.

Each edge router has a number of GRE tunnels with a DDoS scrubbing service. We advertise via BGP our public IPv4 prefixes to the DDoS provider over the GRE tunnels. All inbound internet traffic gets routed to us via the DDoS provider over the GRE tunnels.

​

-------------------

​

Here's a list of steps I've taken to identify the point of origin for the traffic:

​

• I have checked access-lists on all interfaces on the router (including the outside interface). There is a rule in each list that should be denying this traffic (deny ip any any fragments), however none of these rules have incrementing hit counts. I've also tried adding a deny ip at the top of each list for the explicit hosts, did not define 'fragments', and added the 'log' statement. This didn't capture or log the traffic either.
• I have checked access rules and logs on downstream firewalls, however this traffic has not been logged in any way (either pass or deny).
• I have performed a local packet capture on the edge routers using a 'monitor capture' (both ip cef & process-switched), however this did not capture the traffic at all. (Worth noting the local captures worked fine if I substituted the suspicious hosts for known good hosts in the filter).
• I logged a Cisco TAC case to ask why the above packet captures didn't work, and they are stumped.
• I ruled out a problem with the monitoring server by plugging a laptop directly into the port mirror interface on the router - this DID capture the traffic. I can now rule out a problem within our monitoring environment, and know for certain that the traffic must exist on the router.
• I have engaged our ISPs to see if they can see the traffic. They claim they can't see it - however I'm still pursuing this avenue as I believe they have not looked properly.
• I have created additional traffic export policies to mirror traffic on the other inside interfaces on the router - these did NOT capture the traffic. (Seems to only exist on outside interface).
• There are no GRE headers on the traffic captures on the outside interface, so I suspect it's not coming to us from our DDoS provider, however I logged a support case with them anyway - they claim they cannot see the traffic, and have assured us that they would not route traffic to us that wasn't in our prefix list.
• I can't back-trace the traffic via MAC address as the Cisco traffic-export overwrites the observed MAC address of each packet with the MAC address of the router's export interface.

​

-------------------

​

So far the only place I can see the traffic is in the traffic export on our outside interfaces - but I can't even figure out which direction it's going. Does anyone have any tips or suggestions for further troubleshooting? I'm absolutely stumped.