Sunday, November 17, 2019

Moronic Monday!

It's Monday, you've not yet had coffee and the week ahead is gonna suck. Lets open the floor for a weekly Stupid Questions Thread, so we can all ask those questions we're too embarassed to ask!

Post your question - stupid or otherwise - here to get an answer. Anyone can post a question and the community as a whole is invited and encouraged to provide an answer. Serious answers are not expected.



Need help identifying point of origin for traffic transiting edge router.

Hey all, I've been working on a bit of a problem over the last week and I'm at my wits end. I've written a bit of a novel below - so give it a read if you have the time and let me know your thoughts.

We have a port mirror set up on the outside interface of our edge router at each of our sites. This port mirror sends traffic to different monitoring servers. Our security team recently noticed some unusual traffic transiting the outside interface.

The traffic is of moderate volume (20-100mbps) and is comprised entirely of IP fragments. The source and destination are both public IPv4 addresses that do not belong to us, nor do they exist anywhere on our network.

To route this traffic, our router will just be using the 0.0.0.0/0 route outbound to our ISP.

So either our ISP is routing this traffic to us in error, or something on our network is trying to punch out to that destination address. Problem is, I can't even figure out if the traffic is inbound / outbound from our network.

The traffic is also present across multiple sites, these sites use the same router hardware and topology, but they have a different ISP. The traffic is the same at both sites, the only difference is the source address which is different (but still a public IPv4 address).

Furthermore, the destination IP address in both cases has the owner listed as "DoD Network Information Center" (suspicious ?!?!).

-----------------

Some background on our physical topology - Each router interfaces directly withb the ISP NTU. Each router has two downstream switches, each with directly connected firewalls and appliances.

E.g.:

Firewall <----> Switch <----> Router <----> ISP handoff <----> ISP

Routing topology is as follows:

We receive the 0.0.0.0/0 route via BGP from the ISP at each site.

Each edge router has a number of GRE tunnels with a DDoS scrubbing service. We advertise via BGP our public IPv4 prefixes to the DDoS provider over the GRE tunnels. All inbound internet traffic gets routed to us via the DDoS provider over the GRE tunnels.

-------------------

Here's a list of steps I've taken to identify the point of origin for the traffic:

  • I have checked access-lists on all interfaces on the router (including the outside interface). There is a rule in each list that should be denying this traffic (deny ip any any fragments), however none of these rules have incrementing hit counts. I've also tried adding a deny ip at the top of each list for the explicit hosts, did not define 'fragments', and added the 'log' statement. This didn't capture or log the traffic either.
  • I have checked access rules and logs on downstream firewalls, however this traffic has not been logged in any way (either pass or deny).
  • I have performed a local packet capture on the edge routers using a 'monitor capture' (both ip cef & process-switched), however this did not capture the traffic at all. (Worth noting the local captures worked fine if I substituted the suspicious hosts for known good hosts in the filter).
  • I logged a Cisco TAC case to ask why the above packet captures didn't work, and they are stumped.
  • I ruled out a problem with the monitoring server by plugging a laptop directly into the port mirror interface on the router - this DID capture the traffic. I can now rule out a problem within our monitoring environment, and know for certain that the traffic must exist on the router.
  • I have engaged our ISPs to see if they can see the traffic. They claim they can't see it - however I'm still pursuing this avenue as I believe they have not looked properly.
  • I have created additional traffic export policies to mirror traffic on the other inside interfaces on the router - these did NOT capture the traffic. (Seems to only exist on outside interface).
  • There are no GRE headers on the traffic captures on the outside interface, so I suspect it's not coming to us from our DDoS provider, however I logged a support case with them anyway - they claim they cannot see the traffic, and have assured us that they would not route traffic to us that wasn't in our prefix list.
  • I can't back-trace the traffic via MAC address as the Cisco traffic-export overwrites the observed MAC address of each packet with the MAC address of the router's export interface.

-------------------

So far the only place I can see the traffic is in the traffic export on our outside interfaces - but I can't even figure out which direction it's going. Does anyone have any tips or suggestions for further troubleshooting? I'm absolutely stumped.



OOB Modem Suggestions

We use a mix of PSTN and cellular modems for OOB connectivity at our managed sites, worldwide. The cellular modems are only certified/homologated in some countries (US, EU, a handful of Asia) and are more than double the price of PSTN modems, so for a lot of sites we only use the PSTN modem. The problem is, the modem that we use for non-US/EU (MultiTech MT9234ZBA) has gone EOS and I'm struggling to find a replacement that even comes close to reaching the number of countries that the MT9234ZBA was certified for.

Does anyone know of any PSTN modem that has global-ish certification? MultiTech doesn't have a replacement and neither does the vendor that we use for US/EU (US Robotics).

We're going to be in some really deep shit soon. Any leads would be much appreciated.



Catalyst 9200 Alternates?

Operating a very simple infrastructure here

3 Extreme Networks Summit 400-48t and 5 450-48t aggregating to a Catalyst 3500.

Cisco has recommended a go in with 8 9200's and 9300 as the aggregator. Pricing seems a little steep with Cisco. I looked at Dell and their prices are 25% lower than Cisco, but I have heard performance issues.

Is Cisco worth the premium? Stacking with uplink is required for us. Thoughts?



Powered switch in media box?

Is there any issue with a powered switch in a media box ?



What are some good options for back up internet?

I have shitty windstream internet that goes out several times per year for several days at a time. Unfortunately it's my only option living in a rural area... What would be a good backup system that would allow me to only use when needed. Like a pay as you need setup. I have Att and Verizon towers in reach



Switching ASIC Packet Processing specialization for SWE

I'm a SWE with 1 year experience working at a large network devices manufacturer (think Cisco, Juniper etc.). I'm on a platform team, working closely with the switching ASICs on our ToR switches. I've been getting more work in the areas of Packet Processing and Traffic Management lately, and was wondering if these are reasonable areas to try and specialize in for my career going forward. This requires a lot of general networking knowledge as well as an in-depth understanding of the hardware architecture and specific chip revisions. Pretty much I think this is really interesting stuff but I'm a bit concerned that it might be a bit niche, and that the large investment I made in learning how these systems work in the detail required would only help me if I work for a small group of large networking companies for my whole career.

If anyone with experience in these areas could give some advice, that would be much appreciated.



Esxi pfsense bridged lan

I currently have 5 interfaces on my lab. 1being from the motherboard, and the other 4 from an intel nic. I would like to use all 4 of the interfaces on my 4 port nic as lan ports and my 5th motherboard interface for wan.

How would i go about this? I tried creating a vswitches jnside esxi for wan and the 4 lan interfaces and passing those separately. This seems to work only when there is a single ethernet plugged into the lan nic. I would like to he able to plug 4 ethernets into my nic.



Cloud/SaaS. Am I missing something?

So Cloud/SaaS may make sense if you have users distributed all over the world accessing resources via the web.

But I’ve seen a recent trend of enterprises taking internal, on-prem applications and rushing to stick them up in the cloud. This tends to make performance for those applications much worse for enterprise users depending on the WAN configuration.

Look, no offense against SD-WAN, but I think it’s a fair statement that most enterprises are still using a provider managed L3VPN/L2VPN as their WAN. So, the vast majority of enterprise WAN is built that way. Usually Internet access in these environments is centralized at the Enterprise data center(s).

Depending on the size of the company sometimes you’ll see DIA at the WAN spoke, but usually not unless they have a UTM/NGFW appliance at every branch office. I know many National and Global corps may do that, but the vast majority of enterprises which are SMB and regional/semi-national usually don’t have UTM/NGFW at each site, so their WAN users only access the Internet across the L3VPN back to the datacenter.

This makes Cloud/SaaS bad. Because you’re adding significant latency between the users and the server. Depending on how robust they set up their WAN, they may have to send traffic across the country to their data center, and then back across the country the other direction to reach that cloud app.

This all leads to a traumatic user experience where everything will load super slow and anything interactive will be super laggy.

But the cloud has been sold to leadership as being better. It has more resilience and redundancy built into it, more robust with better resources, and it’s what everyone is going to. And we no longer have to worry about upkeep!

So that’s why the user complaints are met with outrage and accusations. IT said this would be better and now it’s so slow our business units have less productivity!

This leads to bad situations like “increase our wan sites bandwidth now!” “But sir, they’re using less than 5% of their bandwidth.” “I don’t care, the Chief of Sales is yelling at me get them more bandwidth!” One month later: “what are you guys doing?! He said it’s just as slow but we bought 10x the bandwidth!”

What is the solution here? DIA at every spoke? Does not scale well due to the cost of UTM/NGFW at every location. Bring applications back on premises? A non-starter for most enterprises who want to continue scaling back overhead and infrastructure. SD-WAN? It sounds like a silver bullet, but carries its own problems, and the industry is slow to adapt.

Am I missing something? Why are businesses rushing to the cloud, and then universally hating it when they get there? Is this merely a passing fad? Or will “solutions” designed to fix these self-created problems begin gaining more market share?



What is the exact requirement for a password to be considered a valid encrypted secret

I need to be a little vague to avoid test comp, but there is one question I keep thinking back on even days after my test that is bugging me.

When you enter an enable secret command and specify an encryption level, let's say 5, and enter a password what exactly is the criteria that it's looking for to determine that what you've entered is a hashed value and not plain text?

For instance, enable secret 5 C1$C0T3$T$aRE$TuP1D Is clearly in plain text but could it be messed up enough that the router thinks it's hashed and be accepted? What exactly is it looking for to make that distinction? I'm upset that Cisco is even making me consider this little nuance, but I can't find an answer anywhere.