Cisco Firepower Rant II

Three months ago I wrote a Cisco Firepower rant (https://www.reddit.com/r/networking/comments/9363af/cisco_firepower_rant/). It received more attention than I imagined, not only from the /r/networking community but appearently also from the vendor side. I had a nice chat with a TAC engineer, got a PM from the supposedly new ngfw product owner, who decided not to return my PM after I told him I would be willing to write up some feedback, but not through a cisco customer program as he suggested, but only anonymously via reddit. And last but not least it looks like Mark Garrett (BoA @Cisco) also got the memo and even left a comment.

Before writing up my post I thought there might be different opinions on the product, but it was overwhelmingly one sided, with 96% upvotes out of ~ 27.5k views and comments that mostly reflected my own personal experiences. On one side it was quite satisfying to get so much matching feedback, but on the other hand it was quite frustrating to see everybody still struggling with a product that should be rock solid after so many years of development.

I would like to use this post to talk about some lessons I have learned in the last few years, but before that let's stick to Firepower for a few more paragraphs. As of today the newest stable release is 6.2.3.6 - basically the 6st patch release for the newest minor release 6.2.3 and let me just tell you one important thing. Nothing. has. changed. It's still the same mess that I described in my first rant and in the mean time I found even more issues that cost me quite some time.

As you might know Cisco introduced IPSec VPN in 6.1.0 (2016-09-29) and AnyConnect VPN in 6.2.1 (2017-05-15). Up until a few months ago I was able to avoid VPN features, because I was still fighting against basic features like HA (which by the way also break in 6.2.3.6, and also broke when upgrading from 6.2.3.5 to 6.2.3.6), but I finally had the honor of migrating several VPN services from ASA to a FPR4100 cluster and let me tell you it is a real mess. If you configure a VPN tunnel, make too many changes to the tunnel and re-deploy, the CSM code that is used to generate the required configuration (e.g. remove old configuration and replace with new configuration) gets totally confused resulting in out-of-order configuration commands (csm tries to delete config elements that are still referenced) and you will run into the typical rollback procedure where the configuration is wiped and basically reloaded - not very cool if you have a lot of VPN tunnels that die and must be re-established in the process. I have seen this phenomena with both classic ipsec site-to-site tunnels but also with AnyConnect client-to-site configurations... So by all means if you are in the situation of migrating VPNs think twice about migrating vpn configuration from asa to ftd, since apart from the buggy mgmt plane implementation there is neither an API nor a migration tool that will help you port the configuration... and now imagine what it is like to migrate a large installation with 100 tunnels and many anyconnect profiles, clicking through a painfully slow web ui, hoping that the changes you have made can be deployed and you won't need to delete your configuration and start from scratch again...

As mentioned earlier let me use this post to talk about a few things I learned during the last few years. Lessons learned over the last years of about 2000h of doing firepower deployment, upgrades and day-to-day operations.

1 Don't put your eggs in one basket

Just because you are already using product xy from vendor z, don't artificially limit your choices and prefer a product from your existing vendor. When I started in IT my mindset was to aggregate as much as possible (vendor wise). It's the pipe dream of having one-throat-to-choke, the belief that it will be easier to resolve issues because there is only one number to call, but from my experience it doesn't matter if it is different vendors or different TAC apartments, you will always have to go through the blame game. Apart from the support situation you create an over reliance on a single external entity that will push their own agenda to upsell you on stuff you don't need... or you really need the 2nd product that will fix operational challenges from their other product (I am looking at you cisco assurance center)

2 Don't mistake your VAR with a consultancy

Even though many VARs also do consulting you should keep their intentions in mind. They will try to up sell you on stuff that they are knowledgeable with. They are mostly bound to the vendors they partner with and will act in their own interest to achieve their economic goals (e.g. sell new product from vendor X to receive additional incentives). If your org is large enough and you lack technical expertise to make the right decision get somebody on board who gets paid to tell you the truth. I am not saying your VAR can't take the consultants role but keep in mind who you are dealing with and what their primary interest is.

3 Test before you buy

If you are spending a lot of money on infrastructure you must know what you want. Make a criteria catalog of features that you want to use and create a test suite that bidders must abide to. I know that this is more relevant to larger organizations that have the resources to articulate what they want and have the manpower to accompany POCs but if we are talking about new a product or a major architectural change this is something you should consider. See what the proposed solution feels like and rate them based on the criteria list you created. Even if your organization ends up making the wrong decision (maybe due to price or politics) you will end up with leverage against whoever screwed up and decided to buy the inferior solution. At the end of the day it is your organization that is spending money, you will own the faulty product and it is partly your fault for getting screwed over, if you don't do your homework and blindly trust another company.

4 Don't be a fanboy

To some degree we are all biased. If you have had good experiences with Palo Alto you will tend to trust them and might consider their Traps product over Cisco's AMP for Endpoint because you got totally screwed by firepower. Try to fight the urge of attaching yourself to some brand. While it is totally reasonable to be aware of a vendors strong and weak points (e.g. vendor x support has always been top notch, while vendor y support always ends up with you talking to an indian tac engineer who you cannot understand that contacts you right before his shift ends) you should try to focus on technical details and blend out the shiny powerpoint presentations, promises and marketing (I am looking at you Gartner for placing firepower into the leader quadrant - seriously do you have any credibility left?)

Don't be a vendor marionette and concentrate on the underlying technology. IMO if you want to have credibility as an engineer you should shy away from programs like Cisco Champion or VMware vExpert that basically reward you for having the "right" opinions. Having strong opinions on products is one thing, but what technical reason do you have to prostitute yourself for a vendor? If you value your integrity as a technical expert you should think about how you want to be perceived by the people around you. Do you want to be valued for your honest and objective view on things or as a fanboy?

5 Realize when it is time to move on

I would consider myself to be very resistant to learning from mistakes and it took me quite some time to realize that investing so much time in a product / technology that has no future was a mistake. I think I was over attached to Cisco, or rather vendors in general and a bit naive. Hopefully I will be able to move out of network security soon and focus more on technologies that are more rewarding. Realizing what it is you don't want to do is important, if you end up in the wrong field you shouldn't waste your time on something that only drags you down.

Thanks for making it through my post, let's hope this is the last time I had to write about firepower.