Fed vs. Contractor

I'm currently an IT contractor and have been recently thinking about if I should look into going over to the Fed side. Been contracting since I separated from the military about 10 years ago. I've read about all the pros and cons of each such as better pay as a contractor but less job stability, pension and TSP on the Fed side, benefits, etc.

I'm in a location where there's consistently lots of contractor job openings in my field and in the past it's only taken me a few weeks to get an interview and be hired. Also even if the contract I'm on is won by another company, as a contractor you get first right of refusal so it's not like it's a done deal that you just lost your job. I've always had good benefits, vacation time and matching 401(k)s with my companies. I don't have a family to support or a mortgage (like pretty much ALL the civilians I work with do) so it's not like if I found myself unemployed for a while it would be a big deal.

What concerns me about working on the Fed side is what happens if you get hired but end up not liking the position, place you work, co-workers, supervisor, etc. What then? That seems like I would be trapped after I've worked hard to get this "lucrative" government job. I've heard it can sometimes not be easy to transfer to another Fed position, and even if you can, comparing usajobs with contractor sites, there are WAY less openings in my field and location (we're talking single digits compared with hundreds). Compare that with the contracting side, if after I get hired and end up not liking the position, I can be out of there and in a new job within a month or so.

Reading all the posts like this, the vast majority say working on the Fed side is better, so I'm wondering if there's anybody else who works in this sector who decided the contractor side was a better fit?