Tuesday, June 19, 2018

DC network with combined Nexus and Catalyst

Some context first:

I have designed my first small datacenter in the past year. I have put this over multiple revisions and looking at a lot of different options and design structures. I have to design a network that can scale easily with more racks but will start small (2 racks). I designed it using a spine-leaf structure, where everything was using Catalyst (as oppose to Nexus). I used Catalyst for multiple reasons but the main were:

  • It is easy to get knowledge for these switches when necessary (we are a small company which doesn't have a lot of financial resources).
  • We could use cheaper leaf switches, where just a lot of 1Gbit/s ethernet interfaces are required.
  • We really didn't need the advanced features of Nexus anyway, so why pay for it...

OOB would use Catalyst as well and would be separated on the firewall.

To allow for easy automation and keep the network simple (since there would only be eight switches, including the two OOB switches) I would solely use Catalyst.

After about a year of research, planning, designing, testing and talking to Cisco Engineers and other (third party) consultants (I really wanted to do this right) the main office (I work at a branch office) stept in with their architect. They were never involved in this process, but knew it was happening. (I guess they never cared about it.) The architect started looking at my design and overhauled it completely. I was overwhelmed and we had a heated discussion and after a while we had an initial agreement that the spine switches would become Nexus and the leaf switches would stay Catalyst.

After a good night sleep and reviewing his solution, I saw that it went against the goals I had with the design: making it simpler and cheaper. So I send an e-mail to the architect the next day saying I didn't think this was the right solution and we should go back to the original design. With a small change to allow for more ports on the spine, same type of switch, just a different version with more ports. Since this was one of his reasons behind the Nexus choice, to allow for more ports on the spine (10Gbit/s per port at the spine).

Fast forward about 2 weeks, complete with full radio silence about this from the main office, they told me everything was a GO. So I asked the main office what the network components were. These turned out to be the Nexus spine and the Catalyst leaf. This in turn frustrated me and we didn't have a good meeting, to say the least. The solution he proposed never came up to me and neither was in any whitepapers I read or was mentioned by any consultant (from Cisco or third party) I talked to. The architect from the main office was clearly sure of this. His arguments were that it would be cheaper compared to my alternative solution with the switch with more ports and that he has done it in all datacenters, so it needs to be coherent. He has implemented this in two other datacenters (both are about 3-4 racks each) that are being managed by the main office. I am managing the datacenter in the branch office.

So now you kinda know about the backstory, I have some questions for everyone here:

  • Is it normal to design a small network/datacenter like this with Nexus and Catalyst switches combined? If so, what are your experiences and your reasoning behind it? I am very curious and want to learn from it.
  • For the Enterprise/Campus network guys out there, is this normal in those kinds of networks?

Hardware selected:

Original Idea:

  • WS-C3850-12XS (12 SFP+) * 2 (Spine + Stacked + Powerstack)
  • 2960X-48TD-L * 4 (Leaf + 2 Stacks, 1 per rack)
  • WS-C2960L-24TS-LL * 2 (OOB, 1 per rack)

Architects Idea:

  • Nexus 3524-X (24 SFP+) * 2 (Spine + Stacked)
  • 2960X-48TD-L * 4 (Leaf + 2 Stacks, 1 per rack)
  • WS-C2960L-24TS-LL * 2 (OOB, 1 per rack)

My revised answer:

  • WS-C3850-24XS (24 SFP+) * 2 (Spine + Stacked + Powerstack)
  • 2960X-48TD-L * 4 (Leaf + 2 Stacks, 1 per rack)
  • WS-C2960L-24TS-LL * 2 (OOB, 1 per rack)

I would have preferred 9K, but I couldn't find one with 24 SFP+ ports. 9400 would be overkill I think.

My idea was that the 3850 would be better compared to the Nexus one, since it has a higher backplane capacity and the same amount of ports. Also, it would make all the switches the same and thus easier to manage.

Kudos if you have made it to this point and I wanna thank you for reading it all ;)



No comments:

Post a Comment