Sanity-Check for network upgrade

Hey Guys, I'm quite new to the field of networking, especially in the "not consumer" segment but I'm responsible for the infrastructure of a student dorm with ~100-200 Students. We are completely self organized and apparently I have the most knowledge of networks, so here I am.

We've got a bit of money and want to upgrade our old hardware as some of it is decades old. Because I learned most of what I know by myself and never received any form of training I would like to get some feedback on my upgrade plans.

​

First our current setup:

We have a 1GBit Fiber connection to our ISP that is connected to our old Router/Firewall/Web-/Mailserver. This thing is ancient and nobody really knows who set it up or how. It was used as our one server for everything and is therefore cluttered like crazy. We have moved pretty much all services away except DHCP server and Firewall. The Firewall is an iptables script which also does a lot of NATing.

The next hop is our Core switch: A Cisco Catalyst 3650 series device. Connected to that are our somewhat new Servers, which are running Proxmox with all services (Web, Mail,...) virtualized and a Ceph instance. They are connected with a 1Gbit Link to the Core switch. There is also a second switch that is only connected to the servers with 3 Cables each. I think it was supposed to be an aggregated Link so that they can communicate with 3Gig between themselves, but I don't know if that was configured correctly. Anyway Ceph is really slow and everything is pointing to bad connectivity between the nodes.

There are 2 cables from our server room to each floor. They are labeled as CAT-5 but have shielding and foil, so they are probably more like CAT-6 or 7 but again decades old. They can transmit gigabit fine however. One of those cables is used to connect an old Netgear 100Mbit switch on each floor to the core switch. The end users and Wi-Fi APs are connected to these Netgear devices.

​

Now my Upgrade-Plan:

We do not have much money and construction work is tedious due to fire safety regulations, so we only want to upgrade the hardware. So no new cables through walls or anything like that.

We can upgrade our Fiber connection to 10GBit. We basically only need to buy a new SFP+ module/NIC.

I really want to replace the old Router and I am thinking of building a firewall that is capable of 10GBit/s throughput. I wanted to use pfSense or OpenSense for it and some secondhand server hardware with a dual SFP+ NIC.

The Core switch should be replaces with a device with 2 10Gbit Links. One for the uplink to the firewall. I want to put a 10Gig NIC in every Server and connect them to a 10Gig Switch which is then connected to the second 10G Port of the new Core Switch. This should speed up Ceph. I'm thinking of either a Netgear switch with RJ-45 Ports or one from Mikrotik with SFP+ and use Copper DACs to the servers. Not sure which.

We got a really good deal on more Cisco Catalyst 3650 switches that we want to put in every Floor. Then we want to use both cables to have an aggregated 2 GBit link each.

​

That's my plan so far but I have some specific questions:

1. We have an old Cisco Catalyst 2960 S Switch lying around that has the 2 10Gbit ports we want for our new core switch. If I understand it correctly it is no longer supported by Cisco though. Can we still use? What exactly means end of life for Cisco devices? They don't get any updates, but is that so bad? Or do we need to buy a new device?
2. What kind of hardware do we need to achieve 10Gig throughput on a firewall? Is that even possible without spending thousands of dollars? I could only find firewall devices for 1 Gig, hence why I want to build one myself.
3. Which OS would you recommend for the Firewall? I've heard that pfSense is quite good, but they apparently implemented Wireguard super fishy or something and now people say to use OpenSense?
4. Would you rather use ordinary Ethernet cables to connect the Servers to their new Switch or Copper DACs? Price is pretty similar and we need to buy new NICs for them anyway so either one would work.
5. If we use Copper DACs for SFP+ ports, we would need to get vendor specific modules on each end. We can buy these at fs.com I think? But which vendor do we choose for the Server side? Could we just use a mikrotik branded cable and the server NIC would accept that?

​

I would be really thankful for any advice or Feedback you could give me. The network should not be super complex as we have a high turnover rate of tenants and someone new has to maintain it like every 2-4 years. Some of this plan is probably overkill, but we have money that we do not need for anything else and as we are all students we want to use it as an opportunity to learn and get experience that might help in finding jobs in this field.