Can't access switch from IP after adding an IP to a different VLAN

So, I started a new job recently. This is my first one as THE network admin for a company. My previous networking jobs have all been grunt level. I am starting to figure out what is going on with the network, but this is causing me headaches, and I was hoping someone might be able to point me in the right direction of where to look.

As I was trying to figure out something else, I come across a switch (which I will call S1) I can't SSH into by using the documented IP address. Get a PuTTY Fatal Error: "Network error: Software caused connection abort." I can console in just fine, and its right around the corner, so its not a huge problem. Also figure out I can SSH into S1 if I am connected to it directly. (The drop at my desk has a switch between it and S1, with a trunk line between the two)

Then, I am throwing some commands at a different switch (S2, which is the one between me and S1), and I lose access to it. Same PuTTY error. Come to realize that the command that lost me access was putting an IP on the VLAN (VlanA) that the port my computer is on. The IP I had been using to access it was for a management VLAN (VlanB). Remove the IP from VlanA, can once again SSH into S2 using the IP for VlanB. Put the IP back on VlanA, have to use the IP for VlanA to access it.

Realizing that, I look at the Running-Config I had output to text when I was consoled into S1, and find that VlanA has an IP attached to it. I use that, and am able to SSH in.

I am also having issues accessing switches at other sites using the documented IP, so I am suspicious that this is a related issue.

For reference: My Computer ----VlanA---> S2 ------Trunk---> S1 -------Trunk-----> Firewall/router

In my experience so far, simply adding an IP to the VLAN I am directly on shouldn't have stopped me from using the IP of the other VLAN to SSH into the switch. Either I should have access using that IP, or not due to ACLs, routing, or Firewall rules, but I am new to this level of Network administration, so that could be wrong.

Any tips/clues/pointers/educated questions would be appreciated.