Events/Troubleshooting/Attempted solutions so far -
Beginning of April - The ASA I inherited, running 9.9(1), started crashing due to bug CSCvi16029, so I updated it to 9.9(2)66, and also upgraded my ASDM to 7.12.1 and my AnyConnect to from 3.x.x to 4.8.02045 on April 8.
April 9 - Everyone except for my IT-coworker and I are unable to RDP into their workstations. They can connect the VPN and ping their workstations, but get that "Remote Desktop can't connect" error (this never happened before). I also cannot RDP via VPN into any other workstations or servers on the network besides my own, which isn't configured any differently in RDP settings than any of the other machines.
Discovered that RemoteVPN->DNS setting is pointing to old decom'd DC, so I fix setting and am able to reach payroll server until HR person tried to RDP into it and then everyone was locked out again except myself to my own workstation.
April 13 - Everyone's ability to RDP suddenly came back up, then dropped, then came back up again by the end of the day.
I discovered a No NAT rule that allows all trusted users access to all necessary internal VLANs, and the only "deny" ACLs I could find are blocking QUIC and "hostile traffic" which doesn't seem to have anything to do with RDP.
April 21 - RDP stopped working again for office staff, so I did a packet capture while unsuccessfully trying to RDP into the payroll server:
1: 14:52:58.060497 x.x.x.x.55749 > x.x.x.x.3389: S 3006513269:3006513269(0) win 8192 <mss 1366,nop,wscale 2,nop,nop,sackOK>
2: 14:53:01.058483 x.x.x.x.55749 > x.x.x.x.3389: S 3006513269:3006513269(0) win 8192 <mss 1366,nop,wscale 2,nop,nop,sackOK>
3: 14:53:07.058621 x.x.x.x.55749 > x.x.x.x.3389: S 3006513269:3006513269(0) win 8192 <mss 1366,nop,nop,sackOK>
Tried sh cap asp | inc x.x.x.x (payroll server ip) which showed no drops from the firewall.
April 22 - I noticed on the the Firewall Dashboard in ASDM that the workstations denying RDP access are listed at "Top 10 Protected Servers under SYN Attack" and sure enough, there are the ip addresses of the Business Officec computers plus the 3389 port #.
I then learned this, and lowered the TCPMSS to 1300 and everyone's RDP started working again for almost a week ...
April 28 - Discovered that the RemoteVPN->DNS setting keeps reverting back to the old decommissioned DC, so I sent this info to Cisco TAC.
Yesterday - the Business Manager and I were able to to RDP into everything, but other staff could not connect to their workstations. I asked Cisco TAC if the ASA has some kind of DNS mapping in its config that's causing the VPN/DNS settings to revert, so they looked at my "show tech" and noticed that a firewall-object-network object had been configured linking the old DC to the ip address which now belongs to the new DC. So, I fixed that to point to the new DC.
The Business Manager lost her ability to RDP around 2pmEST, so I did a packet trace to her machine, and am waiting to hear back fro Cisco still, and there's a Zoom Board Meeting on Tuesday that's breathing down my neck ...
Ingress Capture – RDP packet
17:11:44.716134 x.x.x.x.59777 > x.x.x.x.3389: S 3869117268:3869117268(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK>
17:11:47.709954 x.x.x.x.59777 > x.x.x.x.3389: S 3869117268:3869117268(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK>
17:11:53.709374 x.x.x.x.59777 > x.x.x.x.3389: S 3869117268:3869117268(0) win 8192 <mss 1300,nop,nop,sackOK>
Egress Capture – RDP Packets
17:14:25.581436 x.x.x.x.59788 > x.x.x.x.3389: S 2051428495:2051428495(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK>
17:14:28.578232 x.x.x.x.59788 > x.x.x.x.3389: S 2051428495:2051428495(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK>
17:14:34.579163 x.x.x.x.59788 > x.x.x.x.3389: S 2051428495:2051428495(0) win 65535 <mss 1300,nop,nop,sackOK>
No comments:
Post a Comment