Issues with Cisco 9500-48Y4C in StackWise-Virtual configuration

Hi All. Sorry for the long post.

I've hit a weird problem with Cisco 9500-48Y4C switches when in a SWV stack.

The switches are running IOS XE 17.3.3 (The version recommended by Cisco).

After stacking the switches and reloading, the switches boot, form the stack, then shut all of their interfaces down, including the SVL and DAD links. The stack is up though and the standby switch knows it's a standby (console disabled, etc). Reloading the Active also reboots the standby switch, so they're definitely communicating. All interfaces show not connected in show int status, and all of the transceivers show as unknown, including the SVL and DAD links:

Port Name Status Vlan Duplex Speed Type

Twe1/0/47 DAD Link notconnect 4094 auto auto unknown

Twe1/0/48 DAD Link notconnect 4094 auto auto unknown

Hu1/0/51 SVL notconnect 4094 auto auto unknown

Hu1/0/52 SVL notconnect 4094 auto auto unknown

Twe2/0/47 DAD Link notconnect 4094 auto auto unknown

Twe2/0/48 DAD Link notconnect 4094 auto auto unknown

Hu2/0/51 SVL notconnect 4094 auto auto unknown

Hu2/0/52 SVL notconnect 4094 auto auto unknown

After 40 minutes, all of the transceiver types are suddenly known and the DAD links come up. After another 40 minutes the SVL Links come up, along with all other interfaces on the switches.

Port Name Status Vlan Duplex Speed Type

Twe1/0/47 connected 4094 full 10G SFP-10GBase-CU3M

Twe1/0/48 connected 4094 full 10G SFP-10GBase-CU3M

Hu1/0/51 connected 4094 full 100G QSFP 100G CU2M

Hu1/0/52 connected 4094 full 100G QSFP 100G CU2M

Twe2/0/47 connected 4094 full 10G SFP-10GBase-CU3M

Twe2/0/48 connected 4094 full 10G SFP-10GBase-CU3M

Hu2/0/49 notconnect 1 full 100G QSFP 100G LR4

Hu2/0/50 notconnect 1 full 100G QSFP 100G LR4

Hu2/0/51 connected 4094 full 100G QSFP 100G CU2M

Hu2/0/52 connected 4094 full 100G QSFP 100G CU2M

Has anybody seen this before, or know where the hell I should start troubleshooting? There are no errors thrown in syslog and no logs pointing to a service starting or anything before the links come up, just the link up messages. I'm going to raise a TAC case as well, but I'm hoping somebody here might have an idea what it might be.

The switches are brand new. I have 4 of them, configured in two stacks, both stacks doing the same thing. They shipped with IOS XE 16.12.4 which did exactly the same. I updated one stack using ISSU and the other manually to 17.3.3 and both methods updated the standby switch as expected.

I've included some show outputs and config that might help tell the whole story below.

show switch output:

Switch/Stack Mac Address : 0000.0000.0000 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role Mac Address Priority Version State

-------------------------------------------------------------------------------------

*1 Active 0000.0000.00c0 5 V02 Ready

2 Standby 0000.0000.0060 1 V02 Ready

show Stackwise-virtual output:

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Enabled

Domain Number : 1

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1 1 HundredGigE1/0/51

HundredGigE1/0/52

2 1 HundredGigE2/0/51

HundredGigE2/0/52

show Stackwise-virtual link output:

-----------------------------------------------

Switch SVL Ports Link-Status Protocol-Status

------ --- ----- ----------- ---------------

1 1 HundredGigE1/0/51 D R

HundredGigE1/0/52 D R

2 1 HundredGigE2/0/51 D R

HundredGigE2/0/52 D R

show Stackwise-virtual dual-active-detection output:

In dual-active recovery mode: No

Recovery Reload: Enabled

Dual-Active-Detection Configuration:

-------------------------------------

Switch Dad port Status

------ ------------ ---------

1 TwentyFiveGigE1/0/47 down

TwentyFiveGigE1/0/48 down

2 TwentyFiveGigE2/0/47 down

TwentyFiveGigE2/0/48 down

show rommvar output:

Switch 1

SWITCH_PRIORITY="5"

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="1"

D_STACK_DISTR_STACK_LINK1="Hu1/0/51,Hu1/0/52,"

D_STACK_DISTR_STACK_LINK2=""

D_STACK_DAD="Twe1/0/47,Twe1/0/48,"

Switch 2

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="1"

D_STACK_DISTR_STACK_LINK1="Hu2/0/51,Hu2/0/52,"

D_STACK_DISTR_STACK_LINK2=""

The relevant config from each stack:

stackwise-virtual

domain 1

!

interface TwentyFiveGigE1/0/47

stackwise-virtual dual-active-detection

description DAD Link

!

interface TwentyFiveGigE1/0/48

stackwise-virtual dual-active-detection

description DAD Link

!

interface HundredGigE1/0/51

stackwise-virtual link 1

description SVL

!

interface HundredGigE1/0/52

stackwise-virtual link 1

description SVL

!

interface TwentyFiveGigE2/0/47

stackwise-virtual dual-active-detection

description DAD Link

!

interface TwentyFiveGigE2/0/48

stackwise-virtual dual-active-detection

description DAD Link

!

interface HundredGigE2/0/51

stackwise-virtual link 1

description SVL

!

interface HundredGigE2/0/52

stackwise-virtual link 1

description SVL