Hey all,
I'm having issues with multicast which I've been pulling my hair out over for the past few weeks.
I've got a Cisco 92160YC-X switch running PIM-SM on Vlan 200, with all multicast subscribers/publishers also in Vlan 200.
Multicast traffic functions for a while, and then stops. During the period when it stops, I can see IGMP query messages sent from the SVI IP address on Vlan 200, and IGMP reports from each host. I see this on TCPdumps on the hosts, as well as igmp debugging on the switch.
Upon issuing a 'clear ip mroute *' multicast traffic begins to flow again.
I'm testing with a python multicast script from Redhat - https://access.redhat.com/articles/22304, as well as an multicast test application using multicast group 239.255.0.1.
This only appears to occur with multicast group 239.255.0.1, but I'm not 100% on this yet. The test application uses 239.255.0.1, and the python script uses a definable address. The python script continues to work on another group - 229.255.0.1 when 239.255.0.1 stops working.
Here's the relevant sanitised configuration.
Switch configuration: Switch(config)# show run int vl200 interface Vlan200 no shutdown no ip redirects ip address 10.1.1.2/24 ip pim sparse-mode ip pim dr-priority 100 Switch(config)# show run | i rp-address ip pim rp-address 10.1.1.2 group-list 224.0.0.0/4 override
Hosts used for testing: Vlan 200 IP hosts: 10.1.1.12, 10.1.1.13 & 10.1.1.14
Interfaces for the hosts, useful for IGMP snooping output below
Switch# show interface status | i Host Eth1/1 Host1 connected 200 full 10G SFP-H10GB-CU2M Eth1/2 Host2 connected 200 full 1000 1000base-T Eth1/3 Host3 connected 200 full 10G SFP-H10GB-CU2M Eth1/4 Host4 connected 200 full 10G SFP-H10GB-CU2M Eth1/5 Host5 connected 200 full 10G SFP-H10GB-CU2M Eth1/6 Host6 connected 200 full 10G SFP-H10GB-CU2M Eth1/7 Host7 connected 200 full 10G SFP-H10GB-CU2M
Host packet capture:
sudo tcpdump -v -n -i eno3 igmp 03:48:27.163568 IP (tos 0xc0, ttl 1, id 59596, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.1.1.2 > 224.0.0.1: igmp query v2 03:48:35.401867 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.13 > 239.255.0.1: igmp v2 report 239.255.0.1 03:48:27.164166 IP (tos 0xc0, ttl 1, id 59596, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.1.1.2 > 224.0.0.1: igmp query v2 03:48:27.271682 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.12 > 224.0.0.251: igmp v2 report 224.0.0.251 03:48:36.630537 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.12 > 239.255.0.1: igmp v2 report 239.255.0.1 03:48:27.164166 IP (tos 0xc0, ttl 1, id 59596, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.1.1.2 > 224.0.0.1: igmp query v2 03:48:27.271682 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.12 > 224.0.0.251: igmp v2 report 224.0.0.251 03:48:36.630537 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.12 > 239.255.0.1: igmp v2 report 239.255.0.1 03:48:27.164169 IP (tos 0xc0, ttl 1, id 59596, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.1.1.2 > 224.0.0.1: igmp query v2 03:48:29.913727 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.14 > 239.255.0.1: igmp v2 report 239.255.0.1
NXOS Bash tcpdump showing IGMP reports from the hosts coming back into VLAN 200
bash-4.2# tcpdump -v -n -i Vlan200 tcpdump: listening on Vlan200, link-type EN10MB (Ethernet), capture size 65535 bytes 06:36:41.706046 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.11 > 224.0.0.251: igmp v2 report 224.0.0.251 06:36:42.787077 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.16 > 239.255.0.1: igmp v2 report 239.255.0.1 06:36:44.415473 IP (tos 0xc0, ttl 1, id 4888, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.1.1.23 > 239.255.255.250: igmp v2 report 239.255.255.250 06:36:45.672004 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.1.1.14 > 239.255.0.1: igmp v2 report 239.255.0.1 06:36:45.696663 IP (tos 0xc0, ttl 1, id 44482, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.1.1.22 > 239.255.255.250: igmp v2 report 239.255.255.250 IGMP snooping groups: Switch(config)# show ip igmp snooping groups Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port Vlan Group Address Ver Type Port list 1 */* - R Eth1/47 200 */* - R Vlan200 200 239.255.0.1 v2 D Eth1/7 Eth1/3 Eth1/4 Eth1/5 Eth1/6 200 239.255.255.250 v2 D Eth1/34 Eth1/35 Eth1/36 Eth1/37 Eth1/38
RPF checks are passing as expected. There is only one path to each host.
Switch(config)# show ip mroute summary rpf-failed IP Multicast Routing Table for VRF "default" Route Statistics unavailable - only liveness detected Total number of routes: 12 Total number of (*,G) routes: 2 Total number of (S,G) routes: 9 Total number of (*,G-prefix) routes: 1 Group count: 2, rough average sources per group: 4.5 Group: 232.0.0.0/8, Source count: 0 Source packets bytes aps pps bit-rate oifs (*,G) 0 0 0 0 0.000 bps 0 RPF Failed (pkts/bytes): 0/0 Group: 239.255.0.1/32, Source count: 4 Source packets bytes aps pps bit-rate oifs (*,G) 0 0 0 0 0.000 bps 1 RPF Failed (pkts/bytes): 0/0 10.1.1.12 4261 218096 51 0 27.200 bps 1 RPF Failed (pkts/bytes): 0/0 10.1.1.13 5171 264506 51 0 27.200 bps 1 RPF Failed (pkts/bytes): 0/0 10.1.1.14 64 5343 83 0 27.200 bps 1 RPF Failed (pkts/bytes): 0/0 10.1.1.15 42 4221 100 0 20.400 bps 1 RPF Failed (pkts/bytes): 0/0
The multicast routing table when hosts aren't receiving multicast traffic.
Switch# show ip mroute IP Multicast Routing Table for VRF "default" (*, 232.0.0.0/8), uptime: 05:02:35, pim ip Incoming interface: Null, RPF nbr: 0.0.0.0 Outgoing interface list: (count: 0) (*, 239.255.0.1/32), uptime: 05:02:29, igmp ip pim Incoming interface: Vlan200, RPF nbr: 10.1.1.2 Outgoing interface list: (count: 1) Vlan200, uptime: 05:02:29, igmp, (RPF) (10.1.1.12/32, 239.255.0.1/32), uptime: 05:02:32, ip pim mrib Incoming interface: Vlan200, RPF nbr: 10.1.1.12, internal Outgoing interface list: (count: 1) Vlan200, uptime: 05:02:29, mrib, (RPF) (10.1.1.13/32, 239.255.0.1/32), uptime: 05:02:35, ip pim mrib Incoming interface: Vlan200, RPF nbr: 10.1.1.13, internal Outgoing interface list: (count: 1) Vlan200, uptime: 05:02:29, mrib, (RPF) (10.1.1.14/32, 239.255.0.1/32), uptime: 05:02:13, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.14, internal Outgoing interface list: (count: 1) Vlan200, uptime: 05:02:13, mrib, (RPF) (10.1.1.16/32, 239.255.0.1/32), uptime: 05:02:35, ip pim mrib Incoming interface: Vlan200, RPF nbr: 10.1.1.16, internal Outgoing interface list: (count: 1) Vlan200, uptime: 05:02:29, mrib, (RPF) (*, 239.255.255.250/32), uptime: 05:02:34, igmp ip pim Incoming interface: Vlan200, RPF nbr: 10.1.1.2 Outgoing interface list: (count: 1) Vlan200, uptime: 05:02:34, igmp, (RPF) (10.1.1.22/32, 239.255.255.250/32), uptime: 04:54:28, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.22, internal Outgoing interface list: (count: 1) Vlan200, uptime: 04:54:28, mrib, (RPF) (10.1.1.23/32, 239.255.255.250/32), uptime: 04:58:22, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.23, internal Outgoing interface list: (count: 1) Vlan200, uptime: 04:58:22, mrib, (RPF) (10.1.1.24/32, 239.255.255.250/32), uptime: 04:55:47, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.24, internal Outgoing interface list: (count: 1) Vlan200, uptime: 04:55:47, mrib, (RPF) (10.1.1.25/32, 239.255.255.250/32), uptime: 04:59:10, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.25, internal Outgoing interface list: (count: 1) Vlan200, uptime: 04:59:10, mrib, (RPF) (10.1.1.26/32, 239.255.255.250/32), uptime: 04:57:27, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.26, internal Outgoing interface list: (count: 1) Vlan200, uptime: 04:57:27, mrib, (RPF) Switch# show ip igmp snooping groups Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port Vlan Group Address Ver Type Port list 1 */* - R Eth1/47 200 */* - R Vlan200 200 239.255.0.1 v2 D Eth1/7 Eth1/4 Eth1/3 Eth1/5 200 239.255.255.250 v2 D Eth1/37 Eth1/36 Eth1/35 Eth1/34 Eth1/38 Switch# Switch# clear ip mroute * Switch# show ip mroute IP Multicast Routing Table for VRF "default" (*, 232.0.0.0/8), uptime: 00:00:14, pim ip Incoming interface: Null, RPF nbr: 0.0.0.0 Outgoing interface list: (count: 0) (*, 239.255.0.1/32), uptime: 00:00:12, igmp ip pim Incoming interface: Vlan200, RPF nbr: 10.1.1.2 Outgoing interface list: (count: 1) Vlan200, uptime: 00:00:12, igmp, (RPF) (10.1.1.12/32, 239.255.0.1/32), uptime: 00:00:07, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.12, internal Outgoing interface list: (count: 1) Vlan200, uptime: 00:00:07, mrib, (RPF) (10.1.1.13/32, 239.255.0.1/32), uptime: 00:00:14, ip pim mrib Incoming interface: Vlan200, RPF nbr: 10.1.1.13, internal Outgoing interface list: (count: 1) Vlan200, uptime: 00:00:12, mrib, (RPF) (10.1.1.14/32, 239.255.0.1/32), uptime: 00:00:08, ip mrib pim Incoming interface: Vlan200, RPF nbr: 10.1.1.14, internal Outgoing interface list: (count: 1) Vlan200, uptime: 00:00:08, mrib, (RPF) (10.1.1.16/32, 239.255.0.1/32), uptime: 00:00:14, ip pim mrib Incoming interface: Vlan200, RPF nbr: 10.1.1.16, internal Outgoing interface list: (count: 1) Vlan200, uptime: 00:00:12, mrib, (RPF) (*, 239.255.255.250/32), uptime: 00:00:14, igmp ip pim Incoming interface: Vlan200, RPF nbr: 10.1.1.2 Outgoing interface list: (count: 1) Vlan200, uptime: 00:00:14, igmp, (RPF) Switch# show ip igmp snooping groups Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port Vlan Group Address Ver Type Port list 1 */* - R Eth1/47 200 */* - R Vlan200 200 239.255.0.1 v2 D Eth1/4 Eth1/5 Eth1/7 Eth1/3 200 239.255.255.250 v2 D Eth1/35 Eth1/36 Eth1/37 Eth1/34 Eth1/38
No comments:
Post a Comment