Trex packet generator ARP issue

This is my setup to benchmark my server https://ibb.co/0KWwT8Z

Trex version v2.88

Trex server MAC

ens2f0: f8:f2:1e:5c:bf:78 ens2f1: f8:f2:1e:5c:bf:79 

DUT server MAC

eth0: fa:16:3e:1c:be:82 eth1: fa:16:3e:d5:37:b1 

trex_cfg.yaml

# cat /etc/trex_cfg.yaml ### Config file generated by dpdk_setup_ports.py ### - version: 2 #interfaces: ['07:00.0', '07:00.1'] interfaces: ['ens2f0', 'ens2f1'] port_info: - ip: 10.66.0.100 default_gw: 10.66.1.33 - ip: 10.68.0.100 default_gw: 10.68.3.217 platform: master_thread_id: 0 latency_thread_id: 8 dual_if: - socket: 0 threads: [1,2,3,4,5,6,7,16,17,18,19,20,21,22,23] 

My scenario file

- duration : 5 generator : distribution : "seq" clients_start : "16.0.0.1" clients_end : "16.0.0.255" servers_start : "48.0.0.1" servers_end : "48.0.255.255" clients_per_gb : 10 min_clients : 10 dual_port_mask : "1.0.0.0" tcp_aging : 0 udp_aging : 0 cap_info : - name: cap2/rtp_160k.pcap cps : 10.0 ipg : 10000 rtt : 10000 w : 4 - name: cap2/rtp_250k_rtp_only_1.pcap cps : 10.0 ipg : 10000 rtt : 10000 w : 4 - name: cap2/rtp_250k_rtp_only_2.pcap cps : 10.0 ipg : 10000 rtt : 10000 w : 4 

Run command

./t-rex-64 --cfg /etc/trex_cfg.yaml -f /tmp/sfr.yaml -d 10 -m 1 -c 1 -k 2 --nc 

In Test i am seeing Trex sending UDP stream but RX end not receiving any data and result is 100% packet drops. when i run tcpdump on my DUT device i can see its sending UDP stream to Trex RX end using correct MAC address. fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79

DUT server tcpdump

 16.0.0.17.58005 > 48.0.0.17.1024: [no cksum] UDP, length 1231 15:38:34.237250 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype IPv4 (0x0800), length 378: (tos 0x0, ttl 125, id 11822, offset 0, flags [none], proto UDP (17), length 364) 16.0.0.18.9873 > 48.0.0.18.1026: [no cksum] UDP, length 336 15:38:34.237260 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype IPv4 (0x0800), length 1492: (tos 0x0, ttl 125, id 11852, offset 0, flags [none], proto UDP (17), length 1478) 16.0.0.15.23195 > 48.0.0.15.1026: [no cksum] UDP, length 1450 15:38:34.237266 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype IPv4 (0x0800), length 434: (tos 0x0, ttl 125, id 11832, offset 0, flags [none], proto UDP (17), length 420) 

As soon as test start i am seeing correct ARP entry mac-ip but in 2 second it get incomplete even my test is running and keep sending data.

[root@foo-2 ~]# arp -a | grep 10.68.0.100 ? (10.68.0.100) at f8:f2:1e:5c:bf:79 [ether] on eth1 [root@foo-2 ~]# arp -a | grep 10.68.0.100 ? (10.68.0.100) at <incomplete> on eth1 

Any idea why ARP getting incomplete in 2 second?

​

UPDATE:

When i run tcpdump on DUT server then i found Trex RX side interface ask for ARP and DUI replied back but when DUT ask for ask then Trex not replying back (10.68.3.217 is eth1 IP of DUT server)

who is 01:03:05:07:09:00 and 01:03:05:07:09:01 ?

[root@foo-2 ~]# tcpdump -i eth1 -nn arp -e -vv tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 16:03:26.527165 f8:f2:1e:5c:bf:79 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 (01:03:05:07:09:00) tell 10.68.0.100, length 46 16:03:26.527192 f8:f2:1e:5c:bf:79 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.3.217 (01:03:05:07:09:01) tell 10.68.0.100, length 46 16:03:26.527201 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.68.3.217 is-at fa:16:3e:d5:37:b1, length 28 16:03:32.107249 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28 16:03:33.131245 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28 16:03:34.155245 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28 16:03:35.183759 fa:16:3e:d5:37:b1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28 16:03:36.203260 fa:16:3e:d5:37:b1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28 16:03:37.227240 fa:16:3e:d5:37:b1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28 

​