I have a home lab with two 8 core 2.4 ghz Xeon servers with two ConnectX 4 100gb single port cards and mellanox qsfp cable. I’m trying to make sure I’m getting maximum throughput between the machines using ib_send_bw and bidirectional I’m getting about 165 gbits, where I would expect closer to ~190. Is 165 more typical of this setup?
CPU Xeon (Haswell) 2.4 8 core v3 OS: Ubuntu 20.04 (kernel 5.4 generic / low latency) Driver/Firmware: OFED 5.0 Hardware: PCI-E 16x (verified with mlnx_tune and lspci), c states disabled, p states using intel Network: direct plugin MTU 4096 Test: ib_send_bw bidirectional, duration 10 sec, rdma
I have noticed that ib_send_bw seems to be single threaded? One machine has a single core maxed out. Is this interrupts? I’ve never had that issue with cards like these
I considered switching to CentOS 8.1. Had better luck with CentOS 7.x with enterprise networking stuff and Mellanox states the 20.04 Ubuntu driver is “beta”. I need eventually to have ZFS support, which Ubuntu just seems easier.
No comments:
Post a Comment