Packet delay on local interface

Hi guys,

I am working on a very strange performance VOIP releated issue: I can see packet being delayed for 1-3 seconds on the local interface of my cloud pbx (Debian 9) ! before ! the generated packet is leaving the interface towards my clients (doesn't matter if over a VPN or external).

I can also replicate this issue over iperf3 quite good, it looks exactly like on my RTP packets. It absoulty doesn't matter if the packet is routed directly over my WAN (to a D-NAT Port Mapping) or over VPN-Tunnel.

This is what the delay looks like in Wireshark (right part is the Time Delta):

https://i.imgur.com/gkMgsPn.png

Doesn't matter if I capture this on tshark or tcpdump - always the same. Just to be clear once again: I can see this delay on generated packets at my PBX! These packets are directly from my PBX to my phones over the Internet or VPN. I should never see this kind of delay on my own interface while capturing those packets (yes this delay will also happen if i don't capture anything).

First I though this might be a performance issue but I can't see any of that on my VM: CPU is fine, RAM is fine, IO-Load is perfectly fine, there is nothing else running. The VM is bored.

​

This delay results in a bunch of packets (not only one) being delivered one second later. This doesn't mean jitter this is complete silence in an RTP stream.

​

Have you guys seen anything like that before?

Of course i did the major things like reinstall OS, patch etc.

​

Any Input is welcome!

Kind regards