When UDP packets vanish: tracing silent drops on your own machine
A UDP stream was missing roughly a third of its datagrams. No switch alarms, no NIC errors—just gaps in the sequence numbers. The first instinct is to blame the network, yet in this case the culprit was the receiving machine itself. The packets arrived safely, but the host’s socket buffer overflowed and silently discarded them before the application could read them. Distinguishing host-side drops from network failures is critical, and it starts with a single kernel counter.
Where the trail leads: follow the receive path
On Linux the journey of a UDP datagram is straightforward: NIC → kernel socket receive buffer → recv() call. If your code doesn’t pull datagrams out fast enough, the buffer fills and the kernel drops the excess. The system counts these drops under “receive buffer errors,” visible in netstat -su or /proc/net/snmp. When RcvbufErrors climbs, the network did its job; the host simply couldn’t keep up. One glance at that counter can replace days of “is it the switch?” uncertainty.
Burst behavior trumps average throughput
Investigation revealed the socket’s default 208 KB buffer was overwhelmed by brief sender bursts. While average throughput looked healthy on every dashboard, millisecond-long surges filled the buffer faster than a single receive thread could empty it. The relevant metric wasn’t mean flow—it was peak burst versus drain rate. Raising the buffer and offloading processing helped, but the real fix began with faster draining.
Practical steps to stop the silent drops
Start by moving any heavy lifting out of the receive loop: copy the datagram into a queue and return immediately to recv(). Next, enlarge the socket buffer with setsockopt(SO_RCVBUF) and raise net.core.rmem_max so the kernel honors the change. For high-volume feeds, batch system calls with recvmmsg() to cut per-packet overhead. If one core still lags, SO_REUSEPORT lets multiple threads share the same port with separate buffers. Usually you need both a larger buffer and a faster drain.
Source: DEV Community. AI-assisted editorial synthesis — TechnoExpress.