webnet/tailscale
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

net/batching: implement NodeAttrNeverGSOEqualTail

Browse Source 
This NodeCapability works around the UDP GSO bugs introduced by
torvalds/linux@b10b446 (v7.0-rc1). These bugs were later fixed by
torvalds/linux@78effd8 and torvalds/linux@5f17ae0 (v7.1-rc5). These
Linux kernel bugs cause mangled UDP headers and UDP checksums, resulting
in high levels of packet loss.

The aforementioned bugs have already made their way downstream into
various distros, e.g. Ubuntu 26.04 LTS. Impacted users are now dealing
with poor UDP performance in tailscaled, and in any other software that
makes use of UDP GSO.

Not all users of the affected kernels are impacted as the relevant
kernel code path sits between kernel and netdev driver, and behaviors
vary by driver/device capability.

We cannot detect impact at runtime, as this would require gathering all
netdevs, and performing loopback tests. This is invasive and in many
cases impossible.

So, we are left to choose between disabling UDP GSO for all users on
affected kernels, whether they experience real impact or not, or try
and work around the bugs. Disabling UDP GSO for a user that is not
impacted can cut max throughput in half, and consume more CPU cycles.

This commit attempts to workaround the bugs by avoiding UDP GSO when
batches are small, and injecting a 1-byte sentinel tail payload when
they are large. This tail payload is smaller than "GSO size", which
sidesteps the primary trigger of all fragments in a batch being
equal in length.

The end result is slightly increased payload and packet overhead, but
functional UDP GSO for all Linux 7.0-7.1.4 users, regardless of
netdev/driver.

Updates #19777

Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit is contained in:
Jordan Whited
2026-05-29 11:36:35 -07:00
committed by Jordan Whited
parent 7355116c05
commit 8b58bd6c64
4 changed files with 234 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tailcfg/tailcfg.go
+11 -1
View File
@@ -187,7 +187,8 @@ type CapabilityVersion int
// - 138: 2026-03-31: can handle C2N /debug/tka.
// - 139: 2026-05-22: Client understands [NodeAttrEmitRuntimeMetrics]
// - 140: 2026-05-27: Client understands [NodeAttrDisableUDPGRO], [NodeAttrDisableUDPGSO], [NodeAttrDisableTUNUDPGRO], [NodeAttrDisableTUNTCPGRO]
const CurrentCapabilityVersion CapabilityVersion = 140
// - 141: 2026-05-28: Client understands [NodeAttrNeverGSOEqualTail]
const CurrentCapabilityVersion CapabilityVersion = 141
// ID is an integer ID for a user, node, or login allocated by the
// control plane.
@@ -2830,6 +2831,15 @@ const (
// Currently only consulted on Linux; may apply to other platforms as they
// gain TUN TCP GRO support.
NodeAttrDisableTUNTCPGRO NodeCapability = "disable-tun-tcp-gro"
// NodeAttrNeverGSOEqualTail enables a sentinel-tail workaround in the
// underlay UDP packet TX path on Linux. Applies to magicsock and peer relay
// UDP sockets. The workaround avoids emitting UDP GSO batches whose
// fragments are all equal in length, at a small payload and packet overhead
// cost. It exists so control can mitigate kernel regressions that mangle
// UDP headers or checksums for equal-length GSO batches, without requiring
// a client release. See https://github.com/tailscale/tailscale/issues/19777.
NodeAttrNeverGSOEqualTail NodeCapability = "never-gso-equal-tail"
)
const (