FreeBSD guests downloaded their test binaries from vnet's
files.tailscale VIP at roughly 250 kB/s in CI, and transfers sometimes
wedged outright for many minutes, which is why TestSubnetRouterFreeBSD
timed out in about a third of its runs. Locally the same path moves
data at 100+ MB/s, so the problem was never CPU; it was TCP behavior
under two independent constraints, both diagnosed with a new
throughput harness (TestVnetPerfFreeBSDDownload), a VNET_TCP_DEBUG
endpoint sampler, and pcaps:
First, throughput is capped at receive-window/RTT. FreeBSD starts its
receive window at 64 kB and autoscales it in slow 16 kB steps, and on
an oversubscribed CI runner the effective RTT of the userspace vnet
data path reaches hundreds of milliseconds, giving almost exactly the
observed 250 kB/s. Fix: raise the FreeBSD guest's TCP buffer sysctls
in cloud-init before the downloads, and raise netstack's receive
buffer sizing for the reverse (upload) direction.
Second, the outright wedge: when netstack bursts more data than the
QEMU socket plus the guest's virtio RX ring can absorb, a wide swath
of segments is dropped downstream of vnet, and netstack's loss
recovery then crawls, retransmitting one or two segments per 200 ms
RTO for minutes at a time (a 33 MB transfer was observed taking 526
seconds against an otherwise idle receiver). Rather than depending on
recovery from mass loss, make the path effectively lossless by keeping
the maximum in-flight data (the 1 MB netstack send buffer) below the
downstream buffering: grow the guests' virtio RX rings from 256 to
1024 descriptors, enlarge the vnet-QEMU unix socket buffers, and grow
the netstack link endpoint queue from 512 to 4096 packets so a send
burst can't overflow it.
Also fixed along the way, found while chasing the above:
* pcapWriter fsync'd after every packet, serializing all traffic
behind disk writes when a test enables pcap; a pcap-enabled run
was capped at about 290 kB/s. Keep the per-packet Flush but drop
the per-packet fsync.
* Traffic originating from vnet's own netstack (control plane,
DERP, file servers) bypassed conditionedWrite, so SetLatency and
SetPacketLoss silently didn't apply to it.
* writeEthernetFrameToVM held one global mutex (and a shared
scratch buffer) across writes to all VMs, so one guest slow to
drain its socket stalled traffic to every VM on the server. The
write lock is now per-VM-connection.
TestSubnetRouterFreeBSD now passes locally in 31s (down from 4.5
minutes), still passes with the vnet simulating a 100 ms RTT
(downloads at 2-8 MB/s, previously 250-600 kB/s), and passes in 65s
with KVM disabled while pinned to two host CPUs, a harsher environment
than the CI runners. The benchmark test is opt-in via
--run-perf-tests (in addition to --run-vm-tests) so CI doesn't spend
a matrix job re-measuring it on every run. VMTEST_NO_KVM=1
forces TCG for reproducing slow-host behavior.
Fixestailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I1a7945a7e9c7d083b0ea2a3530eda0e9757dff18
This file was never truly necessary and has never actually been used in
the history of Tailscale's open source releases.
A Brief History of AUTHORS files
---
The AUTHORS file was a pattern developed at Google, originally for
Chromium, then adopted by Go and a bunch of other projects. The problem
was that Chromium originally had a copyright line only recognizing
Google as the copyright holder. Because Google (and most open source
projects) do not require copyright assignemnt for contributions, each
contributor maintains their copyright. Some large corporate contributors
then tried to add their own name to the copyright line in the LICENSE
file or in file headers. This quickly becomes unwieldy, and puts a
tremendous burden on anyone building on top of Chromium, since the
license requires that they keep all copyright lines intact.
The compromise was to create an AUTHORS file that would list all of the
copyright holders. The LICENSE file and source file headers would then
include that list by reference, listing the copyright holder as "The
Chromium Authors".
This also become cumbersome to simply keep the file up to date with a
high rate of new contributors. Plus it's not always obvious who the
copyright holder is. Sometimes it is the individual making the
contribution, but many times it may be their employer. There is no way
for the proejct maintainer to know.
Eventually, Google changed their policy to no longer recommend trying to
keep the AUTHORS file up to date proactively, and instead to only add to
it when requested: https://opensource.google/docs/releasing/authors.
They are also clear that:
> Adding contributors to the AUTHORS file is entirely within the
> project's discretion and has no implications for copyright ownership.
It was primarily added to appease a small number of large contributors
that insisted that they be recognized as copyright holders (which was
entirely their right to do). But it's not truly necessary, and not even
the most accurate way of identifying contributors and/or copyright
holders.
In practice, we've never added anyone to our AUTHORS file. It only lists
Tailscale, so it's not really serving any purpose. It also causes
confusion because Tailscalars put the "Tailscale Inc & AUTHORS" header
in other open source repos which don't actually have an AUTHORS file, so
it's ambiguous what that means.
Instead, we just acknowledge that the contributors to Tailscale (whoever
they are) are copyright holders for their individual contributions. We
also have the benefit of using the DCO (developercertificate.org) which
provides some additional certification of their right to make the
contribution.
The source file changes were purely mechanical with:
git ls-files | xargs sed -i -e 's/\(Tailscale Inc &\) AUTHORS/\1 contributors/g'
Updates #cleanup
Change-Id: Ia101a4a3005adb9118051b3416f5a64a4a45987d
Signed-off-by: Will Norris <will@tailscale.com>
Brad Fitzpatrick
Will Norris
Maisem Ali