The tailscale/gokrazy-kernel module was a fork of rtr7/kernel that
stalled at Linux 6.8.9 (July 2024). All of the kernel config options we
had added in that fork (ENA, Xen for EC2, virtio-mmio for qemu microvm,
virtio RNG, IPv6 policy routing, netlink diag, etc) are now present in
the gokrazy project's own gokrazy/kernel.amd64 module, which tracks
current kernel.org releases (Linux 7.1.3 as of this change) and is the
gokrazy project's supported kernel for x86_64 PCs and VMs.
Switch the tsapp and natlabapp images, the natlab VM tests, and the
CI workflow to gokrazy/kernel.amd64, drop the tailscale/gokrazy-kernel
dependency, and update gokrazy/kernel.arm64 to latest while here.
Verified with TestEasyEasy, TestJustIPv6, and TestTailscaleSSH in
tstest/natlab/vmtest with --run-vm-tests.
Updates #1866
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I90c3765a4e18f5609b4d77b51ac38d17c8e3688a
The Discover tests step runs under set -euo pipefail, so when every
Test function in a file is filtered out by the exclude regex (as with
vnetperf_test.go, whose TestVnetPerf* tests need special invocation),
the final grep -vE produces no output and exits 1, failing the whole
step. Tolerate empty results from both greps so such files simply
contribute no matrix entries. Same for a hypothetical future test
file containing only helpers and no Test functions.
Updates tailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If20c9a1de37e97b1553bd7d5559e2b876a45c19b
FreeBSD guests downloaded their test binaries from vnet's
files.tailscale VIP at roughly 250 kB/s in CI, and transfers sometimes
wedged outright for many minutes, which is why TestSubnetRouterFreeBSD
timed out in about a third of its runs. Locally the same path moves
data at 100+ MB/s, so the problem was never CPU; it was TCP behavior
under two independent constraints, both diagnosed with a new
throughput harness (TestVnetPerfFreeBSDDownload), a VNET_TCP_DEBUG
endpoint sampler, and pcaps:
First, throughput is capped at receive-window/RTT. FreeBSD starts its
receive window at 64 kB and autoscales it in slow 16 kB steps, and on
an oversubscribed CI runner the effective RTT of the userspace vnet
data path reaches hundreds of milliseconds, giving almost exactly the
observed 250 kB/s. Fix: raise the FreeBSD guest's TCP buffer sysctls
in cloud-init before the downloads, and raise netstack's receive
buffer sizing for the reverse (upload) direction.
Second, the outright wedge: when netstack bursts more data than the
QEMU socket plus the guest's virtio RX ring can absorb, a wide swath
of segments is dropped downstream of vnet, and netstack's loss
recovery then crawls, retransmitting one or two segments per 200 ms
RTO for minutes at a time (a 33 MB transfer was observed taking 526
seconds against an otherwise idle receiver). Rather than depending on
recovery from mass loss, make the path effectively lossless by keeping
the maximum in-flight data (the 1 MB netstack send buffer) below the
downstream buffering: grow the guests' virtio RX rings from 256 to
1024 descriptors, enlarge the vnet-QEMU unix socket buffers, and grow
the netstack link endpoint queue from 512 to 4096 packets so a send
burst can't overflow it.
Also fixed along the way, found while chasing the above:
* pcapWriter fsync'd after every packet, serializing all traffic
behind disk writes when a test enables pcap; a pcap-enabled run
was capped at about 290 kB/s. Keep the per-packet Flush but drop
the per-packet fsync.
* Traffic originating from vnet's own netstack (control plane,
DERP, file servers) bypassed conditionedWrite, so SetLatency and
SetPacketLoss silently didn't apply to it.
* writeEthernetFrameToVM held one global mutex (and a shared
scratch buffer) across writes to all VMs, so one guest slow to
drain its socket stalled traffic to every VM on the server. The
write lock is now per-VM-connection.
TestSubnetRouterFreeBSD now passes locally in 31s (down from 4.5
minutes), still passes with the vnet simulating a 100 ms RTT
(downloads at 2-8 MB/s, previously 250-600 kB/s), and passes in 65s
with KVM disabled while pinned to two host CPUs, a harsher environment
than the CI runners. The benchmark test is opt-in via
--run-perf-tests (in addition to --run-vm-tests) so CI doesn't spend
a matrix job re-measuring it on every run. VMTEST_NO_KVM=1
forces TCG for reproducing slow-host behavior.
Fixestailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I1a7945a7e9c7d083b0ea2a3530eda0e9757dff18
The label "natlab" is a bit confusing and also used for other things.
Instead, change the trigger label to "run-natlab-tests".
Updates #13038
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
The natlab vmtest suite (tstest/natlab/vmtest) and the integration nat
tests are gated behind --run-vm-tests because they need KVM and are
slow. Until now nothing in CI exercised them apart from a single
canary TestEasyEasy run on every PR.
Add .github/workflows/natlab-test.yml that runs the full opt-in suite
on demand (workflow_dispatch), on PRs labeled "natlab", and on main
every 12 hours via cron. The workflow has two phases:
- "prepare" builds the gokrazy VM image, downloads the Ubuntu and
FreeBSD cloud images once via the new natlabprep tool, and emits
a dynamic JSON matrix of every TestX function it finds in the two
opt-in packages.
- "test" is a per-test matrix that depends on prepare. Each matrix
job restores the shared caches and runs a single test, so adding
a new TestFoo is automatically picked up on the next run without
any workflow edits.
Rename the existing natlab-integrationtest.yml to natlab-basic.yml
since it's the small smoke variant (just TestEasyEasy on every PR);
the new natlab-test.yml is the bigger suite. The job inside is
renamed to EasyEasy for the same reason.
Move the macOS arm64 host check from vmtest.Env.Start into
vmtest.Env.AddNode so a test that adds a vmtest.MacOS node skips
immediately on a non-macOS host, and add an explicit
skipIfNotMacOSArm64 helper at the top of the two macOS-only tests
so the platform requirement is obvious to readers.
Quiet the takeAgentConnOne miss log in tstest/natlab/vnet by default
(it was the overwhelming majority of bytes in CI logs, with no signal
in healthy runs) and replace it with a periodic "still waiting" line
that only fires after 10s, so a truly stuck agent connection still
surfaces.
Updates #13038
Change-Id: I4582098d8865200fd5a73a9b696942319ccf3bf0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Brad Fitzpatrick
Claus Lensbøl