Files whose names contain characters with Unicode decompositions (such
as umlauts or voiced kana) could not be opened or written over
Taildrive.
Background: keyboards and IMEs emit NFC (precomposed) characters on
every platform, so filenames on Linux (ext4 etc) and Windows (NTFS)
disks are usually NFC bytes. NFD (decomposed) names mostly come from
Apple software: HFS+ forced a variant of NFD on write, and Apple's
frameworks still decompose paths via fileSystemRepresentation. APFS
preserves whatever bytes it is given but does normalization-insensitive
lookups (it stores a hash of the normalized name), so canonically
equivalent names find the same file. ext4 and NTFS lookups, by
contrast, are byte-exact.
On the wire, the macOS WebDAV client sends paths in NFD form (they
pass through the decomposing file system representation, and unlike
Apple's NFS client there is no "nfc" mount option). Windows and Linux
WebDAV clients pass names through as the application provided them,
typically NFC. WebDAV itself mandates no normalization, and PROPFIND
hrefs reflect the server's on-disk bytes.
The two forms are canonically equivalent but byte-wise different, so a
macOS client requesting the NFD form of an NFC-named file on a Linux
or Windows host got a 404 from the exact-byte lookup. Even against an
APFS host, where the filesystem absorbs the mismatch, the client-side
StatCache could still infer a 404: a cached directory listing in one
form caused depth 0 PROPFINDs in the other form to be treated as not
found without ever reaching the server. The inverse direction (NFD
bytes on a Linux disk, copied there from a Mac, requested in NFC form
by a Windows or Linux client) was broken too.
Alternative regimes considered: normalizing names at storage time (as
Nextcloud and Syncthing's autoNormalize do) would rename user files in
shared directories as a side effect of serving them; normalizing
request paths to a fixed form on the wire is unsound because the
on-disk form is unknowable a priori (ext4 can hold either form, or
both). Instead, adopt the APFS model: preserve bytes, but make lookups
normalization-insensitive.
Concretely, wrap the remote file server's webdav.Dir in a
normalizingFS that, when an exact path lookup fails, rescans the
parent directory for an entry whose name is canonically equivalent,
comparing the NFC form of both sides (which also sidesteps Apple's
nonstandard decomposition tables). Exact matches always win, and newly
created files keep the exact bytes the client sent. Also NFC-normalize
StatCache keys so canonically equivalent names share a cache entry.
The change is covered at three levels: unit tests for the StatCache,
an in-process two-node test in drive/driveimpl, and a new TestTaildrive
VM integration test in tstest/natlab/vmtest that shares a directory
between two Ubuntu VMs and exercises the NFC/NFD cases over the real
stack with curl playing the part of a macOS WebDAV client.
Fixes#15020
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9c2f157e604efc629828581e08d5b3191dbb7d4e
Change tstest's exported functions (AssertNotParallel, Replace,
Parallel, RequireRoot, SkipOnKernelVersions, MinAllocsPerRun, FixLogs,
UnfixLogs, CheckIsZero, ResourceCheck) to take testenv.TB instead of
testing.TB or *testing.T, so importing tstest from non-test code no
longer links the testing package and its flag registration side
effects into the binary. Add testenv.Verbose to replace the one use of
testing.Verbose, and a deptest check to keep testing out of tstest's
dependency graph.
Callers are unaffected: *testing.T and testing.TB both satisfy
testenv.TB.
Updates tailscale/corp#45223
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib373ff66ceff638d071582baf8367245987e9155
* tstest/natlab/vmtest: make cloud VM boot robust without KVM
Adding heavier distro images (Fedora) surfaced several ways the cloud VM
boot path breaks under TCG software emulation (no /dev/kvm), especially
with multiple concurrent VMs on few cores.
- Add a virtio-rng device to the cloud path so early boot doesn't block in
getrandom() waiting for the CRNG to seed.
- When no hardware acceleration is available, relax the stuck-console
watchdog (tuned for KVM's ~1-2s first output) and serialize VM boots so a
heavy guest doesn't starve its siblings' emulation threads.
- Bound the bring-up context to the test deadline and dump each VM's console
on failure, so a hang surfaces as a diagnosable Fatalf instead of an
opaque `go test -timeout` panic (which skips cleanups).
Fixestailscale/corp#44794
Updates tailscale/corp#44793
Signed-off-by: Brendan Creane <bcreane@gmail.com>
* tstest/natlab/vmtest: add Fedora and DNS-backend test coverage
Add the first RHEL-family distro and the machinery to assert and provision
distinct DNS backends, so adding a distro isn't "basically equivalent" to
the others.
- Add a Fedora 43 image (NetworkManager + systemd-resolved, SELinux
enforcing). restorecon-relabel the curl'd binaries so they exec under
enforcing mode.
- Add DNSBackend/AssertDNSBackend, reading the dns_manager_linux_mode_*
clientmetric to assert which backend tailscaled selected.
- Add a WithDNSMode node option. WithDNSMode(DNSDirect) masks
systemd-resolved and writes a plain resolv.conf pointing at natlab's fake
DNS, forcing the direct backend -- so one image covers multiple backends.
Fixestailscale/corp#44796
Updates tailscale/corp#44793
Signed-off-by: Brendan Creane <bcreane@gmail.com>
---------
Signed-off-by: Brendan Creane <bcreane@gmail.com>
* tstest/integration: run a test against a real Windows tailscaled service
Updates #20381
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
* tstest/integration: serialize Windows service tests and clean up state
Updates #20381
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
* tstest/integration: address review feedback on Windows service tests
Updates #20381
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
* tstest/integration: use background context for service teardown
Updates #20381
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
* tstest/integration: run Windows service test via the normal windows CI run
Updates #20381
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
* tstest/integration: address review feedback on Windows service tests
Updates #20381
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
---------
Signed-off-by: Yaruk Asghar <yaruk@tailscale.com>
The "vm" CI job ran a single test (TestRunUbuntu2404 from
tstest/integration/vms) on a privileged self-hosted runner. Its coverage
is nearly all redundant with the modern natlab vmtest suite, which boots
real Ubuntu VMs and already exercises connectivity, kernel TUN, SSH,
Taildrop, ACME, and OS DNS integration on GitHub-hosted runners.
The two things it tested that natlab didn't are added back as natlab
tests so the runner can be decommissioned:
TestUbuntuSystemdUnit runs tailscaled via the stock systemd unit that
Linux packages ship (cmd/tailscaled/tailscaled.service with
tailscaled.defaults as its EnvironmentFile) instead of launching the
binary directly, verifying the unit's directives and its Type=notify
readiness handshake.
TestDNSExtraRecordsSearchDomains verifies that control-plane DNS
ExtraRecords and search domains are resolvable through the guest's OS
resolver (libc to systemd-resolved to quad-100).
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I8d0dfb8b8153289e7ca78f3af03dfece9497bfe8
The tailscale/gokrazy-kernel module was a fork of rtr7/kernel that
stalled at Linux 6.8.9 (July 2024). All of the kernel config options we
had added in that fork (ENA, Xen for EC2, virtio-mmio for qemu microvm,
virtio RNG, IPv6 policy routing, netlink diag, etc) are now present in
the gokrazy project's own gokrazy/kernel.amd64 module, which tracks
current kernel.org releases (Linux 7.1.3 as of this change) and is the
gokrazy project's supported kernel for x86_64 PCs and VMs.
Switch the tsapp and natlabapp images, the natlab VM tests, and the
CI workflow to gokrazy/kernel.amd64, drop the tailscale/gokrazy-kernel
dependency, and update gokrazy/kernel.arm64 to latest while here.
Verified with TestEasyEasy, TestJustIPv6, and TestTailscaleSSH in
tstest/natlab/vmtest with --run-vm-tests.
Updates #1866
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I90c3765a4e18f5609b4d77b51ac38d17c8e3688a
The tun-layer per-peer data plane (jailed packet filter selection and
masquerade NAT rewrites) had no end-to-end coverage: nothing asserted
that a peer the control server marks jailed actually has its flows
dropped, or that masquerade addresses assigned by control actually
carry rewritten traffic in both directions.
Add a two-gokrazy-node natlab test driving both knobs from the test
control server and probing with HTTP requests between the nodes'
webservers after each netmap transition, asserting each response
carries the serving node's greeting so masqueraded flows provably
reach the intended node. TSMP pings are deliberately not used as
probes: tstun answers those before running the packet filter, so
they succeed even for jailed peers.
Updates #12542
Change-Id: Ia978e8d368f08a5a1117280f12bd50310969d0ec
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The BIRD (BGP) integration previously lived half in cmd/tailscaled
(which created a chirp client via a build-tag-gated file on some
platforms) and half in wgengine (which carried the client in its
Config and toggled the "tailscale" protocol as the node gained or
lost primary subnet router duty).
Move it all to a new feature/bird package, installed on the engine
via the new wgengine.HookNewBird hook, like other feature/* packages.
wgengine.Config.BIRDClient (and the wgengine.BIRDClient interface)
are replaced by a BIRDSocket path from which the engine constructs
the feature's Bird handle at startup. The subnet router overlap
detection and protocol toggling move into feature/bird, preserving
the previous ordering: state is recomputed before Reconfig's
ErrNoChanges early return and applied after the router is configured.
tailscaled keeps BIRD support by default on the platforms that
previously had it (linux, darwin, freebsd, openbsd) via
feature/condregister.
Also, add an integration test, as this feature lacked much test
coverage previously.
Updates #12614
Change-Id: I7866a50779e454c87933b358735f7dcd9e2b126f
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add TestTailscaleSSH to tstest/natlab/vmtest, exercising the Tailscale
SSH server (tailscale up --ssh, not a system sshd) end to end: an
Ubuntu client node SSHes over the tailnet into an Ubuntu server node
as both root and a non-root user, and into a gokrazy node.
The gokrazy sessions exercise the gokrazy special cases in the SSH
code: util/osuser hard-codes the login shell to serial-busybox ash and
synthesizes a root user when lookup fails (so any username works and
becomes root, unlike Ubuntu where nonexistent users are rejected), and
the incubator's findSU refuses su on gokrazy, handling sessions
in-process.
To support this, testcontrol gains an SSHPolicy field that's sent in
MapResponses along with the CapabilitySSH node capability, tta's /up
handler accepts an ssh=true parameter, and vmtest gains a
TailscaleSSH node option that wires the two together with a
permissive any-principal policy.
Updates tailscale/corp#44813
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f6b9c41a72e05d8c94dd7f6ab1937cf24b81c92
LocalBackend.Start previously shut down the previous control client in
a goroutine, letting it run concurrently with the new one. An in-flight
lite map update carrying stale Hostinfo.RequestTags could then be
processed by the control plane after the new client had already changed
the node's tags. Control treats such a request as an invalid tag
transition and expires the node key to force a reauth, so retagging a
node with "tailscale up --advertise-tags" intermittently logged the
machine out.
Instead, detach the old client under b.mu and shut it down
synchronously with the lock released, before creating the new client.
Shutdown cancels the old client's in-flight requests and waits for its
goroutines to exit, so the cancellation of any stale update reaches the
server before the new client sends its first request. Per the deadlock
history in #18052, Shutdown must not be called with b.mu held; this
uses the same pattern as DisconnectControl.
Also teach the testcontrol server to model the control plane's tag
transition handling (including expiring the node key on an invalid
transition and ignoring updates from canceled requests), add an
integration test reproducing the race, and add an ipnlocal test
verifying that Start waits for the old client to shut down.
Updates #20365
Updates #18052
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If8c8e145bdadcef1b1b8fe6209453cf5f5a8d616
FreeBSD guests downloaded their test binaries from vnet's
files.tailscale VIP at roughly 250 kB/s in CI, and transfers sometimes
wedged outright for many minutes, which is why TestSubnetRouterFreeBSD
timed out in about a third of its runs. Locally the same path moves
data at 100+ MB/s, so the problem was never CPU; it was TCP behavior
under two independent constraints, both diagnosed with a new
throughput harness (TestVnetPerfFreeBSDDownload), a VNET_TCP_DEBUG
endpoint sampler, and pcaps:
First, throughput is capped at receive-window/RTT. FreeBSD starts its
receive window at 64 kB and autoscales it in slow 16 kB steps, and on
an oversubscribed CI runner the effective RTT of the userspace vnet
data path reaches hundreds of milliseconds, giving almost exactly the
observed 250 kB/s. Fix: raise the FreeBSD guest's TCP buffer sysctls
in cloud-init before the downloads, and raise netstack's receive
buffer sizing for the reverse (upload) direction.
Second, the outright wedge: when netstack bursts more data than the
QEMU socket plus the guest's virtio RX ring can absorb, a wide swath
of segments is dropped downstream of vnet, and netstack's loss
recovery then crawls, retransmitting one or two segments per 200 ms
RTO for minutes at a time (a 33 MB transfer was observed taking 526
seconds against an otherwise idle receiver). Rather than depending on
recovery from mass loss, make the path effectively lossless by keeping
the maximum in-flight data (the 1 MB netstack send buffer) below the
downstream buffering: grow the guests' virtio RX rings from 256 to
1024 descriptors, enlarge the vnet-QEMU unix socket buffers, and grow
the netstack link endpoint queue from 512 to 4096 packets so a send
burst can't overflow it.
Also fixed along the way, found while chasing the above:
* pcapWriter fsync'd after every packet, serializing all traffic
behind disk writes when a test enables pcap; a pcap-enabled run
was capped at about 290 kB/s. Keep the per-packet Flush but drop
the per-packet fsync.
* Traffic originating from vnet's own netstack (control plane,
DERP, file servers) bypassed conditionedWrite, so SetLatency and
SetPacketLoss silently didn't apply to it.
* writeEthernetFrameToVM held one global mutex (and a shared
scratch buffer) across writes to all VMs, so one guest slow to
drain its socket stalled traffic to every VM on the server. The
write lock is now per-VM-connection.
TestSubnetRouterFreeBSD now passes locally in 31s (down from 4.5
minutes), still passes with the vnet simulating a 100 ms RTT
(downloads at 2-8 MB/s, previously 250-600 kB/s), and passes in 65s
with KVM disabled while pinned to two host CPUs, a harsher environment
than the CI runners. The benchmark test is opt-in via
--run-perf-tests (in addition to --run-vm-tests) so CI doesn't spend
a matrix job re-measuring it on every run. VMTEST_NO_KVM=1
forces TCG for reproducing slow-host behavior.
Fixestailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I1a7945a7e9c7d083b0ea2a3530eda0e9757dff18
Tailscaled had no way to seed device-scope syspolicy settings short of
environment variables or a custom store wired up out of tree. Add a
--syspolicy-file flag whose default points at a well-known JSON file
that, when present, is parsed as a map[string]any and registered as a
device-scope policy source. The default path is
/etc/tailscale/syspolicy.json on every non-Windows platform (Linux, the
BSDs, illumos/Solaris, and tailscaled-without-the-GUI on macOS) and
%ProgramData%\Tailscale\syspolicy.json on Windows. The flag lets users
running tailscaled by hand (development, custom installs) point it at
an alternate file, and "" disables the load entirely.
JSON values map to setting types as expected: strings to
StringValue/PreferenceOptionValue/VisibilityValue/DurationValue (e.g.
"24h" parsed by time.ParseDuration), booleans to BooleanValue, numbers
to IntegerValue, and string arrays to StringListValue. The file is
validated against the registered setting definitions at load time so
unknown keys and value/type mismatches fail startup loudly rather than
producing surprising defaults at first read.
When HuJSON support is linked into the build (default; opt out with
ts_omit_hujsonconf), the file may use HuJSON (comments, trailing
commas). With ts_omit_hujsonconf it must be pure standard JSON. This
mirrors the pattern used by ipn/conffile.
On Windows the JSON file and the existing HKLM registry store both
register at DeviceScope. rsop merges later-registered same-scope
sources over earlier ones, so per-key values in the file override the
registry while keys absent from the file fall back to the registry.
The loader is registered via a feature.Hook from a file gated by
!ts_omit_syspolicy, and called from main after flag parsing. tsnet
still does not depend on the root syspolicy package, so embedders
don't pick this up implicitly.
Fixes#20305
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6326461c14efb226979ac162998a9c6373ce493
Adds two Gokrazy-based vmtests covering the tailscaled web client at
port 5252:
* TestWebClientLocalAccess enables the web client on a single node
and exercises the canonical owner session flow against the node's
own Tailscale IP: an unauthenticated GET /api/auth that identifies
the caller, a GET /api/auth/session/new that issues a
TS-Web-Session cookie, and a final GET /api/auth that reports
authorized=true with the cookie.
* TestWebClientRemoteAccess runs the same session flow from a peer
node on the same tailnet against a second target node's web
client, exercising netstack interception of incoming :5252
traffic, cross-node WhoIs, and the same-user "owner" path. It
then flips the test control server's AllNodesSameUser off,
re-logs in the client under a fresh identity, and asserts that
GET /api/auth/session/new returns 401 with body "not-owner" --
exercising the cross-user rejection in client/web/auth.go.
To make the natlab test environment exercise the same code path
as production (check mode, where the web client posts to
/machine/webclient/init via Noise and waits on a control-issued
auth URL), this also:
* Allowlists the natlab fake control hostname "control.tailscale"
in client/web/auth.go's controlSupportsCheckMode so the web
client follows the check-mode branch rather than the
no-check-mode shortcut that immediately marks new sessions
authenticated.
* Adds /machine/webclient/{init,wait} handlers to testcontrol.
init returns a placeholder auth ID and URL; wait returns
Complete=true immediately, so the web client's awaitUserAuth
resolves on its first call. Together these let the tests drive
the full check-mode session lifecycle without a real
browser-click loop.
To support the multi-request HTTP flows from the test harness,
this also adds:
* vmtest.Env.HTTPGetStatus, a sister of HTTPGet that returns the
upstream status code, body, and Set-Cookie cookies (as a
vmtest.HTTPResponse) and accepts cookies on the outgoing
request, so tests can drive flows that depend on cookie
continuity.
* Cookie pass-through in cmd/tta's /http-get handler: it forwards
the Cookie request header upstream and surfaces upstream
Set-Cookie response headers downstream. This is what lets
HTTPGetStatus carry a session cookie across requests.
Previously the only tests of the web client were in-process
httptest-based handler tests in client/web/web_test.go; nothing
exercised the actual port 5252 listener wiring, the cross-node
auth path, cookie-driven session state transitions through the
check-mode control round-trip, or the not-owner rejection end
to end.
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Idb01486a89b53ac02c6ad3358bcfcceca90dbc36
Move all the FooForTest methods on LocalBackend to instead be
methods on a new unexported forTest type which is then given out
to callers in other packages via an exported ForTest method
(panicking in non-test contexts) that returns that unexported type.
This is unusual style (exported returning unexported) but declutters
godoc and makes call sites both more explicit and easier to read
without the "ForTest" suffix polluting the symbols. Now FooForTest()
changes into ForTest().Foo().
This was motivated by a pending change moving a bunch of code out of
LocalBackend into other packages that required adding more ForTest
methods to LocalBackend to keep the tests (now in other packages)
working. Instead, do this refactor now so the future change is prettier.
Updates #12614
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib25e6d76d48dc8622ac3a955e0b1220d582e63a8
prometheus/common v0.66/v0.67 introduced a mandatory
model.ValidationScheme on expfmt.TextParser as part of
prepping for UTF-8 metric/label names in Prometheus 3.0. The
zero value is intentionally UnsetValidation, which panics on
the first call to IsValidMetricName / IsValidLabelName with
Invalid name validation scheme requested: unset
so the long-standing "var parser expfmt.TextParser" pattern
crashes at runtime. Several big downstreams have hit the same
sharp edge:
https://github.com/thanos-io/thanos/issues/8823https://github.com/grafana/loki/pull/21401
Switch our two callers (parseMetrics in tsnet's
TestUserMetricsByteCounters and the client-metrics scraper in
tstest/natlab/vmtest) to the new expfmt.NewTextParser
constructor with model.LegacyValidation. LegacyValidation
matches the classic ASCII metric/label naming rules that
tailscaled's exporter uses today; if and when we ever emit a
metric with a UTF-8 name, we can revisit.
Goes to v0.69.0 (the latest at the time of writing) rather
than v0.67.5 so we pick up the unrelated security fixes for
cross-host redirects.
Done in advance so a follow-up change can pull in
github.com/tailscale/policybottest (which depends on
palantir/policy-bot, which transitively requires
prometheus/common at v0.67+) without dragging this debugging
into that PR.
Updates tailscale/corp#13972
Change-Id: I4b37db9ad3bebef1a32d9020bf6f8790bab25336
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Env.Start boots all VM nodes in parallel; each calls
createCloudInitISO -> ensureDebugSSHKey concurrently. When
/tmp/vmtest_key doesn't yet exist, the first goroutine creates it
with os.WriteFile, which opens with O_CREATE|O_TRUNC and briefly
leaves the file existing-but-empty between the open and the
subsequent write. A concurrent goroutine that hits that window
sees ReadFile succeed with zero bytes, then fails ssh.ParsePrivateKey
with "ssh: no key found", causing boot to fail with:
boot: creating cloud-init ISO: parse /tmp/vmtest_key: ssh: no key found
Observed in CI on TestSiteToSite (3 nodes). Wrap the function in
a package-level Mutex so the first caller fully writes the key
before any other caller reads it.
Updates #20228
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6399dcba0c397bb8041931d3de1c6063a11c568
Add a retry loop with BatchMode=yes to absorb the race window
between Env.Start() returning (when tta reports the tailscale
backend as Running) and cloud-init finishing the user/SSH-key
setup. In CI, the second VM's tta agent has been observed
connecting only a few hundred milliseconds before the test SSHes
in, which is inside the window where /root/.ssh/authorized_keys
hasn't fully landed yet. SSH key auth then fails and ssh(1) falls
back to interactive password prompts (3x), wasting time and
producing a confusing "Permission denied (publickey,password)"
error.
BatchMode=yes makes the client fail fast on auth failure instead
of prompting, and the retry loop handles SSH transport-level
errors (exit code 255) for up to 30 seconds with 500ms backoff.
Remote command non-zero exits still pass through unchanged.
Fixes#20228
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I17f7422e9e27bf7b995f505c0184cbb2b230ed81
Both tests started flaking after my 910735448 ("tstest/natlab/vnet:
send unsolicited IPv6 Router Advertisements") added background RA
traffic on v6-enabled networks.
TestPacketSideEffects races the periodic unsolicited-RA goroutine
against its synchronous packet-count assertions: when the multicast
RA fires after the test has registered its sinks, both sinks receive
it and "got 1 packet, want N" becomes "got N+2".
TestProtocolQEMU's reader was doing raw Read on the SOCK_STREAM unix
socket and comparing the whole result to the expected length-prefixed
packet. The kernel is free to coalesce the on-register RA frame and
the test packet into one Read, in which case bytes.Equal fails and
the entire chunk (including the test packet's bytes) gets discarded
as "unexpected", leading to a 5s i/o timeout. Parse the QEMU uint32
length-prefix framing with io.ReadFull instead so we read exactly one
frame per iteration regardless of how the kernel buffers them. The
SOCK_DGRAM path (TestProtocolUnixDgram) keeps the original raw Read
since datagram boundaries are preserved.
These where the top two flakes in oss on the flakes dashboards.
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I32983656b692921a0f43a4a5e9a8a6ab2555ee49
Flakeytest seems to not work on vmtest. We have a few PRs that will fix
the problem on these tests, so skip to unblock.
Updates #19843
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
This adds testcontrol support for expiring individual node keys,
in order to enable test scenarios involving to key-expiry and
extension.
Updates #19326
Signed-off-by: Gesa Stupperich <gesa@tailscale.com>
9be21088f4 changed sending disco pings so
a callMeMaybe would be not be gated by endpoints existing if the node
was running off of a cached netmap.
This commit partly reverts that change, but keeps in a few bug fixes in
that commit and the tests that was introduced and now skipped.
The behaviour prior to 9be21088f4 is
retained.
Updates #20085
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Add a vmtest that guards the fix in #20025: after an in-process control
client swap (profile switch / interactive re-login), magicsock's NetInfo
dedup cache (netInfoLast) must be cleared so the structurally-identical
post-switch NetInfo (same PreferredDERP, same NAT shape) is re-reported to
the new control session rather than suppressed as unchanged.
The test brings a node up, pins its home DERP so the reported NetInfo is
identical across the switch, records the home DERP the test control learned,
switches to a fresh login profile on the same control/network/NAT/DERP, and
asserts the control re-learns the same non-zero home DERP for the node's new
identity. Without ResetNetInfoLast the assertion times out at HomeDERP=0.
To support this, vnet now serves the test control on port 443 (TLS) in
addition to port 80: an immediate re-login makes a fresh noise dial, and
because the prior dial was recent the control client forces an HTTPS (443)
dial (controlhttp.Dialer.forceNoise443), which the harness previously did
not answer. The control endpoint gets its own self-signed cert (the existing
selfSignedDERPCert helper, renamed to the generic selfSignedCert); the cert
is not validated since control noise dials authenticate via the Noise
handshake, so it only needs a TLS peer to complete the forced 443 dial.
Add Env.ForcePreferredDERP and Env.Relogin helpers for the above.
Updates #20024
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
This adds a fake vnet ACME service, TXT-backed SetDNS support, and a
VM test that fetches a certificate with tailscale cert, serves it with
tailscale serve, and verifies HTTPS from a second node.
This adds coverage motivated by #19915.
Updates #13038
Change-Id: Ie1e53409509337d81c8fbceb63f59f3dfbd48207
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds support for Gokrazy GAF (Gokrazy Archive Format) zip
auto-updates, starting to wire up Tailscale's clientupdate mechanism
to Gokrazy's update mechanism.
Currently there's just a CLI command to update from a GAF URL,
with an --unsigned flag for use in a new natlab vmtest.
Next step would be publishing unstable track GAF files on
pkgs.tailscale.com, with detached signatures, and then making the
clientupdate mechanism also download those and check signatures.
Updates #20002
Change-Id: Ib03c56f17a57f8a4638398ef83549dac4813323d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test was asserting that a tailnet ping between two nodes traversed
DERP rather than going direct. But that wasn't really the point of the test,
and I kept forgetting ways that magicsock could find direct paths and
thus break this test.
So loosen it.
We really just want to see whether DERP worked at all and was used in the process
of getting a ping through, whether it was direct or not.
And that "tailscale debug derp" worked at all, which was what the bug
was about to begin with.
No need for all the "must be over DERP" stuff.
Updates #15579
Change-Id: I70ca63dc10919efa3d193b7af1d31a4a3b9d3950
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
vnet only ever sent IPv6 RAs in response to a Router Solicitation. In
practice this meant gokrazy VMs running with a dual-stack LAN never
installed vnet's IPv6 default route: gokrazy brings the link up via
DHCPv4 and the kernel never emits an RS on its own under that init
path. Off-link IPv6 destinations like the fake DERP servers were
therefore unreachable from any gokrazy test node that also had v4
on the same interface. (Pure-v6 nodes happened to work because the
kernel sends an RS as part of v6-only autoconf.)
Fix this in two complementary ways:
- Send an unsolicited RA every 5s to the link-local all-nodes group
on every v6-enabled network. This matches what real routers do
(RFC 4861 §6.2.1, MaxRtrAdvInterval; we use a much shorter
interval than the spec's 200s default so short-lived tests don't
have to wait).
- Send a unicast RA to a newly-registered MAC as soon as a client
first transmits on the wire. Without this the first periodic RA
can land before any VM has connected and the next one isn't
until the next tick, which can be longer than the test runs.
Factor the RA serialization out into buildIPv6RouterAdvertisement so
the solicited, periodic, and per-client paths all share one body.
Update TestSelfSignedDERPHashPinning to use a dual-stack hard-NAT
builder and assert zero errors from DebugDERPRegion (instead of
filtering "over IPv6" errors as it had to before this change). The
new builder also sets TS_DEBUG_STRIP_ENDPOINTS=1 on tailscaled so
disco can't find a direct path: without endpoint stripping, the now-
working non-NATted IPv6 LAN gives the two hard-NAT'd nodes a direct
route, defeating the test's "must traverse DERP" assertion. (Hard
NAT alone was enough before this change because v6 routing was
broken.) Also update sendBetweenClients in the vnet unit tests to
tolerate the new on-register RA noise on its read path.
Updates #13038
Updates #19973
Change-Id: Ic281dc53702a25fa773c46313f453837814233e8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
serveDebugDERPRegion built its TLS config with
ServerName: cmp.Or(derpNode.CertName, derpNode.HostName), which for a
"sha256-raw:<hex>" CertName passed the raw fingerprint to Go's stock
verifier as a hostname; the handshake always failed with a hostname
mismatch. This is the second half of #15579; the first half (tailscaled
itself failing with "unexpected multiple certs presented") was fixed in
Extract a tlsConfigForNode helper that mirrors derphttp.Client.tlsClient
so that sha256-raw and domain-fronting CertName values are dispatched
to tlsdial.SetConfigExpectedCertHash and tlsdial.SetConfigExpectedCert
respectively, falling back to HostName when CertName is empty.
The core fix here was originally written by @imnuke in #19965; that PR
also added a unit test in ipn/localapi/debugderp_test.go which is
replaced in this commit by a new vmtest that exercises the whole stack:
vnet now serves a self-signed cert valid for each fake DERP node's
HostName and exposes its SHA-256 fingerprint, and vmtest grows a new
SelfSignedDERPCertPinning EnvOption that swaps the test DERP map's
nodes to CertName="sha256-raw:<hex>" with InsecureForTests cleared.
TestSelfSignedDERPHashPinning then stands up two hard-NAT'd nodes, has
them communicate over DERP, and calls DebugDERPRegion on each. Before
this fix the test fails with the exact x509 hostname-mismatch error
from the original bug; after, it passes.
Updates #15579
Change-Id: I61f38ffebc7ac5abc962639db1ae88f5cd8633b1
Co-authored-by: Nuke <nuke@imnuke.dev>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, sharding required tests to opt in by calling tstest.Shard,
which used a process-global counter to assign each test to a shard.
This had two problems: most tests didn't call it, so they ran on every
shard (defeating the purpose), and shard assignments were unstable
(depended on call order, so adding a test could reshuffle others).
Remove tstest.Shard and tstest.SkipOnUnshardedCI entirely. Instead,
have testwrapper implement sharding automatically for all tests: when
TS_TEST_SHARD=N/M is set, it uses "go list -json" (no compilation) to
find test source files, scans them for top-level Test/Benchmark/
Example/Fuzz function names, and filters by fnv32a(name) % M == N-1.
The filtered names are passed as an anchored -run regex to go test.
Using go list instead of "go test -list" avoids linking the test binary
twice (Go's build cache does not cache test binary linking).
Fixes#19886
Change-Id: I62ab7b3d757324d4c5fd0b5de50c1e3742681791
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Some tests in another repo were broken by tailscale/tailscale#19607.
This fixes them, by finishing off the rest of the migration away from
netmap.NetworkMap on the IPN bus in containerboot.
Containerboot used to rebuild a full NetworkMap-shaped view while
reacting to IPN bus notifications. Now it insteads has its own
netmapState type (immutable) of exactly what it needs to track, and
sends those immutable values around, making cheap edits of new
immutable values when an IPN bus edit arrives.
This should make cmd/containerboot scale to much larger tailnets now too.
Fixes#19852Fixestailscale/corp#42347
Updates #12542
Change-Id: I88adaf061f85f677f954a764935e6654329d75a6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Originally found when adding tests for working with cached netmaps, and
finding the added tests to be flakey.
When working off of a cached netmap, if a node exists in the cached
netmap but does not yet have any endpoints, DERP connections are
available but not direct ones. By sending callMeMaybe to nodes
without endpoints in the cached netmap, we can establish direct
connections for this edge case.
Aditionally, ensure that TSMP disco advert messages are not sent if the
endpoint does not have a valid address yet.
Fixes#19843
Updates #19597
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
NodeMutationAdd was a misleading name: a PeersChanged entry in a
MapResponse can represent either a truly new peer or a full
replacement for an existing peer that couldn't be expressed as a
PeerChangedPatch. Calling it "Add" implied it was always a completely
new node, which is wrong. (I'd changed my mind on the design of
mapping add/delete events to NodeMutations halfway through #19607 and
forgot to update the name, even though I'd updated half the docs)
Rename it to NodeMutationUpsert to reflect the actual semantics: the
node should be inserted or replaced in the peer map regardless of
whether it already existed.
Updates #19607
Updates #12542
Change-Id: Iebd3daddb3318cba02e115a1b184fcb3ee8f83d6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
There are two places where tailscaled transitions into a paused state:
1. tailscaled’s controlclient is initially created,
2. tailscale down, or the GUI equivalent, commands it to.
This patch unifies the implementation of both scenarios into
LocalBackend.shouldPauseControlClientLocked to prevent the
implementation from drifting.
The flaky tstest/integration.TestNoControlConnWhenDown test exposed
this mismatch, but only by accident. This patch also changes
TestNode.MustDown so that it runs `tailscale down` and then waits for
the testcontrol server to finish handling any associated /machine/map
requests.
Fixes#19831
Signed-off-by: Simon Law <sfllaw@tailscale.com>
For large tailnets (~50k+ nodes) with frequent peer churn (ephemeral
GitHub Actions workers etc.), tailscaled used to rebuild the full
netmap and fan it out on the IPN bus on every MapResponse that
added or removed a peer. There were two O(N) costs per delta: the
full netmap rebuild + every Notify.NetMap encode to every bus watcher.
This change tackles both:
1. Plumb O(1) peer add/remove through the delta path. PeersChanged
and PeersRemoved no longer prevent the delta happy path; instead,
they mutate the per-node-backend peer map in place.
2. Restrict ipn.Notify.NetMap emission to the platforms whose host
GUIs still depend on it (Windows, macOS, iOS) and migrate
in-tree consumers off it everywhere else:
- Migrate reactive consumers (containerboot, kube agents,
sniproxy, tsconsensus, etc.) off Notify.NetMap to the
previously-added Notify.SelfChange signal so they no longer
have to subscribe to the full netmap.
- Add ipn.NotifyNoNetMap so GUI clients on "legacy-emit" platforms
that have already migrated can opt out of the per-watcher
NetMap encode.
- Gate Notify.NetMap emission on the producer side by a compile-
time GOOS check, so the supporting code is dead-code-eliminated
on Linux and other geese where no GUI consumer needs it.
Re-running BenchmarkGiantTailnet from tstest/largetailnet, which was
added along with baseline numbers on unmodified main in ad5436af0d,
the per-delta cost (one peer add+remove pair) is now ~O(1) regardless
of tailnet size N:
N no-watcher (ms/op) bus-watcher (ms/op)
before now factor before now factor
10000 32 0.11 300x 166 0.13 1300x
50000 222 0.11 2000x 865 0.13 6700x
100000 504 0.12 4100x 1765 0.13 13400x
250000 1551 0.12 12500x 4696 0.15 32400x
Updates #12542
Change-Id: I94e34b37331d1a8ec74c299deffadf4d061fda9e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
When tailscaled is running in userspace-networking mode behind an
exit node (e.g. as a SOCKS5 proxy), it resolves a hostname and then
dials a single resolved IP through the tunnel. If the name has both
A and AAAA, Go's net.Resolver merges them and we pick ips[0], which
on an IPv6-native host is usually AAAA. If the exit node has no IPv6
egress (or vice versa), the dial fails silently through the tunnel
and the user sees a hang.
Resolve all candidates and race connect attempts across address
families with a 300ms happy-eyeballs delay, matching Go's net.Dialer
default and the existing pattern in net/dnscache (commit ee0a03b14).
First success wins; losers are cancelled and any conns they produce
are closed. A failBoost channel wakes the launcher when a connect
fails fast (e.g. ICMP "no route" via the tunnel) so we don't sit on
the 300ms timer when the answer is already known.
userDialResolve is refactored into userDialResolveAll (returns the
full candidate list) plus a thin single-IP wrapper for callers like
UserDialPlan that don't race. UserDial's per-IP dispatch (netstack
vs peer dialer vs SystemDial vs std) is extracted to dialOneUser so
each candidate can route correctly on its own merits.
Also fix serveDial in localapi to pass the original hostname to
UserDial rather than a pre-resolved IP, so the race can fire.
This fix is single-ended: it works against any exit node, including
old ones, with no protocol changes. The trade-off versus filtering
on the exit-node side via PeerAPI DoH is that every dial through an
unreachable-family exit node costs one failed connect attempt per
cache window, rather than zero, which is acceptable given the
simplicity.
Fixes#19792Fixes#13257
Change-Id: I9d7645d0034caf3ee22ecdd8070798353f77e94b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
serveMap cloned s.nodes[nk], mutated the clone outside the mutex,
then wrote it back via updateNodeLocked. A concurrent UpdateNode,
SetNodeCapMap, or other writer landing between the clone and the
writeback would be silently clobbered. Mutate the live node under
the mutex instead.
Surfaces in tsnet's TestListenService as a flaky ErrUntaggedServiceHost
panic: the test calls control.UpdateNode to attach a tag, a concurrent
updateRoutine map request from the host races, and the host's next
netmap arrives with Tags=[].
Updates #19822
Change-Id: I6c5ebd5e5bf79a40316f53f627157230773cb469
Signed-off-by: James Tucker <james@tailscale.com>
Add Go tests that drive a real headless Chromium (via chromedp) against
the built cmd/tsconnect/pkg/ artifact and verify the @tailscale/connect
public API surface end-to-end. The package has not been republished in
three years, in part because no test exercises the produced artifact at
runtime — only tsc --noEmit and a Go build run in CI.
TestCreateIPN loads pkg.js into the browser, calls createIPN with a junk
auth key, and asserts that pkg.createIPN / pkg.runSSHSession are
functions and that createIPN() returns an IPN with the documented
run/login/logout/ssh/fetch methods. No control-plane traffic.
TestFetchTailnetPeer stands up a full local tailnet (testcontrol +
DERP + a tsnet.Server peer) and verifies that the browser-side WASM
client can join over WebSocket-noise to the same control, connect to
DERP over WSS, and then ipn.fetch() an HTTP service hosted on the tsnet
peer through the tailnet. The test asserts the response body matches a
known string. Browser state transitions are logged: NoState -> NeedsLogin
-> Starting -> Running.
Tests are opt-in via --run-headless-browser-tests (matching the existing
--run-vm-tests pattern in tstest/natlab/vmtest) so they never fire in
casual `go test ./...` runs. When the flag is set, a test is skipped if
cmd/tsconnect/pkg/ has not been built, and fails with t.Error if no
chromium binary is found on $PATH (honoring $CHROME_BIN as an override).
findChromium also falls back to /Applications/Google Chrome.app and
/Applications/Chromium.app on darwin, since macOS Chrome's executable
lives inside an .app bundle and is not on $PATH by default. The
.github/workflows/test.yml wasm job is extended to install
google-chrome-stable and run the tests with the flag after build-pkg.
To prevent silently testing a stale pkg/main.wasm (built from an older
checkout than the rest of the test invocation), build-pkg now writes
pkg/build-info.json recording the sha256 of the raw (pre-wasm-opt)
go-build output. The test does its own `go build` of
cmd/tsconnect/wasm with the same -tags/-trimpath/-ldflags (factored
into a new cmd/tsconnect/wasmbuild package shared by both call sites)
and t.Fatalfs with a "rebuild" instruction on mismatch. Cost is
near-zero because the Go build cache from the prior build-pkg makes
the rebuild a cache hit.
The new wasmbuild package also replaces cmd/tsconnect's hardcoded -tags
string with a minimal-feature-set computation. wasmbuild.Keep names the
small set of feature/featuretags entries the browser client actually
needs (netstack, logtail, dns, health, c2n, ipnbus); wasmbuild.Tags()
emits a ts_omit_<f> for every other
omittable feature in feature/featuretags.Features, with transitive deps
expanded via featuretags.Requires. An init() panics if Keep references
a feature unknown to feature/featuretags so a rename there fails
loudly. Net effect on size: 32M raw / 9.4M brotli before this change,
25M raw / 4.4M brotli after — vs the last-published 1.39.98 at 21M /
3.8M. The transitive package-import graph is unchanged (176
tailscale.com/* packages either way): featuretags omits eliminate
dead code via `const HasX = false`, not imports. Trimming the import
graph would require a separate, larger refactor splitting interface
packages by build tag.
Writing TestFetchTailnetPeer surfaced several real issues, all fixed
here:
* cmd/tsconnect built the wasm with the nethttpomithttp2 tag, but
control/ts2021 (since commit 1d93bdce2, "control/controlclient:
remove x/net/http2, use net/http", Oct 2025) requires HTTP/2 from
net/http's bundled implementation. With nethttpomithttp2 set, the
bundle is excluded and the wasm client cannot speak HTTP/2 to any
control plane, including production. Drop the tag. Wasm size grows
~1 MB raw / ~300 KB brotli (more than offset by the feature
pruning above). The last published @tailscale/connect (1.39.98,
early 2023) pre-dates the regression, which is why no consumer has
reported the breakage.
* tstest/integration/testcontrol.Server's /ts2021 noise upgrade
endpoint rejected anything but POST. WebSocket clients (the only
transport available to browser-WASM) come in as GET. Allow both;
the controlhttp AcceptHTTP path dispatches on the Upgrade header,
so the websocket library still enforces GET for WS upgrades.
This matches production, where the same controlhttpserver.AcceptHTTP
routes purely on the Upgrade header without checking method.
* derp/derphttp's urlString built the DERP URL from node.HostName
only, dropping node.DERPPort. Non-WS clients use a separate code
path (connectToHost) that honors DERPPort, but WebSocket-only
clients (browser-WASM) went through urlString and so could not
reach a DERP running on any port other than 443. Include the port
when it differs from the scheme default.
Also move addWebSocketSupport from cmd/derper (where it was main-only)
to derp/derpserver.AddWebSocketSupport so tstest/integration.RunDERPAndSTUN
can wrap its DERP handler with WebSocket support — without that, the
test DERP would not accept the browser's wss connection.
Fixes#9394
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iff9cdee303e3b239924249b5bffb2fd04e02f391
If the context given to DialContext has a shorter lifetime than the OS
TCP SYN timeout, and TCP SYNs are dropped from the path to the remote,
DialContext would never fall back to try IPv6 after IPv4.
Instead, use the normal happy eyeballs race if there is more than one
address. This does remove the implicit prioritization of IPv4 over IPv6
in cases where there is only a single IPv4 remote address.
Updates #13346
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Add a VM-based natlab test that exercises the peer-relay feature
(feature/relayserver) end-to-end across three Tailscale nodes whose
network topology makes a direct A<->B UDP path impossible: both peers
are behind HardNAT (FreeBSD/pfSense-style endpoint-dependent NAT) with
no port-mapping services, while the relay node is behind One2OneNAT so
its STUN-discovered WAN endpoint is reachable from both peers. The
test enables the relay server via EditPrefs, then waits for an a->b
PingDisco whose PingResult.PeerRelay is set (proving magicsock chose
the peer-relay path, not DERP), and finally asserts that the relay's
DebugPeerRelaySessions LocalAPI reports the session.
The existing TestPeerRelayPing in tstest/integration runs three
tailscaled processes on the loopback interface with no NATs; this new
vmtest covers peer relay through real per-VM kernels and NATs.
To wire control-server capabilities into vmtest, also add a
PeerRelayGrants() EnvOption (sibling of AllOnline,
SameTailnetUser) that flips testcontrol.Server.PeerRelayGrants so the
wildcard packet filter grants tailcfg.PeerCapabilityRelay and
PeerCapabilityRelayTarget; without those caps magicsock won't consider
any peer a candidate relay.
Updates #13038
Change-Id: Ib3440b83ec442da0d3b89ffa48ceea9398ea9062
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
In a lot of places, we construct an error to End a step, then immediately log
it to the governing test as test fatal. Save ourselves a bit of boilerplate by
putting methods on Step for that.
There are a couple cases this doesn't cover, e.g., where we construct the Step
outside a subtest that wants to fail individually, but it helps enough to pay
for its lines.
Updates #13038
Change-Id: I71f9900942962de16609b6b198d3ba13d6958a5f
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
Instead of having two entry points for running natlab tests, start
converting the connectivity tests to use the vmtest framework.
Grid and pair tests have yet to be moved over.
Updates #13038
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Add new clientmetric counters for establishing contact with peers while using
cached network map data. To do this, instrument the magicsock.Conn with a bit
to indicate whether its peer data came from a cached netmap. If so, there are
two conditions we will count as establishing connectivity to a peer:
- Receipt of a CallMeMaybe from a peer via disco.
- Establishing a valid endpoint address for a peer.
In vmtest, add Env.ClientMetrics to scrape metrics from the specified node.
Use this to check that counters were updated in caching tests.
Updates https://github.com/tailscale/projects/issues/13
Updates #12639
Change-Id: Ie8cf3244ac8af4f5bcfe4d0d944078da2ba08990
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
macOS limits Unix socket paths to 104 bytes. The Go test TempDir
path (e.g. /var/folders/.../TestDirectConnection...679197086/001/)
easily exceeds that, causing "bind: invalid argument". Create a
short /tmp/vmtest* directory for all socket files (vnet, QMP,
dgram) so the paths stay well under the limit on every platform.
Updates #13038
Change-Id: I721d24561d1766aaa964692bc77f40a131aa9455
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
startCloudQEMU hardcoded -machine q35,accel=kvm and -cpu host,
which fails on any host without KVM (notably macOS). Replace
with a qemuAccelArgs helper that probes /dev/kvm and falls back
to QEMU's TCG software emulation, matching the pattern already
used by tstest/integration/nat. Also wire the helper into
startGokrazyQEMU so gokrazy VMs pick up KVM when available.
Updates #13038
Change-Id: I7745518db823279b1880957bb14ca2ffdaab4c50
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The natlab vmtest suite (tstest/natlab/vmtest) and the integration nat
tests are gated behind --run-vm-tests because they need KVM and are
slow. Until now nothing in CI exercised them apart from a single
canary TestEasyEasy run on every PR.
Add .github/workflows/natlab-test.yml that runs the full opt-in suite
on demand (workflow_dispatch), on PRs labeled "natlab", and on main
every 12 hours via cron. The workflow has two phases:
- "prepare" builds the gokrazy VM image, downloads the Ubuntu and
FreeBSD cloud images once via the new natlabprep tool, and emits
a dynamic JSON matrix of every TestX function it finds in the two
opt-in packages.
- "test" is a per-test matrix that depends on prepare. Each matrix
job restores the shared caches and runs a single test, so adding
a new TestFoo is automatically picked up on the next run without
any workflow edits.
Rename the existing natlab-integrationtest.yml to natlab-basic.yml
since it's the small smoke variant (just TestEasyEasy on every PR);
the new natlab-test.yml is the bigger suite. The job inside is
renamed to EasyEasy for the same reason.
Move the macOS arm64 host check from vmtest.Env.Start into
vmtest.Env.AddNode so a test that adds a vmtest.MacOS node skips
immediately on a non-macOS host, and add an explicit
skipIfNotMacOSArm64 helper at the top of the two macOS-only tests
so the platform requirement is obvious to readers.
Quiet the takeAgentConnOne miss log in tstest/natlab/vnet by default
(it was the overwhelming majority of bytes in CI logs, with no signal
in healthy runs) and replace it with a periodic "still waiting" line
that only fires after 10s, so a truly stuck agent connection still
surfaces.
Updates #13038
Change-Id: I4582098d8865200fd5a73a9b696942319ccf3bf0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
