Adds a Linux-only framebuffer status display (cmd/fbstatus) that draws
to /dev/fb0 on the Tailscale gokrazy appliance. It shows:
- the Tailscale logo
- the current tailscaled state (starting, needs login, running)
- a QR code with the login URL when enrollment is needed (triggers
StartLoginInteractive automatically so the URL appears without
user action)
- the LAN IP or "Waiting for DHCP (MAC)" pinned at the bottom-left
- Tailscale IPs once connected
VT switching: Ctrl-Alt-F2 drops to a busybox text shell on VT2 (for
debugging with a USB keyboard), Ctrl-Alt-F1 returns to the GUI.
Rendering pauses while the text VT is active.
On boot, fbstatus pokes the gokrazy unix socket API to restart the
breakglass SSH service (which uses DontStartOnBoot by default). It
waits until DHCP assigns an IP so breakglass binds to the LAN address
rather than just localhost.
Included in the tsapp-pi.arm64 gokrazy build by default.
Updates #1866
Change-Id: Ifdce4ad8e8c2e1005c840f579e637974a0a266d3
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds gokrazy/gafpush and a 'make tsapp-push-pi PI=<ip>' Makefile
target for pushing a freshly-built GAF to a running appliance over
the network. See the gokrazy/gafpush package doc for details.
Updates #1866
Change-Id: Id7a0bad712fcf2eddae593f71d5feacee05c5234
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Update ts-gokrazy to b83088f which includes:
- Skip hardware watchdog when nowatchdog is on kernel cmdline
- gokrazy.log_to_serial=1 tees service logs to /dev/console
- Fix /etc/resolv.conf symlink (point at /tmp/resolv.conf where
userspace DHCP writes, not /proc/net/pnp which is always empty)
All these things are more emulating a Raspberry Pi in qemu when doing
local development of the appliance image.
Updates #1866
Change-Id: Iba7847e5deb237b1e485b74a4126e31fd118333a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add three new helpers to the existing progresstracking package:
- Ticker: spawns a 1 Hz goroutine that calls a report function with
the current value of an atomic counter and a total. Returns a stop
function (safe to call multiple times via sync.OnceFunc) that fires
one final report and blocks until the goroutine exits.
- NewWriter: wraps an io.Writer and calls onProgress at most once per
interval with the cumulative byte count.
- CountingWriter: an io.Writer that atomically counts bytes written,
for use with Ticker.
These will be used by the appliance flash and OTA update code in
subsequent commits.
Updates #1866
Change-Id: If353cea6506f5351b6fb19bfdb7bc9b78fe7855e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds a CLI subcommand that downloads a signed Tailscale appliance
image (Gokrazy archive format, GAF) from pkgs.tailscale.com,
constructs a fresh GPT-partitioned disk from it (mbr.img + a
synthesized partition table + boot.img + root.img), formats /perm
as ext4 in pure Go via go-diskfs, and ejects the disk so a user
running on a regular workstation can flash an SD card or homelab
VM disk in one command without installing e2fsprogs.
On macOS the target disk is auto-discovered via diskutil, skipping
the boot disk and anything bigger than 256 GB out of paranoia. On
Linux the user passes --disk=/dev/sdX explicitly. Windows is not
supported yet and the command returns an error.
The GPT layout matches monogok's full-disk layout via the new
public github.com/bradfitz/monogok/disklayout package; a drift-
guard test inside monogok asserts the two implementations stay
byte-identical so OTA updates against monogok-built images keep
working.
Behind a ts_omit_flashappliance build tag (on by default).
Updates #1866
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ic1a8cd185e7039edccb7702ab4104544fcb58d29
Tailscaled had no way to seed device-scope syspolicy settings short of
environment variables or a custom store wired up out of tree. Add a
--syspolicy-file flag whose default points at a well-known JSON file
that, when present, is parsed as a map[string]any and registered as a
device-scope policy source. The default path is
/etc/tailscale/syspolicy.json on every non-Windows platform (Linux, the
BSDs, illumos/Solaris, and tailscaled-without-the-GUI on macOS) and
%ProgramData%\Tailscale\syspolicy.json on Windows. The flag lets users
running tailscaled by hand (development, custom installs) point it at
an alternate file, and "" disables the load entirely.
JSON values map to setting types as expected: strings to
StringValue/PreferenceOptionValue/VisibilityValue/DurationValue (e.g.
"24h" parsed by time.ParseDuration), booleans to BooleanValue, numbers
to IntegerValue, and string arrays to StringListValue. The file is
validated against the registered setting definitions at load time so
unknown keys and value/type mismatches fail startup loudly rather than
producing surprising defaults at first read.
When HuJSON support is linked into the build (default; opt out with
ts_omit_hujsonconf), the file may use HuJSON (comments, trailing
commas). With ts_omit_hujsonconf it must be pure standard JSON. This
mirrors the pattern used by ipn/conffile.
On Windows the JSON file and the existing HKLM registry store both
register at DeviceScope. rsop merges later-registered same-scope
sources over earlier ones, so per-key values in the file override the
registry while keys absent from the file fall back to the registry.
The loader is registered via a feature.Hook from a file gated by
!ts_omit_syspolicy, and called from main after flag parsing. tsnet
still does not depend on the root syspolicy package, so embedders
don't pick this up implicitly.
Fixes#20305
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6326461c14efb226979ac162998a9c6373ce493
Builds on top of the unsigned URL-based GAF update flow added previously
(see referenced issue for context). The pkgs.tailscale.com server now
publishes signed GAFs for the unstable track, with detached ed25519
signatures produced by pkgsign's signdist path (the same distsign scheme
used for every other release artifact). This change consumes them.
The URL-based path (tailscale update --gokrazy-update-from-url=URL) now
verifies the signature by default using clientupdate/distsign.Client,
which fetches distsign.pub from the root of the host serving the GAF and
checks the .sig against the root keys embedded in this binary. The
--unsigned flag stays for TestGokrazyUpdatesItselfToSameImage, whose
in-test fileserver does not publish distsign.pub.
The bare tailscale update path is now wired up for the Tailscale
appliance image. It fetches <pkgs>/<track>/?mode=json, picks the GAF
whose key matches the local device (vm-amd64, vm-arm64, or pi-arm64,
where arm64 is split via /sys/firmware/devicetree/base/model), confirms
the version with the user, and reuses the verified download path above.
To avoid wiping a user's custom Gokrazy build that happens to include
tailscaled, the bare update path is gated on hostinfo.Package == "tsapp",
which is only set when the new ts_appliance build tag is present
(mirroring the existing ts_package_container tag). The
gokrazy/tsapp*/config.json files now pass GoBuildTags ["ts_appliance"]
for the tailscale and tailscaled packages so monogok bakes the tag into
the official appliance builds. The TS_FORCE_ALLOW_TSAPP_UPDATE env var
is an escape hatch for callers who want to force the appliance update
path on a non-appliance build. The URL-based path stays ungated since it
requires explicit user intent (and is exercised by the natlab vmtest).
Updates #20002
Change-Id: I7c7856a88bf3dffb9eb8d3e9111fad0b3906743c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds two Gokrazy-based vmtests covering the tailscaled web client at
port 5252:
* TestWebClientLocalAccess enables the web client on a single node
and exercises the canonical owner session flow against the node's
own Tailscale IP: an unauthenticated GET /api/auth that identifies
the caller, a GET /api/auth/session/new that issues a
TS-Web-Session cookie, and a final GET /api/auth that reports
authorized=true with the cookie.
* TestWebClientRemoteAccess runs the same session flow from a peer
node on the same tailnet against a second target node's web
client, exercising netstack interception of incoming :5252
traffic, cross-node WhoIs, and the same-user "owner" path. It
then flips the test control server's AllNodesSameUser off,
re-logs in the client under a fresh identity, and asserts that
GET /api/auth/session/new returns 401 with body "not-owner" --
exercising the cross-user rejection in client/web/auth.go.
To make the natlab test environment exercise the same code path
as production (check mode, where the web client posts to
/machine/webclient/init via Noise and waits on a control-issued
auth URL), this also:
* Allowlists the natlab fake control hostname "control.tailscale"
in client/web/auth.go's controlSupportsCheckMode so the web
client follows the check-mode branch rather than the
no-check-mode shortcut that immediately marks new sessions
authenticated.
* Adds /machine/webclient/{init,wait} handlers to testcontrol.
init returns a placeholder auth ID and URL; wait returns
Complete=true immediately, so the web client's awaitUserAuth
resolves on its first call. Together these let the tests drive
the full check-mode session lifecycle without a real
browser-click loop.
To support the multi-request HTTP flows from the test harness,
this also adds:
* vmtest.Env.HTTPGetStatus, a sister of HTTPGet that returns the
upstream status code, body, and Set-Cookie cookies (as a
vmtest.HTTPResponse) and accepts cookies on the outgoing
request, so tests can drive flows that depend on cookie
continuity.
* Cookie pass-through in cmd/tta's /http-get handler: it forwards
the Cookie request header upstream and surfaces upstream
Set-Cookie response headers downstream. This is what lets
HTTPGetStatus carry a session cookie across requests.
Previously the only tests of the web client were in-process
httptest-based handler tests in client/web/web_test.go; nothing
exercised the actual port 5252 listener wiring, the cross-node
auth path, cookie-driven session state transitions through the
check-mode control round-trip, or the not-owner rejection end
to end.
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Idb01486a89b53ac02c6ad3358bcfcceca90dbc36
f5eac39ea ("feature/acme, ipn/ipnlocal: start moving ACME/cert state
into an extension") started to move the cert code into feature/acme
but was meant as a baby step.
This goes further, moving almost everything, leaving only some hooks
in ipnlocal.
When we later move "serve" support out to feature/serve, this will
look a bit different in that the hooks currently in ipnlocal will move
to feature/serve (cert support already depends on serve).
As part of this, cert-related tests move to feaure/acme too, which
means some test infra from ipnlocal now moves to shared ipnlocaltest.
(it's not big at the moment, but I imagine it growing)
Updates #12614
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9ea89aa9754f12d54b81751b6bd830f2664241ff
This was missing in the earlier f5eac39ea7 and meant that tsnet users weren't
getting (all of) acme support.
Thanks to @ChaosInTheCRD and @BeckyPauley for debugging.
Updates #12614
Updates #20252
Change-Id: I176a7b179b2ad3726aca484057f0aae7cc3561c8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Move all the FooForTest methods on LocalBackend to instead be
methods on a new unexported forTest type which is then given out
to callers in other packages via an exported ForTest method
(panicking in non-test contexts) that returns that unexported type.
This is unusual style (exported returning unexported) but declutters
godoc and makes call sites both more explicit and easier to read
without the "ForTest" suffix polluting the symbols. Now FooForTest()
changes into ForTest().Foo().
This was motivated by a pending change moving a bunch of code out of
LocalBackend into other packages that required adding more ForTest
methods to LocalBackend to keep the tests (now in other packages)
working. Instead, do this refactor now so the future change is prettier.
Updates #12614
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib25e6d76d48dc8622ac3a955e0b1220d582e63a8
Add a .policy-tests.yml file with tests exercising the policy
that was just landed: the tailcfg/ control-protocol-owners gate,
the "policybot-override:" comment escape hatch (including
defaults-regression guards so the override rule does not
silently accept a normal review or a 👍 comment), and the
always-on "any tailscale/dev review" baseline.
Updates tailscale/corp#13972
Change-Id: I42afb06b0771658c803512cb5de4701450c8a704
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
prometheus/common v0.66/v0.67 introduced a mandatory
model.ValidationScheme on expfmt.TextParser as part of
prepping for UTF-8 metric/label names in Prometheus 3.0. The
zero value is intentionally UnsetValidation, which panics on
the first call to IsValidMetricName / IsValidLabelName with
Invalid name validation scheme requested: unset
so the long-standing "var parser expfmt.TextParser" pattern
crashes at runtime. Several big downstreams have hit the same
sharp edge:
https://github.com/thanos-io/thanos/issues/8823https://github.com/grafana/loki/pull/21401
Switch our two callers (parseMetrics in tsnet's
TestUserMetricsByteCounters and the client-metrics scraper in
tstest/natlab/vmtest) to the new expfmt.NewTextParser
constructor with model.LegacyValidation. LegacyValidation
matches the classic ASCII metric/label naming rules that
tailscaled's exporter uses today; if and when we ever emit a
metric with a UTF-8 name, we can revisit.
Goes to v0.69.0 (the latest at the time of writing) rather
than v0.67.5 so we pick up the unrelated security fixes for
cross-host redirects.
Done in advance so a follow-up change can pull in
github.com/tailscale/policybottest (which depends on
palantir/policy-bot, which transitively requires
prometheus/common at v0.67+) without dragging this debugging
into that PR.
Updates tailscale/corp#13972
Change-Id: I4b37db9ad3bebef1a32d9020bf6f8790bab25336
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The override comment didn't work as expected.
(I'll be updating the policytest package to handle this)
Updates tailscale/corp#13972
Change-Id: Ic5c16eed09c8cb5fa8dab37d43cf05f8dfa75d49
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
GitHub's built-in CODEOWNERS only supports a hard "block until a team
member reviews" rule, with no way to leave an audit trail when the
requirement is intentionally bypassed. Move review enforcement to
palantir/policy-bot (https://github.com/palantir/policy-bot) running
at https://policybot.corp.ts.net, which lets us express the same
tailcfg/ -> control-protocol-owners rule plus an explicit override:
any other @tailscale/dev member can post
policybot-override: <reason>
as a PR comment and that comment counts as their approval, with the
reason recorded in the PR conversation as a permanent audit trail.
CODEOWNERS is kept as a one-screen comment so anyone landing on it
expecting the old behavior is directed to .policy.yml.
Updates tailscale/corp#13972
Change-Id: I2dc3619c498d4c4a6decae29aa123f6d67905eed
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The ACME serialization mutex (acmeMu) was a package-level global, and
several ACME-related fields lived on LocalBackend even though the
cert code is conditional and not linked into every binary. With
multiple tsnet.Servers in one process (each its own LocalBackend),
a process-wide acmeMu also serialized unrelated backends.
Introduce a new feature/acme extension that owns the per-LocalBackend
ACME/cert state in an ipnlocal.CertState value:
- acmeMu, renewMu, renewCertAt (previously package globals)
- pendingACMETLSALPNCerts, pendingCertDomains{,Mu},
getCertForTest, certRefreshCancel (previously LocalBackend
fields, only meaningful when ACME was compiled in)
ipnlocal/cert.go now reaches the state through b.certState(), which
is routed by a feature.Hook installed at init by feature/acme. The
CertState type lives in ipnlocal so cert.go can access its fields
directly without a method explosion; the extension in feature/acme
constructs and owns it.
This is a baby step. The end goal is for the entire cert/ACME code
to live in feature/acme, with ipnlocal only retaining whatever thin
hooks the rest of LocalBackend needs to call into it. The current
split (CertState and most of cert.go in ipnlocal, extension wrapper
in feature/acme) is a deliberately temporary middle ground that
keeps this PR small while making the next moves mechanical.
The package is named feature/acme to match the existing HasACME /
ts_omit_acme naming. condregister/maybe_acme.go wires it in for
non-js builds.
Updates #12614
Updates #20248
Updates #20249
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I520909f24ad11a9622ef33c2290fe36ad44d6f71
setWebClientAtomicBoolLocked and setDebugLogsByCapabilityLocked
each only need the node capabilities to decide what to do, so
take a set.Set[tailcfg.NodeCapability] directly as part of
getting rid of netmap.NetworkMap.
Updates #12542
Change-Id: If7c30b6354fd42dfe82ed6d2e2fe3439de401315
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The engine only used the netmap to look up self addresses and the
self node's primary routes, so pass it the self node directly
rather than the whole netmap.
Updates #12542
Change-Id: I13c0028eed65d2177baf4cf6c449f5e441845a18
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
tsdial.Dialer.SetNetMap rebuilt an O(n peers) map of MagicDNS names on
every netmap change. As we move toward per-peer incremental deltas,
this becomes quadratic. This removes it and replaces it with
SetResolveMagicDNS, a callback into LocalBackend that looks up
hostnames from nodeBackend's new nodeByName index (populated alongside
nodeByAddr/nodeByKey on both full and delta paths). The index stores
both FQDNs and short names as keys.
This is the same treatment applied to netlog (8f210454d), wglog
(988b0905b), and drive (1d6989408): stop pushing *netmap.NetworkMap
into subsystems and instead have them pull from LocalBackend's live
data via callbacks.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I24557ab0c8a27636e08e4779bcfd3ec633db0a78
Env.Start boots all VM nodes in parallel; each calls
createCloudInitISO -> ensureDebugSSHKey concurrently. When
/tmp/vmtest_key doesn't yet exist, the first goroutine creates it
with os.WriteFile, which opens with O_CREATE|O_TRUNC and briefly
leaves the file existing-but-empty between the open and the
subsequent write. A concurrent goroutine that hits that window
sees ReadFile succeed with zero bytes, then fails ssh.ParsePrivateKey
with "ssh: no key found", causing boot to fail with:
boot: creating cloud-init ISO: parse /tmp/vmtest_key: ssh: no key found
Observed in CI on TestSiteToSite (3 nodes). Wrap the function in
a package-level Mutex so the first caller fully writes the key
before any other caller reads it.
Updates #20228
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6399dcba0c397bb8041931d3de1c6063a11c568
Add a retry loop with BatchMode=yes to absorb the race window
between Env.Start() returning (when tta reports the tailscale
backend as Running) and cloud-init finishing the user/SSH-key
setup. In CI, the second VM's tta agent has been observed
connecting only a few hundred milliseconds before the test SSHes
in, which is inside the window where /root/.ssh/authorized_keys
hasn't fully landed yet. SSH key auth then fails and ssh(1) falls
back to interactive password prompts (3x), wasting time and
producing a confusing "Permission denied (publickey,password)"
error.
BatchMode=yes makes the client fail fast on auth failure instead
of prompting, and the retry loop handles SSH transport-level
errors (exit code 255) for up to 30 seconds with 500ms backoff.
Remote command non-zero exits still pass through unchanged.
Fixes#20228
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I17f7422e9e27bf7b995f505c0184cbb2b230ed81
userspaceEngine.PeerForIP read from e.netMap.Peers and
e.lastCfgFull.Peers, both of which go stale when peers arrive via
netmap deltas (which skip Engine.SetNetworkMap and Engine.Reconfig).
Every PeerForIP caller (Engine.Ping, the TSMP disco-key handler,
pendopen diagnostics, tsdial.Dialer.UseNetstackForIP, and
LocalBackend.GetPeerEndpointChanges) would report "no matching peer"
for freshly-added peers.
Fix it the same way SetPeerByIPPacketFunc fixed the outbound packet
hot path: have LocalBackend install a callback that reads the live
nodeBackend. nb.NodeByAddr is built from both SelfNode and Peers
(updateNodeByAddrLocked), so a single lookup covers the common case
with IsSelf set when the matched node ID is SelfNode's. The subnet-
route / exit-node-default-route slow path goes through a new
Engine.PeerKeyForIP that exposes the engine's AllowedIPs BART table
(the same table the outbound packet hot path already consults, with
exit-node selection honored), and resolves the matched key back to a
NodeView via the live nodeBackend.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0d4b0d8997c8e796b7367c46b49b61d4fdc717b0
Another baby step toward removing slices of peers from the engine.
getStatus iterated peerSequence (a key snapshot built in Reconfig
from cfg.Peers) and then asked wgdev for each peer's stats; peers
that weren't active in wgdev silently fell out. Iterate active wgdev
peers directly via RemoveMatchingPeers(returnFalse) instead.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3abd348abc30db706db29b3a785179259e48abda
The watchdog (ipn/ipnlocal/watchdog.go) was abusing PeerForIP with an
invalid netip.Addr as a way to acquire and release the engine's
internal locks for deadlock detection. This does the TODO to break it out
into its own method like all the other similarly named methods.
Splitting this out as a prerequisite for a follow-up rewrite of
PeerForIP itself; not having to preserve the lock-probe overload in
the new implementation keeps that follow-up smaller.
Updates #12542
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I25cbffd11aeb65600d9128845404c4918ef88ead
Detect Hetzner via /sys/class/dmi/id/sys_vendor == "Hetzner" and wire
up Hetzner's public recursive DNS resolvers (185.12.64.1, 185.12.64.2)
for use as a cloud host resolver.
Fixes#20217
Change-Id: I24a4c51956adfdd5731f62c937e3c7a4a733ffc7
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This applies the same treatment from PR #20162 (netlog) and
PR #20171 (wglog) to the local Taildrive filesystem wiring, ending the
per-netmap-update O(n) rebuild of the drive remotes list.
This moves the O(n peers) taildrive-remote list rebuild from every
peer change (which previously happened regardless of whether you were
even using taildrive) to instead happen only as needed.
That running on every netmap update and was a contributor to the
broader quadratic behavior we want to eliminate when a single peer is
added or removed.
Instead, this introduces drive.RemoteSource, a small interface the
Taildrive filesystem pulls from lazily on incoming WebDAV requests,
and caches by a generation counter. ipn/ipnlocal installs a
driveRemoteSource once at NewLocalBackend time and bumps
LocalBackend.driveGen on the three events that can actually flip the
drive-capable peer set: full netmap installs (domain + self caps),
UpdateNetmapDelta (peer add/remove or per-peer address changes), and
updatePacketFilter (since PeerCapability values are derived from the
packet filter rules, not from peer.CapMap).
The hook itself is kept but narrowed: it no longer takes a
*netmap.NetworkMap and its only remaining job is to re-notify IPN bus
listeners of the current local shares list on full installs.
This is a dependency to removing the netmap.NetworkMap type from
upstream callers, like wgengine.Engine in general.
(Also add a bunch more tests)
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7e3d2f5b4a9c8e1d6f0a3b7c9e2d4f8a1b6c5e9d
This applies the same treatment from 8f210454dd (netlog) to wglog,
ending use of netmap.NetworkMap and instead getting the canonical data
from LocalBackend/nodeBackend.
This is a dependency to removing the netmap.NetworkMap from
upstream callers, like wgengine.Engine in general.
Updates #12542
Change-Id: Icb5af0799322def048a6f594b49f7d11273f025d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test transferred only 64 KiB over loopback, which can complete
within a single clock tick on fast CI machines, causing
time.Since(start).Seconds() to return 0 and the
"transfer_time_seconds_total > 0" assertion to fail.
Increase the payload to 1 MiB so zero is genuinely implausible, and
retry up to 3 additional times. If the metric is still zero after 4
total attempts, fail hard — at that size it means the timing logic is
actually broken.
Fixes#20213
Change-Id: I3fab510ce8c567506fea5ad803d35acf40d65700
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
aa5da2e5f2 (in the 1.99.x dev series, unstable) introduced some bugs,
only some of which were later fixed. This fixed another. As of that
change, tkaFilterNetmapLocked ran only on full netmaps through
LocalBackend.setClientStatusLocked and not peer upserts via new or
changed peers. The later ae743642d9 fixed a regression in the
Engine layer but didn't fix the tkaFilter code from re-running on
upserts.
This add a tkaFilterDeltaMutsLocked pass before
nodeBackend.UpdateNetmapDelta. For each NodeMutationUpsert whose
peer fails the same signature check tkaFilterNetmapLocked applies,
rewrite the upsert in place into a NodeMutationRemove targeting the
same node ID, so magicsock's per-mutation dispatch and
nodeBackend.peers both drop the peer, matching the prior full-netmap
semantics.
New tsnet tests added:
- TestTailnetLockFiltersUnsignedDeltaPeer covers the new-peer
case.
- TestTailnetLockFiltersUnsignedDeltaPeerReplacement covers the
existing-peer-replacement case, to an empty signature.
- TestTailnetLockFiltersDeltaPeerWithInvalidSignature like above
but with a bogus signature.
Updates #12542
Updates tailscale/corp#43767
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib35d0391541fee654867c26489847dbc5b7e2ae8
Both tests started flaking after my 910735448 ("tstest/natlab/vnet:
send unsolicited IPv6 Router Advertisements") added background RA
traffic on v6-enabled networks.
TestPacketSideEffects races the periodic unsolicited-RA goroutine
against its synchronous packet-count assertions: when the multicast
RA fires after the test has registered its sinks, both sinks receive
it and "got 1 packet, want N" becomes "got N+2".
TestProtocolQEMU's reader was doing raw Read on the SOCK_STREAM unix
socket and comparing the whole result to the expected length-prefixed
packet. The kernel is free to coalesce the on-register RA frame and
the test packet into one Read, in which case bytes.Equal fails and
the entire chunk (including the test packet's bytes) gets discarded
as "unexpected", leading to a 5s i/o timeout. Parse the QEMU uint32
length-prefix framing with io.ReadFull instead so we read exactly one
frame per iteration regardless of how the kernel buffers them. The
SOCK_DGRAM path (TestProtocolUnixDgram) keeps the original raw Read
since datagram boundaries are preserved.
These where the top two flakes in oss on the flakes dashboards.
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I32983656b692921a0f43a4a5e9a8a6ab2555ee49
Outbound packets produced by netstack (used by tailscaled with
--tun userspace-networking, by tsnet, and by the SOCKS5/HTTP proxies)
enter the wrapper via InjectOutbound{,PacketBuffer} and take the
injectedRead path, which bypasses Filter.RunOut.
RunOut's side effect for UDP/SCTP is to insert the reverse-flow tuple
into the connection-tracking LRU so that Filter.RunIn admits inbound
replies that no explicit ACL rule covers. Skipping it on the injected
path meant a netstack-side dial of UDP would send fine but the reply
would be dropped as "no matching rule". The kernel-TUN path was
already fine because it goes through RunOut.
Fixes#14229Fixes#20064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I816ef55c493a12ff4f561cd89c095559b5c2743b
The Logger previously took a *netmap.NetworkMap at Startup and on every
ReconfigNetworkMap call, denormalizing it into per-IP and self lookup
maps. That denormalization is O(n) over all peers and ran on every
netmap update, contributing to the broader quadratic behavior we want
to eliminate when a single peer is added or removed.
Instead, this makes netlog ask LocalBackend (well, nodeBackend) for
the info it needs, letting us remove the netmap.NetworkMap type
entirely from the netlog package.
This is a dependency to removing the netmap.NetworkMap type from
upstream callers, like wgengine.Engine in general.
Updates #12542
Change-Id: Ib5f2de96e788a667332c0a6f7ac833b3d0053b5c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The earlier aa5da2e5f2 made peer adds and removes through a netmap
delta path that mutates only nodeBackend, on the assumption that
PeerForIP, lookupPeerByIP, the engine's wireguard config
(e.lastCfgFull), the engine BART, wgdev's PeerLookupFunc closure, and
the engine's cached netmap (e.netMap) would all stay correct without
further updates. They don't. I'd totally forgotten that
Engine.PeerForIP has its own alternate IP-to-peer lookup codepath.
Concretely, all of these failed for a peer that arrived via
[tailcfg.MapResponse.PeersChanged] (and never via a full
[tailcfg.MapResponse.Peers] list):
- [wgengine.Engine.PeerForIP] read from e.netMap and e.lastCfgFull
(neither updated on the delta path) and so missed the new
peer. The rando non-data-plane callers (Ping, TSMP, pendopen,
debug endpoints, tsdial.Dialer.UseNetstackForIP for tsnet and
onlyNetstack tailscaled) all returned "no matching peer".
- The engine BART (built from e.lastCfgFull) missed the new peer's
subnet routes / exit-node default routes.
- wgdev's [device.PeerLookupFunc] closure (rebuilt only inside
wgcfg.ReconfigDevice) didn't have the new peer's noise key, so
outbound encryption to the new peer dropped the packet even when
SetPeerByIPPacketFunc returned the right NodePublic.
- And nothing in the delta path triggered NodeMutationRemove to
flow through to authReconfig either, so the same stale state
pointed at removed peers indefinitely.
So just (functionally) revert it for now, to have something easily
cherry-pickable to the 1.100 release branch. Proper fixes can come later
for the next release.
This also adds three new tests:
- TestPingPeerLearnedViaDelta runs disco and TSMP subtests over a
delta-added peer with only self addresses. disco exercises the
cold PeerForIP path (magicsock); TSMP exercises the full data path
through wgdev encryption. Both fail without this fix.
- TestPingSubnetRouteOfDeltaPeer exercises a subnet-router peer
arriving via delta. With s1 in --accept-routes mode, an IP
inside the advertised CIDR must resolve to s2 and a TSMP ping
must round-trip. Hits the BART + lastCfgFull + wgdev staleness
in one go.
- TestPingSelfReturnsIsLocalIP is a regression guard for the
IsSelf early-out in Engine.Ping. Passes on main today; included
here so future refactors of PeerForIP can't regress self
handling without test breakage.
Updates tailscale/corp#43394
Change-Id: I7a049271359bd73e7147ae9e2554e85614c2b8d2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
I previously (in #20096) had only considered the tailscaled deps
and forgot about the CLI deps. This does the CLI ones too.
containerboot and k8s-operator aren't applicable because they build
from oss already.
Updates tailscale/corp#43243
Updates #20067
Change-Id: I66790f822b5d040e7fcf90feabca24669f69cf61
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We aren't supposed to be using CODEOWNERS as blocking
reviews, blocking global cleanups.
(This is why we want to move to go/policybot)
Updates tailscale/corp#13972
Change-Id: I380258e2d4ffd0720d57d891adab06c8ca388617
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add NotifyInProcessNoDisconnect for in-process IPN bus subscribers that
must apply every bus update. When such a subscriber falls behind, block
Notify production instead of sending the terminal fell-behind message and
closing the watch.
This is intentionally not available over LocalAPI, where a slow or stuck
out-of-process client should still be disconnected rather than allowed to
stall tailscaled. In-process callers that use the bit must keep their
callbacks fast and must not call back into LocalBackend from the callback.
Updates #20062
Change-Id: I730ad61a07475243bb226fba2262c1a3ded211ae
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
New-style IPN bus subscribers consume stateful delta streams. Reject
NotifyRateLimit when it is combined with those subscription bits so
tailscaled cannot merge or delay messages that clients need to apply in
order.
Also stop silently dropping notifications when a watcher falls behind.
Remove the watcher, replace its stale queue with one terminal ErrMessage
notification, and close the watch.
Updates #20062
Change-Id: Id9d402ea76f4011cd23f122adf62f30dd4b6f90b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds a fake vnet ACME service, TXT-backed SetDNS support, and a
VM test that fetches a certificate with tailscale cert, serves it with
tailscale serve, and verifies HTTPS from a second node.
This adds coverage motivated by #19915.
Updates #13038
Change-Id: Ie1e53409509337d81c8fbceb63f59f3dfbd48207
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We previously had two GOARCH variants for natlab testing
(gokrazy/natlabapp and gokrazy/natlabapp.arm64), but we didn't have
those variants for the "tsapp" (production, not testing) variant.
This adds arm64 for tsapp too. But there are two major users of arm64
for gokrazy: Raspberry Pis, and cloud VMs. So make two variants.
Updates #1866
Change-Id: Ib9efaf1255101e3cdbc31d49c76fd86dcba21bb1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds support for Gokrazy GAF (Gokrazy Archive Format) zip
auto-updates, starting to wire up Tailscale's clientupdate mechanism
to Gokrazy's update mechanism.
Currently there's just a CLI command to update from a GAF URL,
with an --unsigned flag for use in a new natlab vmtest.
Next step would be publishing unstable track GAF files on
pkgs.tailscale.com, with detached signatures, and then making the
clientupdate mechanism also download those and check signatures.
Updates #20002
Change-Id: Ib03c56f17a57f8a4638398ef83549dac4813323d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Per chat with an SSH client author who wanted a URL in the output.
And then make it more clear what parts are banners.
Updates #cleanup
Change-Id: If5033ad9dc0dba3d833f24ea39e117a455010492
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test was asserting that a tailnet ping between two nodes traversed
DERP rather than going direct. But that wasn't really the point of the test,
and I kept forgetting ways that magicsock could find direct paths and
thus break this test.
So loosen it.
We really just want to see whether DERP worked at all and was used in the process
of getting a ping through, whether it was direct or not.
And that "tailscale debug derp" worked at all, which was what the bug
was about to begin with.
No need for all the "must be over DERP" stuff.
Updates #15579
Change-Id: I70ca63dc10919efa3d193b7af1d31a4a3b9d3950
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
vnet only ever sent IPv6 RAs in response to a Router Solicitation. In
practice this meant gokrazy VMs running with a dual-stack LAN never
installed vnet's IPv6 default route: gokrazy brings the link up via
DHCPv4 and the kernel never emits an RS on its own under that init
path. Off-link IPv6 destinations like the fake DERP servers were
therefore unreachable from any gokrazy test node that also had v4
on the same interface. (Pure-v6 nodes happened to work because the
kernel sends an RS as part of v6-only autoconf.)
Fix this in two complementary ways:
- Send an unsolicited RA every 5s to the link-local all-nodes group
on every v6-enabled network. This matches what real routers do
(RFC 4861 §6.2.1, MaxRtrAdvInterval; we use a much shorter
interval than the spec's 200s default so short-lived tests don't
have to wait).
- Send a unicast RA to a newly-registered MAC as soon as a client
first transmits on the wire. Without this the first periodic RA
can land before any VM has connected and the next one isn't
until the next tick, which can be longer than the test runs.
Factor the RA serialization out into buildIPv6RouterAdvertisement so
the solicited, periodic, and per-client paths all share one body.
Update TestSelfSignedDERPHashPinning to use a dual-stack hard-NAT
builder and assert zero errors from DebugDERPRegion (instead of
filtering "over IPv6" errors as it had to before this change). The
new builder also sets TS_DEBUG_STRIP_ENDPOINTS=1 on tailscaled so
disco can't find a direct path: without endpoint stripping, the now-
working non-NATted IPv6 LAN gives the two hard-NAT'd nodes a direct
route, defeating the test's "must traverse DERP" assertion. (Hard
NAT alone was enough before this change because v6 routing was
broken.) Also update sendBetweenClients in the vnet unit tests to
tolerate the new on-register RA noise on its read path.
Updates #13038
Updates #19973
Change-Id: Ic281dc53702a25fa773c46313f453837814233e8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
serveDebugDERPRegion built its TLS config with
ServerName: cmp.Or(derpNode.CertName, derpNode.HostName), which for a
"sha256-raw:<hex>" CertName passed the raw fingerprint to Go's stock
verifier as a hostname; the handshake always failed with a hostname
mismatch. This is the second half of #15579; the first half (tailscaled
itself failing with "unexpected multiple certs presented") was fixed in
Extract a tlsConfigForNode helper that mirrors derphttp.Client.tlsClient
so that sha256-raw and domain-fronting CertName values are dispatched
to tlsdial.SetConfigExpectedCertHash and tlsdial.SetConfigExpectedCert
respectively, falling back to HostName when CertName is empty.
The core fix here was originally written by @imnuke in #19965; that PR
also added a unit test in ipn/localapi/debugderp_test.go which is
replaced in this commit by a new vmtest that exercises the whole stack:
vnet now serves a self-signed cert valid for each fake DERP node's
HostName and exposes its SHA-256 fingerprint, and vmtest grows a new
SelfSignedDERPCertPinning EnvOption that swaps the test DERP map's
nodes to CertName="sha256-raw:<hex>" with InsecureForTests cleared.
TestSelfSignedDERPHashPinning then stands up two hard-NAT'd nodes, has
them communicate over DERP, and calls DebugDERPRegion on each. Before
this fix the test fails with the exact x509 hostname-mismatch error
from the original bug; after, it passes.
Updates #15579
Change-Id: I61f38ffebc7ac5abc962639db1ae88f5cd8633b1
Co-authored-by: Nuke <nuke@imnuke.dev>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Commit 2b338dd6a8 removed watchdogEngine because it was weird
(so many methods) and increasingly unnecessary after we'd cleaned up
and simplified so much of the locking.
This adds back a watchdog, but an easier to maintain one that's more
idiomatic.
Updates #19759
Change-Id: I86c458473e126c0809f37696446ce7acf4cc4eb9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The Tailscale daemon only refreshed TLS certs as a side effect of inbound
TLS handshakes or "tailscale cert" CLI calls. A node that doesn't see
inbound traffic during the renewal window silently rolls past expiry.
Add a once-per-hour background loop on LocalBackend that enumerates Serve
and Funnel HTTPS hostnames (filtered against the netmap's CertDomains so
we don't poke ACME for other nodes' service hostnames) and calls the
existing GetCertPEM path. The renewal decision (ARI window, then 2/3
expiry fallback) is unchanged; the loop just guarantees it runs.
For visibility during initial issuance or restart with a long-expired
cached cert, add a "tls-cert-pending" health Warnable that's set while
ACME is in flight and no usable cached cert exists. Async renewal of a
still-valid cert intentionally doesn't fire it. And then make the CLI "cert"
subcommand print out a warning if it's blocking due to a cert fetch
in flight, using that health info.
Fixes#19911Fixes#19912
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I144e46c40e957b2e879587decace32a523a6eade
Capture Auto.mapCtx while holding Auto.mu before using it for
incremental map update forwarding. Pause and restart paths can replace
the context under the same mutex, so using it after unlocking races
with those writers.
Add a race regression test for the UserProfiles path that repeatedly
cancels the map context while incremental profile updates are
forwarded.
Fixes#19953
Change-Id: Icc55c4a0dffbc16d6507a2b446b3909d4d0a0278
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds @alexwlchan's proposed "tailscale get" command that reads
current preference values, complementing "tailscale set". It uses the
same flag names as set.
tailscale get # show all settings as a table
tailscale get all # same
tailscale get accept-dns # show a single value
tailscale get --json # output as JSON object
tailscale get --set-flags # output as tailscale set argv
Fixes#11389Fixestailscale/corp#38702
Change-Id: Ie366f27f11ccc56c76fff9a94ed8a9de9c835bd0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, testwrapper only retried tests explicitly annotated with
flakytest.Mark. Authors don't pre-emptively mark tests that haven't
flaked yet, so the first flake of a brand-new test failed CI even
when a re-run would have passed.
testwrapper now retries every failing test within a per-test wall-clock
budget (default: 5 minute per-attempt timeout capped at 1.5x the first
failure duration, 10 minute total). A test that fails and then passes
on retry is reported as flaky; a test that never passes within the
budget remains a real failure (exit non-zero).
For flakeapp's existing log scraping, the wire format is preserved:
the "flakytest failures JSON:" line is now emitted only for tests
that ultimately flaked (passed on retry). Unmarked tests get a fake
issue URL of the form https://github.com/{owner}/{repo}/issues/UNKNOWN
where owner/repo is detected from GITHUB_REPOSITORY, the local git
remote, or falls back to tailscale/tailscale. A new "permanent test
failures JSON:" line is emitted for tests that never passed; flakeapp
ignores it for now (a follow-up can teach it to record real failures
separately).
flakytest.Mark stays as an opt-in API: still useful for tracking a
known-flaky test against a real issue and for TS_SKIP_FLAKY_TESTS.
Updates tailscale/corp#38960
Change-Id: I56dfc9b023486d239f60793a53e9690578ce8017
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds tsnet.Server.ListenSSH which, if the SSH feature is linked,
returns a net.Listener whose Accept yields *tailssh.Session values (as
net.Conn). This lets tsnet apps accept incoming SSH connections to
implement custom TUI applications.
Basic apps can use net.Conn directly (Read/Write/Close). Rich apps
import ssh/tailssh and type-assert for peer identity, PTY, signals,
etc. If feature/ssh isn't imported, ListenSSH returns an error.
Includes a demo guess-the-number game in tsnet/example/ssh-game.
Updates tailscale/corp#37839
Change-Id: I4e7c3c96afb030cdf4da8f2d8b2253820628129a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
If a user explicitly adds a non-ts.net (not a CertDomain domain) domain
like "foo.com" to their serve config as a web target that's also an allowed
funnel domain (using raw "tailscale serve set-config"), then use the new
ALPN cert fetching (from b553969b) to get certs for that domain.
This is just plumbing; there's no new product functionality to
actually enable this easily client-side, and it also has no visible
product surface to enable it server-side.
Updates tailscale/corp#41736
Change-Id: Ie2e421ac9611bce64bba3de6a454b2d505ea0e8a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a "tailscale whoami" subcommand that is equivalent to running
"tailscale whois $(tailscale ip -4)" but more ergonomic. It supports
the --json flag just like whois, and shares the WhoIsResponse
rendering code with whois.
Fixes#19907
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I8f33ba7a5608bab7dffa8213303beb5f345936d3
Parallel subtests share *ipn.Notify pointers (e.g. runningNotify).
When multiple subtests reached the same phase concurrently, they
all wrote to the shared notify's InitialStatus field without
synchronization, triggering the race detector.
Fix by shallow-copying *ipn.Notify before setting InitialStatus,
so each test iteration works on its own copy.
Updates #19380
Change-Id: I9dd40037e02146166f006f4f7c1ddcc47adba191
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, sharding required tests to opt in by calling tstest.Shard,
which used a process-global counter to assign each test to a shard.
This had two problems: most tests didn't call it, so they ran on every
shard (defeating the purpose), and shard assignments were unstable
(depended on call order, so adding a test could reshuffle others).
Remove tstest.Shard and tstest.SkipOnUnshardedCI entirely. Instead,
have testwrapper implement sharding automatically for all tests: when
TS_TEST_SHARD=N/M is set, it uses "go list -json" (no compilation) to
find test source files, scans them for top-level Test/Benchmark/
Example/Fuzz function names, and filters by fnv32a(name) % M == N-1.
The filtered names are passed as an anchored -run regex to go test.
Using go list instead of "go test -list" avoids linking the test binary
twice (Go's build cache does not cache test binary linking).
Fixes#19886
Change-Id: I62ab7b3d757324d4c5fd0b5de50c1e3742681791
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Some tests in another repo were broken by tailscale/tailscale#19607.
This fixes them, by finishing off the rest of the migration away from
netmap.NetworkMap on the IPN bus in containerboot.
Containerboot used to rebuild a full NetworkMap-shaped view while
reacting to IPN bus notifications. Now it insteads has its own
netmapState type (immutable) of exactly what it needs to track, and
sends those immutable values around, making cheap edits of new
immutable values when an IPN bus edit arrives.
This should make cmd/containerboot scale to much larger tailnets now too.
Fixes#19852Fixestailscale/corp#42347
Updates #12542
Change-Id: I88adaf061f85f677f954a764935e6654329d75a6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Use TLS-ALPN-01 for Funnel certificate renewals only when the node
already has a cached certificate, and fall back to DNS-01 with a fresh
order if the ALPN path is unavailable or fails.
Dynamically advertise acme-tls/1 only while an ACME challenge
certificate is pending, and add client metrics for DNS-01 and
TLS-ALPN-01 start/success/failure paths.
Updates tailscale/corp#41736Fixestailscale/corp#42320
Change-Id: I5adc6ea129237f9ef592f84fc1a8953c80bc9d5c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
NodeMutationAdd was a misleading name: a PeersChanged entry in a
MapResponse can represent either a truly new peer or a full
replacement for an existing peer that couldn't be expressed as a
PeerChangedPatch. Calling it "Add" implied it was always a completely
new node, which is wrong. (I'd changed my mind on the design of
mapping add/delete events to NodeMutations halfway through #19607 and
forgot to update the name, even though I'd updated half the docs)
Rename it to NodeMutationUpsert to reflect the actual semantics: the
node should be inserted or replaced in the peer map regardless of
whether it already existed.
Updates #19607
Updates #12542
Change-Id: Iebd3daddb3318cba02e115a1b184fcb3ee8f83d6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The prior aa5da2e5f2 ("process node adds/removes in constant
time") commit missed a bus notification case, where new-style
subscribers set NotifyNoNetmap and then the controlclient map routing
sends a full update (rather than a delta). Those profiles + peers
need to be put on the bus too.
I noticed this only when porting the Android app over to use the
new bus stuff.
Updates #19607
Updates #12542
Change-Id: I82c35011d2c532222ca27f7d4e790522c31bd156
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
In aa5da2e5f2 we made the IPN bus include deltas, including the
PeersRemoved, sending a slice of integer NodeIDs that were
removed. But when updating xcode, I realized there was no way to map
those integers to the stable node IDs used in other places.
I was consdering changing the just-added ipn.Notify.PeersRemoved from
an IntID to a string StableID, but then it doesn't match the MapResponse
wire protocol, which we've tried to match so far.
Instead, just add the integer ID as well. Callers can use whichever
world they want, having both. It's a little regrettable that we still
have two worlds of IDs, but oh well. Neither is really suitable to a
hypothetical future fully federated world of control servers anyway,
so we'll need a third type later anyway, so just live with the two we
have for now.
Updates #12542
Change-Id: Ib8fd48a265e1da1f8779152f141f624a7f7260e9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
For large tailnets (~50k+ nodes) with frequent peer churn (ephemeral
GitHub Actions workers etc.), tailscaled used to rebuild the full
netmap and fan it out on the IPN bus on every MapResponse that
added or removed a peer. There were two O(N) costs per delta: the
full netmap rebuild + every Notify.NetMap encode to every bus watcher.
This change tackles both:
1. Plumb O(1) peer add/remove through the delta path. PeersChanged
and PeersRemoved no longer prevent the delta happy path; instead,
they mutate the per-node-backend peer map in place.
2. Restrict ipn.Notify.NetMap emission to the platforms whose host
GUIs still depend on it (Windows, macOS, iOS) and migrate
in-tree consumers off it everywhere else:
- Migrate reactive consumers (containerboot, kube agents,
sniproxy, tsconsensus, etc.) off Notify.NetMap to the
previously-added Notify.SelfChange signal so they no longer
have to subscribe to the full netmap.
- Add ipn.NotifyNoNetMap so GUI clients on "legacy-emit" platforms
that have already migrated can opt out of the per-watcher
NetMap encode.
- Gate Notify.NetMap emission on the producer side by a compile-
time GOOS check, so the supporting code is dead-code-eliminated
on Linux and other geese where no GUI consumer needs it.
Re-running BenchmarkGiantTailnet from tstest/largetailnet, which was
added along with baseline numbers on unmodified main in ad5436af0d,
the per-delta cost (one peer add+remove pair) is now ~O(1) regardless
of tailnet size N:
N no-watcher (ms/op) bus-watcher (ms/op)
before now factor before now factor
10000 32 0.11 300x 166 0.13 1300x
50000 222 0.11 2000x 865 0.13 6700x
100000 504 0.12 4100x 1765 0.13 13400x
250000 1551 0.12 12500x 4696 0.15 32400x
Updates #12542
Change-Id: I94e34b37331d1a8ec74c299deffadf4d061fda9e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
SetDERPMap spawns a goroutine that calls ReSTUN, which logs via the
test logger. If the test returns before that goroutine logs, the
goroutine races with testing cleanup.
Use tstest.WhileTestRunningLogger so the goroutine's logf call becomes
a no-op once the test finishes.
Fixes#19829
Change-Id: I1097f98e40ffd1c5dd7fb7a715c918255853e3c6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
When tailscaled is running in userspace-networking mode behind an
exit node (e.g. as a SOCKS5 proxy), it resolves a hostname and then
dials a single resolved IP through the tunnel. If the name has both
A and AAAA, Go's net.Resolver merges them and we pick ips[0], which
on an IPv6-native host is usually AAAA. If the exit node has no IPv6
egress (or vice versa), the dial fails silently through the tunnel
and the user sees a hang.
Resolve all candidates and race connect attempts across address
families with a 300ms happy-eyeballs delay, matching Go's net.Dialer
default and the existing pattern in net/dnscache (commit ee0a03b14).
First success wins; losers are cancelled and any conns they produce
are closed. A failBoost channel wakes the launcher when a connect
fails fast (e.g. ICMP "no route" via the tunnel) so we don't sit on
the 300ms timer when the answer is already known.
userDialResolve is refactored into userDialResolveAll (returns the
full candidate list) plus a thin single-IP wrapper for callers like
UserDialPlan that don't race. UserDial's per-IP dispatch (netstack
vs peer dialer vs SystemDial vs std) is extracted to dialOneUser so
each candidate can route correctly on its own merits.
Also fix serveDial in localapi to pass the original hostname to
UserDial rather than a pre-resolved IP, so the race can fire.
This fix is single-ended: it works against any exit node, including
old ones, with no protocol changes. The trade-off versus filtering
on the exit-node side via PeerAPI DoH is that every dial through an
unreachable-family exit node costs one failed connect attempt per
cache window, rather than zero, which is acceptable given the
simplicity.
Fixes#19792Fixes#13257
Change-Id: I9d7645d0034caf3ee22ecdd8070798353f77e94b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add Go tests that drive a real headless Chromium (via chromedp) against
the built cmd/tsconnect/pkg/ artifact and verify the @tailscale/connect
public API surface end-to-end. The package has not been republished in
three years, in part because no test exercises the produced artifact at
runtime — only tsc --noEmit and a Go build run in CI.
TestCreateIPN loads pkg.js into the browser, calls createIPN with a junk
auth key, and asserts that pkg.createIPN / pkg.runSSHSession are
functions and that createIPN() returns an IPN with the documented
run/login/logout/ssh/fetch methods. No control-plane traffic.
TestFetchTailnetPeer stands up a full local tailnet (testcontrol +
DERP + a tsnet.Server peer) and verifies that the browser-side WASM
client can join over WebSocket-noise to the same control, connect to
DERP over WSS, and then ipn.fetch() an HTTP service hosted on the tsnet
peer through the tailnet. The test asserts the response body matches a
known string. Browser state transitions are logged: NoState -> NeedsLogin
-> Starting -> Running.
Tests are opt-in via --run-headless-browser-tests (matching the existing
--run-vm-tests pattern in tstest/natlab/vmtest) so they never fire in
casual `go test ./...` runs. When the flag is set, a test is skipped if
cmd/tsconnect/pkg/ has not been built, and fails with t.Error if no
chromium binary is found on $PATH (honoring $CHROME_BIN as an override).
findChromium also falls back to /Applications/Google Chrome.app and
/Applications/Chromium.app on darwin, since macOS Chrome's executable
lives inside an .app bundle and is not on $PATH by default. The
.github/workflows/test.yml wasm job is extended to install
google-chrome-stable and run the tests with the flag after build-pkg.
To prevent silently testing a stale pkg/main.wasm (built from an older
checkout than the rest of the test invocation), build-pkg now writes
pkg/build-info.json recording the sha256 of the raw (pre-wasm-opt)
go-build output. The test does its own `go build` of
cmd/tsconnect/wasm with the same -tags/-trimpath/-ldflags (factored
into a new cmd/tsconnect/wasmbuild package shared by both call sites)
and t.Fatalfs with a "rebuild" instruction on mismatch. Cost is
near-zero because the Go build cache from the prior build-pkg makes
the rebuild a cache hit.
The new wasmbuild package also replaces cmd/tsconnect's hardcoded -tags
string with a minimal-feature-set computation. wasmbuild.Keep names the
small set of feature/featuretags entries the browser client actually
needs (netstack, logtail, dns, health, c2n, ipnbus); wasmbuild.Tags()
emits a ts_omit_<f> for every other
omittable feature in feature/featuretags.Features, with transitive deps
expanded via featuretags.Requires. An init() panics if Keep references
a feature unknown to feature/featuretags so a rename there fails
loudly. Net effect on size: 32M raw / 9.4M brotli before this change,
25M raw / 4.4M brotli after — vs the last-published 1.39.98 at 21M /
3.8M. The transitive package-import graph is unchanged (176
tailscale.com/* packages either way): featuretags omits eliminate
dead code via `const HasX = false`, not imports. Trimming the import
graph would require a separate, larger refactor splitting interface
packages by build tag.
Writing TestFetchTailnetPeer surfaced several real issues, all fixed
here:
* cmd/tsconnect built the wasm with the nethttpomithttp2 tag, but
control/ts2021 (since commit 1d93bdce2, "control/controlclient:
remove x/net/http2, use net/http", Oct 2025) requires HTTP/2 from
net/http's bundled implementation. With nethttpomithttp2 set, the
bundle is excluded and the wasm client cannot speak HTTP/2 to any
control plane, including production. Drop the tag. Wasm size grows
~1 MB raw / ~300 KB brotli (more than offset by the feature
pruning above). The last published @tailscale/connect (1.39.98,
early 2023) pre-dates the regression, which is why no consumer has
reported the breakage.
* tstest/integration/testcontrol.Server's /ts2021 noise upgrade
endpoint rejected anything but POST. WebSocket clients (the only
transport available to browser-WASM) come in as GET. Allow both;
the controlhttp AcceptHTTP path dispatches on the Upgrade header,
so the websocket library still enforces GET for WS upgrades.
This matches production, where the same controlhttpserver.AcceptHTTP
routes purely on the Upgrade header without checking method.
* derp/derphttp's urlString built the DERP URL from node.HostName
only, dropping node.DERPPort. Non-WS clients use a separate code
path (connectToHost) that honors DERPPort, but WebSocket-only
clients (browser-WASM) went through urlString and so could not
reach a DERP running on any port other than 443. Include the port
when it differs from the scheme default.
Also move addWebSocketSupport from cmd/derper (where it was main-only)
to derp/derpserver.AddWebSocketSupport so tstest/integration.RunDERPAndSTUN
can wrap its DERP handler with WebSocket support — without that, the
test DERP would not accept the browser's wss connection.
Fixes#9394
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iff9cdee303e3b239924249b5bffb2fd04e02f391
A data race in a package matters more than any individual test
result. Two related problems:
1. Where go test's race detector text ("WARNING: DATA RACE" plus
the goroutine stack traces) lands in JSON output is timing-
dependent: it can be attributed to a test that ends up reporting
PASS (e.g. when the racing goroutines outlive the test that
spawned them and TSan prints during a different test's window).
testwrapper's main loop only flushes the logs of failed tests,
so the race report ends up stuck in a passing test's buffer and
is silently dropped. The race builders just see a bare
"FAIL\nFAIL\tpkg\ttime".
2. If the failing test in such a package happens to be marked flaky,
testwrapper retries it. That is the worst possible response to a
race: the flaky test might not even be the racy code, and a
second run without the racy goroutines could "succeed" while
hiding the real bug.
Address both: scan every output line for the race detector's first-
line marker. Track whether the package observed a race at all, on
the pkgFinished testAttempt. When a race was seen, fold every per-
test log buffer into the package-level logs (so the full report
surfaces from the existing pkg-fail flush path), and drop any
flaky-test retry plans for that package so we fail immediately
instead of running another attempt.
Two new tests:
- TestRaceSuppressesFlakyRetry verifies that a flaky test alongside
a racy test does NOT get retried.
- TestRaceAttributedToPassingTest verifies that a race attributed by
test2json to a passing test still surfaces in the output.
Also add a corpus of captured raw test binary outputs under
cmd/testwrapper/testdata/, with one subdirectory per scenario,
documenting the six representative shapes that go test -race can
emit (race in test body, race in goroutines that outlive a test,
race forced into a later test, race in TestMain post-m.Run, and a
parallel-tests split-attribution case via a "=== NAME" redirect
line). See its README.md for details.
Fixes#19603
Change-Id: Ifbfcd67fb3b1882c4907bd9cb2d68a8b5a91dd54
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The Engine watchdog wrapped every wgengine.Engine method call in a
goroutine with a 45s timeout and crashed the process on timeout. It
was added years ago to surface deadlocks during development, but the
underlying deadlocks have long since been fixed, and even when it did
fire it produced obscure stack traces (from inside the watchdog
goroutine, not the original caller) without buying much.
Audit of userspaceEngine's methods shows none have cyclic locking or
unbounded blocking now that ResetAndStop no longer loops waiting for
DERPs to drain (fa49009ee). The watchdog is dead weight; remove it
along with the TS_DEBUG_DISABLE_WATCHDOG escape hatch.
Updates #19759
Change-Id: Iba9d718fe1f8718a6631296e336b138c31b99ff1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
cibuild.On() returns true for any CI environment that sets CI=true,
including Alpine Linux's package build CI. TestTsgoRevInCacheKey was
guarded by cibuild.On() (or use of tsgo), so it ran under Alpine's CI
with stock Go, where go.toolchain.rev isn't blended into build cache
keys, and unsurprisingly failed.
Add cibuild.OnTailscaleCI, which keys off GITHUB_REPOSITORY_OWNER to
distinguish tailscale/tailscale's own GitHub Actions CI from arbitrary
downstream CI, and use it in TestTsgoRevInCacheKey.
Fixes#19754
Change-Id: Id31cfe71903a235f1460dca1e2fdf334e3ba1ee5
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Since f343b496c3 ("wgengine, all: remove LazyWG, use wireguard-go
callback API for on-demand peers"), Reconfig is fully synchronous:
magicConn.UpdatePeers, wgdev.RemovePeer, router.Set, and dns.Set all
return when the work is done, and the peer list is updated under
wgLock before Reconfig returns. So after Reconfig with empty configs,
len(st.Peers) is already 0.
The old loop also waited for st.DERPs to drain to 0, but UpdatePeers
only edits maps; active DERP connections idle out on their own
timeout. The sole caller (LocalBackend.stopEngineAndWait) doesn't
inspect st.DERPs anyway; it just hands the Status to
setWgengineStatusLocked. So the drain-wait was for nothing observable
and could theoretically (or at least appear to readers to) loop
forever holding b.mu. Remove that reader confusion by removing
the backoff loop entirely.
Updates #19759
Change-Id: Ibfac3f0baabcad7604b713c934a8fc37932e0a50
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a VM-based natlab test that exercises the peer-relay feature
(feature/relayserver) end-to-end across three Tailscale nodes whose
network topology makes a direct A<->B UDP path impossible: both peers
are behind HardNAT (FreeBSD/pfSense-style endpoint-dependent NAT) with
no port-mapping services, while the relay node is behind One2OneNAT so
its STUN-discovered WAN endpoint is reachable from both peers. The
test enables the relay server via EditPrefs, then waits for an a->b
PingDisco whose PingResult.PeerRelay is set (proving magicsock chose
the peer-relay path, not DERP), and finally asserts that the relay's
DebugPeerRelaySessions LocalAPI reports the session.
The existing TestPeerRelayPing in tstest/integration runs three
tailscaled processes on the loopback interface with no NATs; this new
vmtest covers peer relay through real per-VM kernels and NATs.
To wire control-server capabilities into vmtest, also add a
PeerRelayGrants() EnvOption (sibling of AllOnline,
SameTailnetUser) that flips testcontrol.Server.PeerRelayGrants so the
wildcard packet filter grants tailcfg.PeerCapabilityRelay and
PeerCapabilityRelayTarget; without those caps magicsock won't consider
any peer a candidate relay.
Updates #13038
Change-Id: Ib3440b83ec442da0d3b89ffa48ceea9398ea9062
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Server.clientsAtomic was introduced in 6b729795c3 as a lock-free
mirror of Server.clients to skip Server.mu on the packet send hot
path. This drops the non-concurrent map and makes all the existing
callers of the old plain map just use the concurrent map, but still
holding Server.mu.
BenchmarkLookupDestHashTrie is unchanged at ~2ns/op.
Fixes#19726
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0894e4d86914d152b9b5fef969a3184bcb96f678
Replace the process-global Server.mu lookup in the packet send hot path
with a global hashtriemap mirror of local clientSet entries. The
authoritative clients map remains guarded by Server.mu; clientsAtomic is
only a lock-free fast path for active local clients.
Misses, stale inactive client sets, duplicate accounting, and mesh
forwarding still fall back to lookupDestUncached. This avoids taking
Server.mu for the common local active-client send path, at the cost of
adding one global concurrent map that mirrors Server.clients for local
peers.
The benchmark uses four destination peers. The before run sets
TS_DEBUG_DERP_DISABLE_PEER_HASHTRIE=true to force the old mutex lookup
path; the after run uses the hashtrie fast path.
goos: linux
goarch: amd64
pkg: tailscale.com/derp/derpserver
cpu: Intel(R) Xeon(R) 6975P-C
│ before │ after │
│ sec/op │ sec/op vs base │
LookupDestHashTrie-16 176.050n ± 1% 1.904n ± 6% -98.92% (p=0.000 n=10)
│ before │ after │
│ B/op │ B/op vs base │
LookupDestHashTrie-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ before │ after │
│ allocs/op │ allocs/op vs base │
LookupDestHashTrie-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
Updates #3560 (very indirectly, historically)
Updates #19713 (as an alternative to that PR)
Change-Id: Ifb72e5c9854ad00e938cd24c6ab9c27312f297e8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
macOS limits Unix socket paths to 104 bytes. The Go test TempDir
path (e.g. /var/folders/.../TestDirectConnection...679197086/001/)
easily exceeds that, causing "bind: invalid argument". Create a
short /tmp/vmtest* directory for all socket files (vnet, QMP,
dgram) so the paths stay well under the limit on every platform.
Updates #13038
Change-Id: I721d24561d1766aaa964692bc77f40a131aa9455
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
startCloudQEMU hardcoded -machine q35,accel=kvm and -cpu host,
which fails on any host without KVM (notably macOS). Replace
with a qemuAccelArgs helper that probes /dev/kvm and falls back
to QEMU's TCG software emulation, matching the pattern already
used by tstest/integration/nat. Also wire the helper into
startGokrazyQEMU so gokrazy VMs pick up KVM when available.
Updates #13038
Change-Id: I7745518db823279b1880957bb14ca2ffdaab4c50
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The natlab vmtest suite (tstest/natlab/vmtest) and the integration nat
tests are gated behind --run-vm-tests because they need KVM and are
slow. Until now nothing in CI exercised them apart from a single
canary TestEasyEasy run on every PR.
Add .github/workflows/natlab-test.yml that runs the full opt-in suite
on demand (workflow_dispatch), on PRs labeled "natlab", and on main
every 12 hours via cron. The workflow has two phases:
- "prepare" builds the gokrazy VM image, downloads the Ubuntu and
FreeBSD cloud images once via the new natlabprep tool, and emits
a dynamic JSON matrix of every TestX function it finds in the two
opt-in packages.
- "test" is a per-test matrix that depends on prepare. Each matrix
job restores the shared caches and runs a single test, so adding
a new TestFoo is automatically picked up on the next run without
any workflow edits.
Rename the existing natlab-integrationtest.yml to natlab-basic.yml
since it's the small smoke variant (just TestEasyEasy on every PR);
the new natlab-test.yml is the bigger suite. The job inside is
renamed to EasyEasy for the same reason.
Move the macOS arm64 host check from vmtest.Env.Start into
vmtest.Env.AddNode so a test that adds a vmtest.MacOS node skips
immediately on a non-macOS host, and add an explicit
skipIfNotMacOSArm64 helper at the top of the two macOS-only tests
so the platform requirement is obvious to readers.
Quiet the takeAgentConnOne miss log in tstest/natlab/vnet by default
(it was the overwhelming majority of bytes in CI logs, with no signal
in healthy runs) and replace it with a periodic "still waiting" line
that only fires after 10s, so a truly stuck agent connection still
surfaces.
Updates #13038
Change-Id: I4582098d8865200fd5a73a9b696942319ccf3bf0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The tailscale.com/wif package brings in the AWS SDK
(github.com/aws/aws-sdk-go-v2/{config,sts,...} and github.com/aws/smithy-go)
to support fetching ID tokens from AWS IMDS for workload identity
federation. Until now, tsnet pulled this in unconditionally via
feature/condregister/identityfederation, costing ~70 unwanted deps for
every tsnet program whether or not it uses workload identity federation.
These AWS SDK deps were originally removed from tsnet on 2025-09-29 by
commit 69c79cb9f ("ipn/store, feature/condregister: move AWS + Kube
store registration to condregister"). They were then accidentally added
back on 2026-01-14 by commit 6a6aa805d ("cmd,feature: add identity
token auto generation for workload identity", PR #18373) when the new
wif package was wired into tsnet via feature/identityfederation.
Drop the blanket import. tsnet programs that want workload identity
federation now opt in with:
import _ "tailscale.com/feature/identityfederation"
The hook lookup in resolveAuthKey already uses GetOk and degrades
gracefully when the feature isn't linked, so existing programs that
don't use workload identity federation see no behavior change. The
tailscale CLI still imports the condregister wrapper directly, so its
behavior is also unchanged.
Lock this in with TestDeps additions: tailscale.com/wif as a BadDep,
plus substring checks in OnDep that fail on any github.com/aws/ or
k8s.io/ dependency creeping back in.
Also, switch cmd/gitops-pusher from the condregister wrapper to a
direct import of feature/identityfederation: gitops-pusher's auth flow
calls HookExchangeJWTForTokenViaWIF directly, so it shouldn't be
subject to the ts_omit_identityfederation build tag.
Updates #12614
Change-Id: I70599f2bdd4d3666b26a859d5b76caa5d6b94507
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Per recent chat with @raggi about all this, I went and looked at this
test again.
Updates #cleanup
Change-Id: Icb7d87b1ed2cebf481ee4e358a3aa603e63fb8a4
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Commit 69c79cb9f (Sep 2025) moved awsstore and kubestore registration
behind condregister build tags so tsnet wouldn't pull in the AWS SDK
and Kubernetes client by default. The accompanying TestDeps BadDeps
entry was missed, so PR #19667 (which re-added those imports) wasn't
caught by the test.
Add the two packages to BadDeps so future regressions fail the test.
Updates #19667
Updates #12614
Change-Id: I903b7c976e5e122cc0c0b956dc73740f5d474fac
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test goroutine read lockCnt immediately after Lock returned, racing
with Close: close(lk.closing) wakes lockSlow's select, whose deferred
Add(-2) on lockCnt can run before Close's CAS clears the LSB. When that
happens, lockCnt is briefly 1 (3 - 2) instead of 0 (1 + 2 - 2 - 1),
producing "lockCnt: got 1; want 0".
Move the lockCnt assertion into the main test goroutine, after both
Close has returned and the Lock goroutine has finished, so both updates
have settled before we read.
Fixes#19647
Change-Id: Ia67036ff73a1beb528cbd621460db9048f3066ad
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The purpose of this package is to test the iOS dependency closure, but
it had drifted from the actual import list of the ipn-go-bridge package
in the corp repo (the Go side of the iOS / macOS app).
Update the imports to match ipn-go-bridge's GOOS=ios import list,
adding many missing packages including wgengine/netstack,
feature/{taildrop,syspolicy,condregister}, the util/syspolicy/*
subpackages, types/{key,lazy,logid,netmap}, tsd, safesocket,
util/{eventbus,must,set}, and several net/* and ipn/* packages.
Drop two now-stale BadDeps entries (for now!): database/sql/driver and
github.com/google/uuid are reached via wgengine/netstack ->
github.com/prometheus-community/pro-bing, which netstack imports on
darwin || ios for ICMP user-ping, so the iOS app already ships them.
But we should fix that later.
Updates #19633
Change-Id: Ic50779fdb195685a2e8ccd7c513eee91b0feeaf8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new vet checker that rejects variables, parameters, named
return values, receivers, range/type-switch bindings, type
parameters, struct fields, and constants named "l" (lowercase ell)
or "I" (uppercase i). Both are hard to distinguish from the digit
"1" and from each other in too many fonts.
Rename the two pre-existing struct fields named "l" (both of type
net.Listener) in drive/driveimpl/drive_test.go to "ln", matching the
convention used elsewhere for net.Listener locals.
Rename the test-fixture struct fields "I" (single int label) to
"Int" in metrics/multilabelmap_test.go and util/deephash/deephash_test.go,
preserving the "first letters of types" convention used alongside
neighboring fields like I8/I16/U/U8.
Also teach pkgdoc_test.go to skip testdata/ directories, which
the go tool ignores; they are not real packages.
Fixes#19631
Change-Id: I71ad2fa990705f7a070406ebcdb8cefa7487d849
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Stop opening an IPN bus subscription with NotifyInitialNetMap purely to
read the current netmap once. Use the LocalAPI debug current-netmap
action (added in 159cf8707) instead, which returns the current netmap
synchronously without subscribing to the bus.
Updates #12542
Change-Id: I8aa2096d65aaea4dfe62634f03ce06b5470e0e51
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Move tailscaled's in-tree reactive users from of IPN bus Notify.NetMap
updates to the narrower Notify.SelfChange signal introduced earlier in
this series. Consumers that need additional state (peers, DNS config,
etc.) fetch it on demand via the LocalAPI.
It is a step toward the larger goal of not fanning Notify.NetMap out
to every bus watcher on Linux/non-GUI hosts.
A future change stops sending Notify.NetMap entirely on Linux and
non-GUI platforms. (eventually once macOS/iOS/Windows migrate to the
upcoming new Notify APIs, we'll remove ipn.Notify.NetMap entirely)
Updates #12542
Change-Id: I51ea9d86bdca1909d6ac0e7d5bd3934a3a4e8516
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a narrow LocalAPI accessor and matching client/LocalBackend method
to look up a single peer's current full [tailcfg.Node] by NodeID, in
O(1) time on the daemon side, without fetching the entire netmap.
Useful for callers that need the latest state of a single peer (e.g.
in response to a peer-mutation event on the IPN bus) without paying
for a full netmap fetch.
Updates #12542
Change-Id: I1cb2d350e6ad846a5dabc1f5368dfc8121387f7c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new bus signal that lets reactive consumers (containerboot, kube
agents, sniproxy, tsconsensus, etc.) react to self-node updates without
having to subscribe to the full netmap. Today those consumers either
watch Notify.NetMap (which on large tailnets is expensive to encode and
ship per watcher) or poll. SelfChange is a cheap, narrow alternative:
addresses, name, key expiry, capabilities, etc.
Consumers that need additional state can react to SelfChange and then
fetch the relevant bits on demand via existing LocalClient methods.
Producer-side, every netmap-bearing setControlClientStatus call now
also publishes SelfChange. Future changes will migrate individual
in-tree consumers off Notify.NetMap to this signal, and eventually
gate the legacy NetMap emission to platforms whose host GUIs still
require it.
Updates #12542
Change-Id: I4441650b0e085d663eb6bf26a03748b7d961ca49
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add two narrow LocalAPI accessors so callers don't have to subscribe to
the IPN bus and pull a full *netmap.NetworkMap just to read DNS-shaped
fields:
- GET /localapi/v0/cert-domains returns DNS.CertDomains.
- GET /localapi/v0/dns-config returns the full tailcfg.DNSConfig.
Migrate in-tree callers off the netmap-on-the-bus pattern:
- kube/certs.waitForCertDomain still wakes on the IPN bus but now
queries CertDomains via LocalClient.CertDomains rather than
reading n.NetMap.DNS.CertDomains. The kube LocalClient interface
and FakeLocalClient gain a CertDomains method.
- cmd/tailscale dns status calls LocalClient.DNSConfig directly
instead of opening a NotifyInitialNetMap watcher.
- cmd/tailscale configure kubeconfig switches from a netmap watcher
+ serviceDNSRecordFromNetMap to LocalClient.DNSConfig +
serviceDNSRecordFromDNSConfig.
This is part of a series moving callers away from depending on the
netmap traveling on the IPN bus, so the bus payload can shrink in a
later change.
Updates #12542
Change-Id: Ie10204e141d085fbac183b4cfe497226b670ad6c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add two narrower accessors alongside the existing
[LocalBackend.NetMap], with docs that distinguish their semantics:
- NetMapNoPeers: cheap (returns the cached *netmap.NetworkMap with
a possibly-stale Peers slice). For callers that only read non-Peers
fields like SelfNode, DNS, PacketFilter, capabilities.
- NetMapWithPeers: documented as returning an up-to-date Peers slice.
For callers that genuinely need to iterate Peers or call
PeerByXxx.
Mark the existing NetMap deprecated and point readers at the two new
accessors. NetMap, NetMapNoPeers, and NetMapWithPeers all currently
return the same value (b.currentNode().NetMap()): this commit is a
no-op behaviorally, just a renaming and migration of in-tree callers.
A subsequent change in the same series will switch
NetMapWithPeers to actually rebuild the Peers slice from the live
per-node-backend peers map (O(N) per call), at which point the
distinction between the two new accessors becomes load-bearing.
Migrate in-tree callers to the appropriate accessor based on what
fields they read:
- NetMapNoPeers (most common): localapi handlers, peerapi accept,
GetCertPEMWithValidity, web client noise request, doctor DNS
resolver check, tsnet CertDomains/TailscaleIPs, ssh/tailssh
SSH-policy/cap reads, several LocalBackend internals
(isLocalIP, allowExitNodeDNSProxyToServeName, pauseForNetwork
nil-check, serve config).
- NetMapWithPeers: writeNetmapToDiskLocked (persist full netmap to
disk for fast restart), PeerByTailscaleIP lookup.
Tests still call the legacy NetMap; they'll see the deprecation
warning but otherwise behave identically.
Also add two pieces of plumbing the next change in this series will
need, but which are already useful on their own:
- [client/local.GetDebugResultJSON]: a generic [Client.DebugResultJSON]
that decodes directly into a target type T, avoiding the
marshal/unmarshal roundtrip callers otherwise need.
- localapi "current-netmap" debug action: returns the current
netmap (with peers) as JSON. Documented as debug-only — the
netmap.NetworkMap shape is internal and may change without notice.
This commit is part of a series breaking up a larger change for
review; on its own it is a no-op refactor.
Updates #12542
Change-Id: Idbb30707414f8da3149c44ca0273262708375b02
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Move the template, request handler, and HTTP/HTTPS server wiring out
of package main and into a new cmd/hello/helloserver package so the
server can be embedded in other binaries. The main package now only
constructs a helloserver.Server with the production addresses and
calls Run.
While here, drop the -http, -https, and -test-ip flags along with the
dev-mode template and fake-data fallbacks they enabled; the binary is
only run in production.
Updates tailscale/corp#32398
Change-Id: Id1d38b981733334cafc596021130f36e1c1eed67
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
If a user passes --advertise-tags=foo,bar (with no colons in any
segment), automatically prepend "tag:" client-side so it goes on the
wire as "tag:foo,tag:bar". Segments that already contain a colon are
left untouched and must be fully-qualified ("tag:foo"), which keeps
the door open for future colon-bearing syntax.
This was originally added in cd07437ad (2020-10-28) and then reverted
in 1be01ddc6 (2020-11-10) over forward-compatibility concerns. But
then it was realized in 2026-04-29 that this was always safe for
future extensiblity anyway (tags can't contain colons-- tag:foo:bar is
invalid anyway, per the 2020 CheckTag restrictions). So if we wanted
to perhaps some hypothetical --advertise-tags=tagset:setfoo or "group:foo",
we'd still have syntax to do, as it can't conflict with tag:group:foo.
Avery signed off on this on Slack: "Ok, I withdraw my objection to
auto-qualifying tag names in advertise-tags and I hope I won't regret
it :)"
Updates #861
Change-Id: I06935b0d3ae909894c95c9c2e185b7d6a219ff32
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Replace the UAPI text protocol-based wireguard configuration with
wireguard-go's new direct callback API (SetPeerLookupFunc,
SetPeerByIPPacketFunc, RemoveMatchingPeers, SetPrivateKey).
Instead of computing a trimmed wireguard config ahead of time upon
control plane updates and pushing it via UAPI, install callbacks so
wireguard-go creates peers on demand when packets arrive. This removes
all the LazyWG trimming machinery: idle peer tracking, activity maps,
noteRecvActivity callbacks, the KeepFullWGConfig control knob, and the
ts_omit_lazywg build tag.
For incoming packets, PeerLookupFunc answers wireguard-go's questions
about unknown public keys by looking up the peer in the full config.
For outgoing packets, PeerByIPPacketFunc (installed from
LocalBackend.lookupPeerByIP) maps destination IPs to node public keys
using the existing nodeByAddr index.
Updates tailscale/corp#12345
Change-Id: I4cba80979ac49a1231d00a01fdba5f0c2af95dd8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test was flaky under stress with "AddRawMapResponse N: node not
connected" failures. The root cause was in testcontrol's addDebugMessage:
it conflated "no streaming poll registered" with "wake-up channel buffer
momentarily full". The single-slot updatesCh is just a lossy wake-up
signal, but the streaming serveMap loop has fast paths
(takeRawMapMessage and the hasPendingRawMapMessage continue) that don't
drain it. A stale notification could remain buffered, causing the next
sendUpdate to fail even though msgToSend had been queued and the
streaming poll would still pick it up.
Detect the real failure case (no streaming poll) by checking
s.updates[nodeID] directly, and treat sendUpdate's buffer-full result as
benign — the message is in msgToSend, which is the source of truth.
Also plumb an optional *health.Tracker through tsp.ClientOpts to the
underlying ts2021.Client and supply one in the tests, eliminating the
"## WARNING: (non-fatal) nil health.Tracker (being strict in CI)" stack
dumps emitted by controlhttp.(*Dialer).forceNoise443 under CI.
Fixes#19583
Change-Id: Ib2334376585e8d6562f000a0b71dea0117acb0ff
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a vmtest that brings up two gokrazy nodes A and B behind two
One2OneNAT networks (so direct UDP works in both directions and any
slowness can't be blamed on NAT traversal), establishes a WireGuard
tunnel A → B with TSMP, then rotates B's disco key four times and
asserts that the data plane recovers in both directions after each
rotation. All pings are TSMP (the data-plane ping; disco pings would
not exercise the WireGuard tunnel itself).
The five pings:
1. A → B (initial; brings up the tunnel; 30s budget)
2. B → A after rotate (LocalAPI rotate-disco-key debug action)
3. A → B after rotate (LocalAPI)
4. B → A after restart (SIGKILL; gokrazy supervisor respawns)
5. A → B after restart (SIGKILL)
Each post-rotation ping gets a 15-second budget. Two unavoidable
multi-second waits dominate today:
- The rotate-then-a→b phase takes ~10s on main because of LazyWG.
After B's WantRunning bounce, B's wgengine resets its
sentActivityAt/recvActivityAt maps and trims A out of the
wireguard-go config as an "idle peer"; B only re-adds A on
inbound activity, by which point A's first few TSMP packets
have been silently dropped at B's tundev. The
bradfitz/rm_lazy_wg branch removes that trimming entirely
(verified locally: this phase drops to <100ms there).
- The restart phases take ~5s for wireguard-go's RekeyTimeout
handshake retry. After SIGKILL+respawn the first WG handshake
init from the restarted node sometimes goes into the void
(likely the brief peer-removed window in the receiver's
two-step maybeReconfigWireguardLocked reconfig during which
the peer is absent from wireguard-go), and wg-go's 5s+jitter
retransmit timer is the next opportunity to retry. That retry
succeeds and the staged TSMP packet flushes. Intrinsic to the
protocol's retransmit policy.
Once LazyWG is removed and the first-handshake-after-reconfig race
is fixed, the budget should drop to 5s.
Supporting changes:
ipn/ipnlocal: DebugRotateDiscoKey now toggles WantRunning off and
back on after rotating the disco key. magicsock.Conn.RotateDiscoKey
only resets local disco state; without also dropping wireguard-go
session keys, peers keep encrypting with their stale per-peer
session against us until their rekey timer fires (WireGuard has no
data-plane signaling to invalidate sessions). Bouncing WantRunning
runs the engine through Reconfig(empty) → authReconfig, which
drops every peer's WG session so the next packet either way
triggers a fresh handshake.
ipn/ipnlocal, ipn/localapi: add a debug-only "peer-disco-keys"
LocalAPI action ([LocalBackend.DebugPeerDiscoKeys]) that returns
a map[NodePublic]DiscoPublic from the current netmap. Tests reach
it via [local.Client.DebugResultJSON]. We do not surface disco
keys via [ipnstate.PeerStatus] because adding a non-comparable
[key.DiscoPublic] field there breaks reflect-based test helpers
(e.g. TestFilterFormatAndSortExitNodes' use of cmp.Diff), and
general LocalAPI clients have no need for disco keys. Since the
debug LocalAPI is gated behind the ts_omit_debug build tag, this
endpoint is automatically stripped from small binaries.
cmd/tta: add /restart-tailscaled handler (Linux-only, via /proc walk)
to drive the SIGKILL phase. On gokrazy the supervisor respawns
tailscaled within a second.
tstest/integration/testcontrol: add Server.AllOnline. When set,
every peer entry in MapResponses is marked Online=true. Several
disco-key handling fast paths in controlclient and wgengine
(removeUnwantedDiscoUpdates, removeUnwantedDiscoUpdatesFromFull
NetmapUpdate, the wgengine tsmpLearnedDisco fast path) only fire
for online peers; without this flag, tests exercising disco-key
rotation only hit the offline-peer code paths, which mask issues
and are several seconds slower in this scenario. Finer-grained
per-node online tracking can be added later.
tstest/natlab/vmtest: add Env.RotateDiscoKey,
Env.RestartTailscaled, Env.PeerDiscoKey, Node.Name, an
[AllOnline] EnvOption that plumbs through to
testcontrol.Server.AllOnline, and an exported
Env.Ping(from, to, type, timeout). Ping replaces the unexported
helper so callers can specify both a ping type (PingDisco for
warming peer state, PingTSMP for asserting end-to-end
connectivity) and a deadline. PeerDiscoKey returns its LocalAPI
error so callers inside tstest.WaitFor can retry transient
failures rather than fataling the test.
Updates #12639
Updates #13038
Change-Id: I3644f27fc30e52990ba25a3983498cc582ddb958
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Commit 78627c132f changed the signature of magicsock.Conn.SetDERPMap to
take an additional bool doReStun parameter. Avoid both the boolean
parameter and the API signature change by restoring SetDERPMap to its
original single-argument form and adding a new SetDERPMapWithoutReSTUN
method for the cache-loading caller that wants to skip the post-set
ReSTUN.
Updates #19490
Change-Id: I97d9e82156bfc546ccf59756d1ea52f039b5de06
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
