Move all the FooForTest methods on LocalBackend to instead be
methods on a new unexported forTest type which is then given out
to callers in other packages via an exported ForTest method
(panicking in non-test contexts) that returns that unexported type.
This is unusual style (exported returning unexported) but declutters
godoc and makes call sites both more explicit and easier to read
without the "ForTest" suffix polluting the symbols. Now FooForTest()
changes into ForTest().Foo().
This was motivated by a pending change moving a bunch of code out of
LocalBackend into other packages that required adding more ForTest
methods to LocalBackend to keep the tests (now in other packages)
working. Instead, do this refactor now so the future change is prettier.
Updates #12614
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib25e6d76d48dc8622ac3a955e0b1220d582e63a8
prometheus/common v0.66/v0.67 introduced a mandatory
model.ValidationScheme on expfmt.TextParser as part of
prepping for UTF-8 metric/label names in Prometheus 3.0. The
zero value is intentionally UnsetValidation, which panics on
the first call to IsValidMetricName / IsValidLabelName with
Invalid name validation scheme requested: unset
so the long-standing "var parser expfmt.TextParser" pattern
crashes at runtime. Several big downstreams have hit the same
sharp edge:
https://github.com/thanos-io/thanos/issues/8823https://github.com/grafana/loki/pull/21401
Switch our two callers (parseMetrics in tsnet's
TestUserMetricsByteCounters and the client-metrics scraper in
tstest/natlab/vmtest) to the new expfmt.NewTextParser
constructor with model.LegacyValidation. LegacyValidation
matches the classic ASCII metric/label naming rules that
tailscaled's exporter uses today; if and when we ever emit a
metric with a UTF-8 name, we can revisit.
Goes to v0.69.0 (the latest at the time of writing) rather
than v0.67.5 so we pick up the unrelated security fixes for
cross-host redirects.
Done in advance so a follow-up change can pull in
github.com/tailscale/policybottest (which depends on
palantir/policy-bot, which transitively requires
prometheus/common at v0.67+) without dragging this debugging
into that PR.
Updates tailscale/corp#13972
Change-Id: I4b37db9ad3bebef1a32d9020bf6f8790bab25336
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
* Revert "control/controlclient: continue map poll during key expiry to receive extensions"
This reverts commit 6a822dcc36. This commit
has caused test failures in the corp repo by unexpected changing the login
behaviour when nodes have a valid node key.
Updates tailscale/corp#43705
Updates #19326
Signed-off-by: Alex Chan <alexc@tailscale.com>
* Revert "tsnet: test key extension after server restart"
This reverts commit 317201375f. This test
relies on changes in 317201375f, which is
also being reverted because it causes test failures in corp.
Updates tailscale/corp#43705
Updates #19326
Signed-off-by: Alex Chan <alexc@tailscale.com>
---------
Signed-off-by: Alex Chan <alexc@tailscale.com>
Another baby step toward removing slices of peers from the engine.
getStatus iterated peerSequence (a key snapshot built in Reconfig
from cfg.Peers) and then asked wgdev for each peer's stats; peers
that weren't active in wgdev silently fell out. Iterate active wgdev
peers directly via RemoveMatchingPeers(returnFalse) instead.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3abd348abc30db706db29b3a785179259e48abda
Outbound packets produced by netstack (used by tailscaled with
--tun userspace-networking, by tsnet, and by the SOCKS5/HTTP proxies)
enter the wrapper via InjectOutbound{,PacketBuffer} and take the
injectedRead path, which bypasses Filter.RunOut.
RunOut's side effect for UDP/SCTP is to insert the reverse-flow tuple
into the connection-tracking LRU so that Filter.RunIn admits inbound
replies that no explicit ACL rule covers. Skipping it on the injected
path meant a netstack-side dial of UDP would send fine but the reply
would be dropped as "no matching rule". The kernel-TUN path was
already fine because it goes through RunOut.
Fixes#14229Fixes#20064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I816ef55c493a12ff4f561cd89c095559b5c2743b
In PR #17809, @bradfitz tried to fix tsnet_test.TestConn by making the
second tailscaled start after the first was fully set up. On slow
runners, the Ping for connectivity to the second server would race
against that server establishing a connection with its DERP home. If
the Ping arrived too soon, the DERP server would respond with
PeerGoneNotHome and the Ping would wait for its full timeout before
failing the test.
This patch introduces waitForHomeDERPConnected and makes startServer
block until the server’s home DERP has established its connection.
This patch also reduces the Ping timeout to 10 seconds for the tsnet
tests, which should be enough that a hung Ping is fast enough for
interactive debugging, but with enough headroom for a RekeyTimeout.
Fixes#12766
Signed-off-by: Simon Law <sfllaw@tailscale.com>
The earlier aa5da2e5f2 made peer adds and removes through a netmap
delta path that mutates only nodeBackend, on the assumption that
PeerForIP, lookupPeerByIP, the engine's wireguard config
(e.lastCfgFull), the engine BART, wgdev's PeerLookupFunc closure, and
the engine's cached netmap (e.netMap) would all stay correct without
further updates. They don't. I'd totally forgotten that
Engine.PeerForIP has its own alternate IP-to-peer lookup codepath.
Concretely, all of these failed for a peer that arrived via
[tailcfg.MapResponse.PeersChanged] (and never via a full
[tailcfg.MapResponse.Peers] list):
- [wgengine.Engine.PeerForIP] read from e.netMap and e.lastCfgFull
(neither updated on the delta path) and so missed the new
peer. The rando non-data-plane callers (Ping, TSMP, pendopen,
debug endpoints, tsdial.Dialer.UseNetstackForIP for tsnet and
onlyNetstack tailscaled) all returned "no matching peer".
- The engine BART (built from e.lastCfgFull) missed the new peer's
subnet routes / exit-node default routes.
- wgdev's [device.PeerLookupFunc] closure (rebuilt only inside
wgcfg.ReconfigDevice) didn't have the new peer's noise key, so
outbound encryption to the new peer dropped the packet even when
SetPeerByIPPacketFunc returned the right NodePublic.
- And nothing in the delta path triggered NodeMutationRemove to
flow through to authReconfig either, so the same stale state
pointed at removed peers indefinitely.
So just (functionally) revert it for now, to have something easily
cherry-pickable to the 1.100 release branch. Proper fixes can come later
for the next release.
This also adds three new tests:
- TestPingPeerLearnedViaDelta runs disco and TSMP subtests over a
delta-added peer with only self addresses. disco exercises the
cold PeerForIP path (magicsock); TSMP exercises the full data path
through wgdev encryption. Both fail without this fix.
- TestPingSubnetRouteOfDeltaPeer exercises a subnet-router peer
arriving via delta. With s1 in --accept-routes mode, an IP
inside the advertised CIDR must resolve to s2 and a TSMP ping
must round-trip. Hits the BART + lastCfgFull + wgdev staleness
in one go.
- TestPingSelfReturnsIsLocalIP is a regression guard for the
IsSelf early-out in Engine.Ping. Passes on main today; included
here so future refactors of PeerForIP can't regress self
handling without test breakage.
Updates tailscale/corp#43394
Change-Id: I7a049271359bd73e7147ae9e2554e85614c2b8d2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
tsnet depends on logpolicy, which in turn depended on util/syspolicy
because of a single LogTarget policy setting it uses.
In this commit, we replace that dependency with a feature.Hook,
which only tailscaled or its platform-specific alternatives should set.
Updates #20031
Signed-off-by: Nick Khyl <nickk@tailscale.com>
If we dispatch a ping too early (after a later patch removes a 250ms
blockage) then the ping may be lost due to the peers not yet knowing
about each other. The ping is retained in order to setup and ensure a
wireguard session prior to test flow.
Updates #19822
Change-Id: I6cfea28931646a9387b6ffc2654e72cd846f4e55
Signed-off-by: James Tucker <james@tailscale.com>
Co-authored-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, sharding required tests to opt in by calling tstest.Shard,
which used a process-global counter to assign each test to a shard.
This had two problems: most tests didn't call it, so they ran on every
shard (defeating the purpose), and shard assignments were unstable
(depended on call order, so adding a test could reshuffle others).
Remove tstest.Shard and tstest.SkipOnUnshardedCI entirely. Instead,
have testwrapper implement sharding automatically for all tests: when
TS_TEST_SHARD=N/M is set, it uses "go list -json" (no compilation) to
find test source files, scans them for top-level Test/Benchmark/
Example/Fuzz function names, and filters by fnv32a(name) % M == N-1.
The filtered names are passed as an anchored -run regex to go test.
Using go list instead of "go test -list" avoids linking the test binary
twice (Go's build cache does not cache test binary linking).
Fixes#19886
Change-Id: I62ab7b3d757324d4c5fd0b5de50c1e3742681791
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
For large tailnets (~50k+ nodes) with frequent peer churn (ephemeral
GitHub Actions workers etc.), tailscaled used to rebuild the full
netmap and fan it out on the IPN bus on every MapResponse that
added or removed a peer. There were two O(N) costs per delta: the
full netmap rebuild + every Notify.NetMap encode to every bus watcher.
This change tackles both:
1. Plumb O(1) peer add/remove through the delta path. PeersChanged
and PeersRemoved no longer prevent the delta happy path; instead,
they mutate the per-node-backend peer map in place.
2. Restrict ipn.Notify.NetMap emission to the platforms whose host
GUIs still depend on it (Windows, macOS, iOS) and migrate
in-tree consumers off it everywhere else:
- Migrate reactive consumers (containerboot, kube agents,
sniproxy, tsconsensus, etc.) off Notify.NetMap to the
previously-added Notify.SelfChange signal so they no longer
have to subscribe to the full netmap.
- Add ipn.NotifyNoNetMap so GUI clients on "legacy-emit" platforms
that have already migrated can opt out of the per-watcher
NetMap encode.
- Gate Notify.NetMap emission on the producer side by a compile-
time GOOS check, so the supporting code is dead-code-eliminated
on Linux and other geese where no GUI consumer needs it.
Re-running BenchmarkGiantTailnet from tstest/largetailnet, which was
added along with baseline numbers on unmodified main in ad5436af0d,
the per-delta cost (one peer add+remove pair) is now ~O(1) regardless
of tailnet size N:
N no-watcher (ms/op) bus-watcher (ms/op)
before now factor before now factor
10000 32 0.11 300x 166 0.13 1300x
50000 222 0.11 2000x 865 0.13 6700x
100000 504 0.12 4100x 1765 0.13 13400x
250000 1551 0.12 12500x 4696 0.15 32400x
Updates #12542
Change-Id: I94e34b37331d1a8ec74c299deffadf4d061fda9e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The tailscale.com/wif package brings in the AWS SDK
(github.com/aws/aws-sdk-go-v2/{config,sts,...} and github.com/aws/smithy-go)
to support fetching ID tokens from AWS IMDS for workload identity
federation. Until now, tsnet pulled this in unconditionally via
feature/condregister/identityfederation, costing ~70 unwanted deps for
every tsnet program whether or not it uses workload identity federation.
These AWS SDK deps were originally removed from tsnet on 2025-09-29 by
commit 69c79cb9f ("ipn/store, feature/condregister: move AWS + Kube
store registration to condregister"). They were then accidentally added
back on 2026-01-14 by commit 6a6aa805d ("cmd,feature: add identity
token auto generation for workload identity", PR #18373) when the new
wif package was wired into tsnet via feature/identityfederation.
Drop the blanket import. tsnet programs that want workload identity
federation now opt in with:
import _ "tailscale.com/feature/identityfederation"
The hook lookup in resolveAuthKey already uses GetOk and degrades
gracefully when the feature isn't linked, so existing programs that
don't use workload identity federation see no behavior change. The
tailscale CLI still imports the condregister wrapper directly, so its
behavior is also unchanged.
Lock this in with TestDeps additions: tailscale.com/wif as a BadDep,
plus substring checks in OnDep that fail on any github.com/aws/ or
k8s.io/ dependency creeping back in.
Also, switch cmd/gitops-pusher from the condregister wrapper to a
direct import of feature/identityfederation: gitops-pusher's auth flow
calls HookExchangeJWTForTokenViaWIF directly, so it shouldn't be
subject to the ts_omit_identityfederation build tag.
Updates #12614
Change-Id: I70599f2bdd4d3666b26a859d5b76caa5d6b94507
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Commit 69c79cb9f (Sep 2025) moved awsstore and kubestore registration
behind condregister build tags so tsnet wouldn't pull in the AWS SDK
and Kubernetes client by default. The accompanying TestDeps BadDeps
entry was missed, so PR #19667 (which re-added those imports) wasn't
caught by the test.
Add the two packages to BadDeps so future regressions fail the test.
Updates #19667
Updates #12614
Change-Id: I903b7c976e5e122cc0c0b956dc73740f5d474fac
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new vet analyzer that checks t.Run subtest names don't contain
characters requiring quoting when re-running via "go test -run". This
enforces the style guide rule: don't use spaces or punctuation in
subtest names.
The analyzer flags:
- Direct t.Run calls with string literal names containing spaces,
regex metacharacters, quotes, or other problematic characters
- Table-driven t.Run(tt.name, ...) calls where tt ranges over a
slice/map literal with bad name field values
Also fix all 978 existing violations across 81 test files, replacing
spaces with hyphens and shortening long sentence-like names to concise
hyphenated forms.
Updates #19242
Change-Id: Ib0ad96a111bd8e764582d1d4902fe2599454ab65
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Prior to this change, closing multiple ServiceListeners concurrently
could result in failures as the independent close operations vie for the
attention of the Server's LocalBackend. The close operations would each
obtain the current ETag of the serve config and try to write new serve
config using this ETag. When one write invalidated the ETag of another,
the latter would fail. Exacerbating the issue, ServiceListener.Close
cannot be retried.
This change resolves the bug by using Server.mu to synchronize across
all ServiceListener.Close operations, ensuring they happen serially.
Fixes#19169
Signed-off-by: Harry Harpham <harry@tailscale.com>
This is a regression test for #19166, in which it was discovered that
after calling Server.ListenService for multiple Services, only the
Service from the most recent call would be advertised.
The bug was fixed in 99f8039101
Updates #19166
Signed-off-by: Harry Harpham <harry@tailscale.com>
Previous to this change, closing the listener returned by
Server.ListenService would free system resources, but not clean up state
in the Server's local backend. With this change, the local backend state
is now cleaned on close.
Fixestailscale/corp#35860
Signed-off-by: Harry Harpham <harry@tailscale.com>
TestListenService needs to setup state (capabilities, advertised routes,
ACL tags, etc.). It is imperative that this state propagates to all nodes
in the test tailnet before proceeding with the test. To achieve this,
TestListenService currently polls each node's local backend in a loop.
Using local.Client.WatchIPNBus improves the situation by blocking until
a new netmap comes in.
Fixestailscale/corp#36244
Signed-off-by: Harry Harpham <harry@tailscale.com>
This helps us distribute tests across CI runners. Most tsnet tests call
tstest.Shard, but two recently added tests do not: tsnet.TestFunnelClose
and tsnet.TestListenService. This commit resolves the oversight.
Fixestailscale/corp#36242
Signed-off-by: Harry Harpham <harry@tailscale.com>
This makes tsnet apps not depend on x/crypto/ssh and locks that in with a test.
It also paves the wave for tsnet apps to opt-in to SSH support via a
blank feature import in the future.
Updates #12614
Change-Id: Ica85628f89c8f015413b074f5001b82b27c953a9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
I omitted a lot of the min/max modernizers because they didn't
result in more clear code.
Some of it's older "for x := range 123".
Also: errors.AsType, any, fmt.Appendf, etc.
Updates #18682
Change-Id: I83a451577f33877f962766a5b65ce86f7696471c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Normalize 0.0.0.0 and :: to wildcard in resolveListenAddr so listeners
match incoming connections.
Fix ephemeral port allocation across all three modes: extract assigned
ports from gVisor listeners (TUN TCP and UDP), and add an ephemeral port
allocator for netstack TCP.
Updates #6815
Updates #12182Fixes#14042
Signed-off-by: James Tucker <jftucker@gmail.com>
TestDial in particular sometimes gets stuck in CI for minutes, letting
chantun drop packets during shutdown avoids blocking shutdown.
Updates #18423
Signed-off-by: James Tucker <jftucker@gmail.com>
When a tsnet.Server dials its own Tailscale IP, TCP SYN packets are
silently dropped. In inject(), outbound packets with dst=self fail the
shouldSendToHost check and fall through to WireGuard, which has no peer
for the node's own address.
Fix this by detecting self-addressed packets in inject() using isLocalIP
and delivering them back into gVisor's network stack as inbound packets
via a new DeliverLoopback method on linkEndpoint. The outbound packet
must be re-serialized into a new PacketBuffer because outbound packets
have their headers parsed into separate views, but DeliverNetworkPacket
expects raw unparsed data.
Updates #18829
Signed-off-by: James Tucker <james@tailscale.com>
Two methods were recently added to the testcontrol.Server type:
AddDNSRecords and SetGlobalAppCaps. These two methods should trigger
netmap updates for all nodes connected to the Server instance, the way
that other state-change methods do (see SetNodeCapMap, for example).
This will also allow us to get rid of Server.ForceNetmapUpdate, which
was a band-aid fix to force the netmap updates which should have been
triggered by the aforementioned methods.
Fixestailscale/corp#37102
Signed-off-by: Harry Harpham <harry@tailscale.com>
This resolves a gap in test coverage, ensuring Server.ListenService
functions as expected in combination with user-supplied TUN devices
Fixestailscale/corp#36603
Co-authored-by: Harry Harpham <harry@tailscale.com>
Signed-off-by: Harry Harpham <harry@tailscale.com>
This file was never truly necessary and has never actually been used in
the history of Tailscale's open source releases.
A Brief History of AUTHORS files
---
The AUTHORS file was a pattern developed at Google, originally for
Chromium, then adopted by Go and a bunch of other projects. The problem
was that Chromium originally had a copyright line only recognizing
Google as the copyright holder. Because Google (and most open source
projects) do not require copyright assignemnt for contributions, each
contributor maintains their copyright. Some large corporate contributors
then tried to add their own name to the copyright line in the LICENSE
file or in file headers. This quickly becomes unwieldy, and puts a
tremendous burden on anyone building on top of Chromium, since the
license requires that they keep all copyright lines intact.
The compromise was to create an AUTHORS file that would list all of the
copyright holders. The LICENSE file and source file headers would then
include that list by reference, listing the copyright holder as "The
Chromium Authors".
This also become cumbersome to simply keep the file up to date with a
high rate of new contributors. Plus it's not always obvious who the
copyright holder is. Sometimes it is the individual making the
contribution, but many times it may be their employer. There is no way
for the proejct maintainer to know.
Eventually, Google changed their policy to no longer recommend trying to
keep the AUTHORS file up to date proactively, and instead to only add to
it when requested: https://opensource.google/docs/releasing/authors.
They are also clear that:
> Adding contributors to the AUTHORS file is entirely within the
> project's discretion and has no implications for copyright ownership.
It was primarily added to appease a small number of large contributors
that insisted that they be recognized as copyright holders (which was
entirely their right to do). But it's not truly necessary, and not even
the most accurate way of identifying contributors and/or copyright
holders.
In practice, we've never added anyone to our AUTHORS file. It only lists
Tailscale, so it's not really serving any purpose. It also causes
confusion because Tailscalars put the "Tailscale Inc & AUTHORS" header
in other open source repos which don't actually have an AUTHORS file, so
it's ambiguous what that means.
Instead, we just acknowledge that the contributors to Tailscale (whoever
they are) are copyright holders for their individual contributions. We
also have the benefit of using the DCO (developercertificate.org) which
provides some additional certification of their right to make the
contribution.
The source file changes were purely mechanical with:
git ls-files | xargs sed -i -e 's/\(Tailscale Inc &\) AUTHORS/\1 contributors/g'
Updates #cleanup
Change-Id: Ia101a4a3005adb9118051b3416f5a64a4a45987d
Signed-off-by: Will Norris <will@tailscale.com>
tsnet users can now provide a tun.Device, including any custom
implementation that conforms to the interface.
netstack has a new option CheckLocalTransportEndpoints that when used
alongside a TUN enables netstack listens and dials to correctly capture
traffic associated with those sockets. tsnet with a TUN sets this
option, while all other builds leave this at false to preserve existing
performance.
Updates #18423
Signed-off-by: James Tucker <james@tailscale.com>
This change allows tsnet nodes to act as Service hosts by adding a new
function, tsnet.Server.ListenService. Invoking this function will
advertise the node as a host for the Service and create a listener to
receive traffic for the Service.
Fixes#17697Fixestailscale/corp#27200
Signed-off-by: Harry Harpham <harry@tailscale.com>
Allow for optionally specifiying an audience for tsnet. This is passed
to the underlying identity federation logic to allow for tsnet auth to
use automatic ID token generation for authentication.
Updates https://github.com/tailscale/corp/issues/33316
Signed-off-by: Mario Minardi <mario@tailscale.com>
Previously the funnel listener would leave artifacts in the serve
config. This caused weird out-of-sync effects like the admin panel
showing that funnel was enabled for a node, but the node rejecting
packets because the listener was closed.
This change resolves these synchronization issues by ensuring that
funnel listeners clean up the serve config when closed.
See also:
https://github.com/tailscale/tailscale/commit/e109cf9fdd405153a8d8c0ec52a87d7c8ce8689b
Updates #cleanup
Signed-off-by: Harry Harpham <harry@tailscale.com>
So they're not all run N times on the sharded oss builders
and are only run one time each.
Updates tailscale/corp#28679
Change-Id: Ie21e84b06731fdc8ec3212eceb136c8fc26b0115
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
And yay: tsnet (and thus k8s-operator etc) no longer depends on
portlist! And LocalBackend is smaller.
Removes 50 KB from the minimal binary.
Updates #12614
Change-Id: Iee04057053dc39305303e8bd1d9599db8368d926
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
I saw this panic while writing a new test for #14715:
panic: send on closed channel
goroutine 826 [running]:
tailscale.com/tsnet.(*listener).handle(0x1400031a500, {0x1035fbb00, 0x14000b82300})
/Users/bradfitz/src/tailscale.com/tsnet/tsnet.go:1317 +0xac
tailscale.com/wgengine/netstack.(*Impl).acceptTCP(0x14000204700, 0x14000882100)
/Users/bradfitz/src/tailscale.com/wgengine/netstack/netstack.go:1320 +0x6dc
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*Forwarder).HandlePacket in goroutine 807
/Users/bradfitz/go/pkg/mod/gvisor.dev/gvisor@v0.0.0-20240722211153-64c016c92987/pkg/tcpip/transport/tcp/forwarder.go:98 +0x32c
FAIL tailscale.com/tsnet 0.927s
Updates #14715
Change-Id: I9924e0a6c2b801d46ee44eb8eeea0da2f9ea17c4
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
this commit reduced the amount of data sent in the metrics
data integration test from 10MB to 1MB.
On various machines 10MB was quite flaky, while 1MB has not failed
once on 10000 runs.
Updates #13420
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
