The wireguard-go device now learns its peer set solely from the live
per-peer config source that LocalBackend installs with
Engine.SetPeerConfigFunc, backed by the route manager. Peers are
created lazily on first packet and converged per peer with
Engine.SyncDevicePeer, so the full-peer-list snapshot in wgcfg.Config
and the diff-and-reconfigure machinery around it (wgcfg.Peer,
ReconfigDevice, and the engine's full device sync in
maybeReconfigWireguardLocked) are dead weight: they duplicated state
that the route manager already owns and forced every netmap change to
rebuild and rehash the entire peer list.
Delete the Peers field and the Peer type from wgcfg, along with
ReconfigDevice and maybeReconfigWireguardLocked. Engine.Reconfig no
longer does any device peer work; it only manages the private key,
addresses, and the non-peer subsystems. Full-netmap application converges the device by
syncing exactly the peers whose routes the route manager reports as
changed or removed.
Updates #12542
Change-Id: Ic776e42cfaa5be6b9329b3d381d5cbde17d7078b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The map[key.NodePublic][]netip.Prefix that flows from route manager
commits to WireGuard device syncs has subtle semantics (a nil value
means the peer was removed or no longer contributes any prefixes)
that were documented on routemanager.Result.AllowedIPs and then
re-documented, or not, at each signature that passed it along. Give
it a named type, PeersWithRouteChanges, and document the nil
semantics once on it.
Updates #12542
Change-Id: I2566361a5331eb11b2b70a5bcdb497cc20ee561d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously tstun.Wrapper.SetWGConfig walked wgcfg.Config.Peers on every
netmap to rebuild its own IP-to-peer table for masquerade NAT rewrites
and jailed-peer classification. Now the tun layer instead consumes the
route manager's shared immutable outbound snapshot directly, via a new
Engine.SetPeerRoutes method: LocalBackend pushes the snapshot (plus this
node's native Tailscale addresses) after every route manager commit that
can change it, and per-packet lookups read the interned PeerRoute
attributes from that table.
When no current peer is jailed or masqueraded, LocalBackend installs a
nil table (gated on RouteManager.HasDataPlaneAttrs), preserving the
per-packet nil-check fast path. The exitNodeRequiresMasq machinery is
deleted: its purpose was populating the table with all peers so that
more-specific entries shadow an exit node's /0, and the always-full
route manager table gives that shadowing inherently.
This is the last step before removing the Peers field from wgcfg.Config.
Updates #12542
Change-Id: Ifce09ca929a3f2511303ca1d6efdd583739494ce
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The tun-layer per-peer data plane (jailed packet filter selection and
masquerade NAT rewrites) had no end-to-end coverage: nothing asserted
that a peer the control server marks jailed actually has its flows
dropped, or that masquerade addresses assigned by control actually
carry rewritten traffic in both directions.
Add a two-gokrazy-node natlab test driving both knobs from the test
control server and probing with HTTP requests between the nodes'
webservers after each netmap transition, asserting each response
carries the serving node's greeting so masqueraded flows provably
reach the intended node. TSMP pings are deliberately not used as
probes: tstun answers those before running the packet filter, so
they succeed even for jailed peers.
Updates #12542
Change-Id: Ia978e8d368f08a5a1117280f12bd50310969d0ec
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Two adjacent bool arguments at call sites are easy to transpose and
hard to read. Use an unexported bitmask type instead, per review
feedback on PR #20414.
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0417270c9c3f4b2522911c39aad375cbaf025a82
The BIRD (BGP) integration previously lived half in cmd/tailscaled
(which created a chirp client via a build-tag-gated file on some
platforms) and half in wgengine (which carried the client in its
Config and toggled the "tailscale" protocol as the node gained or
lost primary subnet router duty).
Move it all to a new feature/bird package, installed on the engine
via the new wgengine.HookNewBird hook, like other feature/* packages.
wgengine.Config.BIRDClient (and the wgengine.BIRDClient interface)
are replaced by a BIRDSocket path from which the engine constructs
the feature's Bird handle at startup. The subnet router overlap
detection and protocol toggling move into feature/bird, preserving
the previous ordering: state is recomputed before Reconfig's
ErrNoChanges early return and applied after the router is configured.
tailscaled keeps BIRD support by default on the platforms that
previously had it (linux, darwin, freebsd, openbsd) via
feature/condregister.
Also, add an integration test, as this feature lacked much test
coverage previously.
Updates #12614
Change-Id: I7866a50779e454c87933b358735f7dcd9e2b126f
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Every netmap change, including an incremental delta of a single peer,
rebuilt the full MagicDNS state twice: dnsConfigForNetmap walked all
peers to build the dns.Config.Hosts map, and resolver.SetConfig then
walked that map again to build its reverse (PTR) index. On a tailnet
with 10k peers that is a lot of garbage per delta.
Instead, add a resolver.MagicDNSHosts hook, installed once by
LocalBackend, that the quad-100 resolver consults on demand at query
time. It is backed by nodeBackend's nodeByName, nodeByAddr, and peers
indexes, which are already maintained incrementally as netmap deltas
arrive. The subdomain-resolve capability check also moves to the hook
(checking the node's CapMap at query time), so dns.Config's
SubdomainHosts is no longer populated.
dns.Config.Hosts remains for control's DNS.ExtraRecords, which are
few and which feed the split-DNS decisions in dns.Manager's
compileConfig, and on Windows it still carries every node's records
because the hosts-file fallback path (compileHostEntries) needs the
complete enumerable set. Those compileConfig decisions also consulted
the per-node Hosts entries (hasHostsWithoutSplitDNSRoutes), so a new
Config.MagicDNSHostsUnrouted bit preserves that signal now that node
records are not listed: with MagicDNS names present but MagicDNS
domain routing off, quad-100 stays in the OS resolver path.
One small behavior change: reverse (PTR) lookups now also answer for
node addresses whose forward records are filtered out by the
IPv6-suppression rule (issue #1152), since nodeByAddr indexes all node
addresses. Previously such addresses were absent from the pushed
Hosts map and thus from the reverse index.
Updates #12542
Updates tailscale/corp#43949
Change-Id: I63b99199c2b3b124c08cb8bbaea1f63165095294
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The conn25 extension's ExtraWireGuardAllowedIPs hook (Transit IPs)
was only appended to wgcfg.Config.Peers in authReconfig. Now that
outbound peer selection comes from the route manager's outbound
table via the engine's PeerByIPPacketFunc (which, when installed,
replaces wireguard-go's AllowedIPs trie lookup entirely) and lazily
created peers get their allowed IPs from the route manager via the
engine's peer config func, those extras never reached either path:
outbound packets to Transit IPs matched no peer, and lazily created
peers didn't accept inbound Transit IP sources.
Teach the route manager a per-peer set of extra allowed IPs, staged
by Mutation.SetExtraAllowedIPs. They appear in the outbound table
and in PeerAllowedIPs (so both outbound routing and per-peer allowed
source prefixes see them) but are excluded from the OS route set,
preserving the hook's contract that the extras reach WireGuard but
not the OS routing table. authReconfig now feeds the hook's results
into the route manager and incrementally syncs any changed peers to
the WireGuard device; the append to cfg.Peers remains only so Reconfig's
full per-peer device sync doesn't strip the extras from active
peers, and goes away with Config.Peers.
Updates #12542
Change-Id: I06c8fa30929fbf8fe171a2d34c47c6fcc3abfa16
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Engine.Reconfig previously diffed cfg.Peers disco keys against the
previous config to find restarted peers and flush their WireGuard
sessions, with a TSMP-learned-key map to suppress resets for key
changes that arrived over a working session. That was the last
per-peer state computed from wgcfg.Config.Peers inside the engine,
and it only ran on full reconfigs, so incremental netmap deltas
never got session resets at all.
Move the detection into nodeBackend, which sees every peer change:
full netmaps in SetNetMap and incremental upserts in
UpdateNetmapDelta both now report which peers changed disco keys,
with the same TSMP suppression and mismatch accounting as before.
LocalBackend acts on the result via a new Engine.ResetDevicePeer
method, which just removes the peer from the WireGuard device and
lets the peer lookup func lazily re-create it with fresh state.
LocalBackend.PatchDiscoKey now records TSMP-learned keys in
nodeBackend instead of forwarding to the engine, so the engine's
PatchDiscoKey method and tsmpLearnedDisco map are gone. The
controlclient patchDiscoKeyer interface becomes the exported
DiscoKeyUpdater so LocalBackend can compile-time assert that it
implements it, alongside its NetmapDeltaUpdater friends, replacing
the test that asserted the same of the engine.
This is one of the last steps toward removing Peers from wgcfg.Config.
Updates #12542
Change-Id: I6b42e460f42924816beae89ca43731cb91b66054
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
routerConfigLocked previously computed router.Config.Routes with
peerRoutes, a from-scratch pass over cfg.Peers on every reconfig.
The route manager already maintains the same set incrementally (ULA
and CGNAT coarsening included) and updateRouteManagerPrefs runs
earlier in authReconfig, so its OS route set is current by the time
the router config is built. Read it from there instead, and delete
peerRoutes and its tests; the routemanager package tests cover the
same behavior. This removes a consumer of cfg.Peers, which is on
its way out.
Updates #12542
Change-Id: I4a5b7a63d530e3fe1b70f0faf3f49def6a10be2e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
AddReportHistoryForTest temporarily swapped out Client.TimeNow without
holding any lock, racing with concurrent GetReport calls reading it
from ReSTUN goroutines during tests. Instead, pass the current time to
addReportHistoryAndSetPreferredDERP explicitly so the test helper never
needs to touch the field.
Fixes#20438
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9c4f0a2f9427b5f1d3e8b06a49f0d2b71c3ee8a4
Engine.PeerForIP was pure delegation to the callback that LocalBackend
installs via SetPeerForIPFunc, so external callers going through the
engine were taking a pointless round trip: LocalBackend called
b.e.PeerForIP, which called right back into LocalBackend, and the
netstack UseNetstackForIP hooks in tsnet and tailscaled did the same
dance one layer removed.
Export LocalBackend.PeerForIP and make those callers use it directly.
The engine-internal cold paths (Ping, TSMP disco advertisements,
pendopen diagnostics) still need the lookup and have no netmap of
their own, so SetPeerForIPFunc stays on the interface, but the lookup
method itself is now unexported and gone from the Engine interface.
In tailscaled the UseNetstackForIP hook moves from netstack setup to
just after the LocalBackend is created, since the backend doesn't
exist yet when netstack is wired up.
Updates #12542
Change-Id: Ib1e1a4fa5c84ee0dcb9ce5d1910047f2bab9453c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add TestTailscaleSSH to tstest/natlab/vmtest, exercising the Tailscale
SSH server (tailscale up --ssh, not a system sshd) end to end: an
Ubuntu client node SSHes over the tailnet into an Ubuntu server node
as both root and a non-root user, and into a gokrazy node.
The gokrazy sessions exercise the gokrazy special cases in the SSH
code: util/osuser hard-codes the login shell to serial-busybox ash and
synthesizes a root user when lookup fails (so any username works and
becomes root, unlike Ubuntu where nonexistent users are rejected), and
the incubator's findSU refuses su on gokrazy, handling sessions
in-process.
To support this, testcontrol gains an SSHPolicy field that's sent in
MapResponses along with the CapabilitySSH node capability, tta's /up
handler accepts an ssh=true parameter, and vmtest gains a
TailscaleSSH node option that wires the two together with a
permissive any-principal policy.
Updates tailscale/corp#44813
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f6b9c41a72e05d8c94dd7f6ab1937cf24b81c92
The engine kept its own longest-prefix-match table (peerByIPRoute),
rebuilt from the full peer list on every reconfig, to route outbound
packets and answer PeerKeyForIP. That's now the route manager's job:
LocalBackend already installs a PeerByIPPacketFunc backed by the
RouteManager's incrementally-maintained outbound table, so the
engine's copy was redundant state with redundant O(n peers) rebuild
work.
Delete the table, the PeerKeyForIP interface method, and the BART-only
default callback. LocalBackend's peerForIP now queries the
RouteManager's outbound table directly for the subnet-route and
exit-node fallback. Engines running without a LocalBackend (such as
wgengine/bench) must install their own outbound peer lookup, since the
device's standard AllowedIPs trie only covers peers that already
exist and can't lazily create them.
Updates #12542
Change-Id: I25100399e273ed6c2bb1f6136b7cd81bc83e7313
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The Discover tests step runs under set -euo pipefail, so when every
Test function in a file is filtered out by the exclude regex (as with
vnetperf_test.go, whose TestVnetPerf* tests need special invocation),
the final grep -vE produces no output and exits 1, failing the whole
step. Tolerate empty results from both greps so such files simply
contribute no matrix entries. Same for a hypothetical future test
file containing only helpers and no Test functions.
Updates tailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If20c9a1de37e97b1553bd7d5559e2b876a45c19b
Previously, any peer added or removed by an incremental netmap delta
was only visible to wireguard-go after a full authReconfig: wgcfg's
ReconfigDevice re-installed a PeerLookupFunc closing over a freshly
built map of every peer's allowed IPs, doing O(n) work per change.
Instead, install the wireguard-go device hooks once, backed by live
state. Engine.SetPeerConfigFunc installs a single long-lived
PeerLookupFunc that queries LocalBackend's per-node RouteManager on
demand, and Engine.SyncDevicePeer does O(1) per-peer device sync
(remove, or update allowed IPs) as each delta mutation is applied.
Full reconfigs keep an O(n peers) device sync for now, but with no
lookup closure to reinstall and no removed-peer resurrection race; a
later change removes full-config peer syncing entirely.
The RouteManager's PeerAllowedIPs accessor backs the new hooks: its
sorted output makes unchanged state a no-op update, and its peer
filtering mirrors nmcfg.WGCfg, so expired peers and peers predating
both DERP and disco contribute no prefixes and thus cannot be lazily
created in the device, which matters because wireguard-go validates
inbound source IPs against per-peer allowed IPs.
The engine's SetPeerByIPPacketFunc callback is now authoritative when
installed, since LocalBackend's implementation covers subnet routes
and exit-node routes via the RouteManager's outbound table; the
engine's own reconfig-time BART table only serves engines running
without a LocalBackend.
The forced authReconfig on peer add/remove stays for now: the
WireGuard device no longer needs it, but OS routes, the quad-100
resolver's MagicDNS hosts map, and tstun's masquerade/jailed peer
config are still derived from the full peer set. Making those
delta-aware is the next step before gating it.
Updates #12542
Change-Id: I3ba8c7c324bca0ad0269279d03f53b1f17fb63a2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Give nodeBackend a RouteManager and keep it in sync as routing
inputs change: full netmaps resync the whole peer set (removals plus
no-op-cheap upserts), incremental netmap deltas mirror their peer
upserts and removes into the same mutation batch, and
authReconfigLocked pushes the routing-relevant prefs (exit node,
subnet route acceptance, OneCGNAT) after resolving the exit node's
stable ID to its current numeric node ID.
A selected exit node that doesn't resolve to a current peer (a
nonexistent node, or MDM's "auto:any" placeholder awaiting
resolution) is not the same as no exit node: per the long-standing
ipn.Prefs.ExitNodeID contract, it blackholes internet traffic rather
than letting it escape to the local network. RouteManager's Prefs
gains an ExitNodeSelected bit so its OS route set keeps the default
routes in that case, with no outbound peer to carry them, matching
what routerConfigLocked does today, as pinned by TestRouterConfigExitNodeBlackhole in the previous commit.
All mutations happen with nodeBackend.mu held, satisfying the
RouteManager's serialized Begin/Commit contract.
Nothing consumes its snapshots yet; the wgengine data plane and OS
router wiring come next.
Updates #12542
Change-Id: I677b6b2c9efb8e41b3d27071bd9db73e01640d3b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Selecting an exit node that doesn't resolve to any current peer (a
nonexistent node, or MDM's "auto:any" placeholder before it is
resolved) installs the blackhole default routes, so internet traffic
is dropped rather than escaping to the local network. That behavior
is documented on ipn.Prefs.ExitNodeID and five-plus years old, but
nothing tested it. Lock it in ahead of an upcoming change that moves
OS route computation to net/routemanager and must preserve it.
Updates #12542
Change-Id: I0f63b0d5ce46061a74c69b75f7f83f115da7c3d4
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
LocalBackend.Start previously shut down the previous control client in
a goroutine, letting it run concurrently with the new one. An in-flight
lite map update carrying stale Hostinfo.RequestTags could then be
processed by the control plane after the new client had already changed
the node's tags. Control treats such a request as an invalid tag
transition and expires the node key to force a reauth, so retagging a
node with "tailscale up --advertise-tags" intermittently logged the
machine out.
Instead, detach the old client under b.mu and shut it down
synchronously with the lock released, before creating the new client.
Shutdown cancels the old client's in-flight requests and waits for its
goroutines to exit, so the cancellation of any stale update reaches the
server before the new client sends its first request. Per the deadlock
history in #18052, Shutdown must not be called with b.mu held; this
uses the same pattern as DisconnectControl.
Also teach the testcontrol server to model the control plane's tag
transition handling (including expiring the node key on an invalid
transition and ignoring updates from canceled requests), add an
integration test reproducing the race, and add an ipnlocal test
verifying that Start waits for the old client to shut down.
Updates #20365
Updates #18052
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If8c8e145bdadcef1b1b8fe6209453cf5f5a8d616
FreeBSD guests downloaded their test binaries from vnet's
files.tailscale VIP at roughly 250 kB/s in CI, and transfers sometimes
wedged outright for many minutes, which is why TestSubnetRouterFreeBSD
timed out in about a third of its runs. Locally the same path moves
data at 100+ MB/s, so the problem was never CPU; it was TCP behavior
under two independent constraints, both diagnosed with a new
throughput harness (TestVnetPerfFreeBSDDownload), a VNET_TCP_DEBUG
endpoint sampler, and pcaps:
First, throughput is capped at receive-window/RTT. FreeBSD starts its
receive window at 64 kB and autoscales it in slow 16 kB steps, and on
an oversubscribed CI runner the effective RTT of the userspace vnet
data path reaches hundreds of milliseconds, giving almost exactly the
observed 250 kB/s. Fix: raise the FreeBSD guest's TCP buffer sysctls
in cloud-init before the downloads, and raise netstack's receive
buffer sizing for the reverse (upload) direction.
Second, the outright wedge: when netstack bursts more data than the
QEMU socket plus the guest's virtio RX ring can absorb, a wide swath
of segments is dropped downstream of vnet, and netstack's loss
recovery then crawls, retransmitting one or two segments per 200 ms
RTO for minutes at a time (a 33 MB transfer was observed taking 526
seconds against an otherwise idle receiver). Rather than depending on
recovery from mass loss, make the path effectively lossless by keeping
the maximum in-flight data (the 1 MB netstack send buffer) below the
downstream buffering: grow the guests' virtio RX rings from 256 to
1024 descriptors, enlarge the vnet-QEMU unix socket buffers, and grow
the netstack link endpoint queue from 512 to 4096 packets so a send
burst can't overflow it.
Also fixed along the way, found while chasing the above:
* pcapWriter fsync'd after every packet, serializing all traffic
behind disk writes when a test enables pcap; a pcap-enabled run
was capped at about 290 kB/s. Keep the per-packet Flush but drop
the per-packet fsync.
* Traffic originating from vnet's own netstack (control plane,
DERP, file servers) bypassed conditionedWrite, so SetLatency and
SetPacketLoss silently didn't apply to it.
* writeEthernetFrameToVM held one global mutex (and a shared
scratch buffer) across writes to all VMs, so one guest slow to
drain its socket stalled traffic to every VM on the server. The
write lock is now per-VM-connection.
TestSubnetRouterFreeBSD now passes locally in 31s (down from 4.5
minutes), still passes with the vnet simulating a 100 ms RTT
(downloads at 2-8 MB/s, previously 250-600 kB/s), and passes in 65s
with KVM disabled while pinned to two host CPUs, a harsher environment
than the CI runners. The benchmark test is opt-in via
--run-perf-tests (in addition to --run-vm-tests) so CI doesn't spend
a matrix job re-measuring it on every run. VMTEST_NO_KVM=1
forces TCG for reproducing slow-host behavior.
Fixestailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I1a7945a7e9c7d083b0ea2a3530eda0e9757dff18
PeerByStableID did an O(n peers) scan, and an upcoming change needs
the same StableNodeID-to-NodeID resolution whenever prefs change (to
resolve the selected exit node for the route manager, which keys
peers by NodeID because that is the identity netmap delta mutations
carry). Maintain a nodeByStableID index alongside the existing
nodeByAddr and nodeByKey indexes, updated on full netmaps and on
delta mutations.
Updates #12542
Change-Id: Id1e5105a7470b02312533f0f46b69e6945cd62f0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new RouteManager type that tracks per-peer self addresses and
advertised routes and incrementally maintains two read-only
snapshots: an IP-to-outbound-peer bart table carrying the per-peer
attributes the data plane needs (jailed state, masquerade addresses),
and a coarsened OS route set (including OneCGNAT consolidation).
Mutations are staged in a transaction (Begin/Commit) and applied to
the snapshots via bart's Persist methods, which path-copy only the
few trie nodes along the affected prefix, so a single-peer delta
costs a bounded amount of work independent of the number of peers,
instead of the O(n) full-world rebuild done today. Snapshots are
published via atomic pointer swap for lock-free reads on the hot
path, and Commit reports which peers' allowed IPs changed so callers
can sync wireguard-go incrementally. This is the same immutable value
snapshot pattern as used in the recent containerboot change,
364b952d62.
Nothing uses it yet; this is pulled out of a future change that wires
it into ipnlocal and wgengine, to make that PR smaller.
Updates #12542
Change-Id: Iccc5258024e6f90311835b79fd2d83b2adb0d09d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
[This commit is pulled out of a branch that ultimately removes the
wgcfg.Config.Peers field and removes all O(n peers) processing when
handling deltas]
magicsock.Conn.UpdatePeers existed so wgengine.Reconfig could tell
magicsock the set of WireGuard peers from cfg.Peers, used only to
garbage collect the derpRoute and peerLastDerp maps and to ReSTUN when
the first peers appear. magicsock already learns the full peer list
directly from LocalBackend via SetNetworkMap, UpsertPeer, and
RemovePeer, so do that bookkeeping there and delete the API and its
cfg.Peers use.
Updates #12542
Change-Id: Id07551fc1950239f08a73a9ab02d69ce78d0de0c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Export the machine's boot time (the btime line from Linux's
/proc/stat) as node_boot_time_seconds, named to match what
Prometheus's node exporter uses for the same value. Combined with
process_start_unix_time, this can be used to distinguish process
restarts from whole node restarts.
The value is parsed once per process lifetime, not per scrape, and
the metric is only published when a value is available, so non-Linux
systems don't export a bogus zero.
Updates tailscale/corp#44743
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I5f53186b97bb1482bd1a5387c0910b0ae26544ff
Like the earlier RemoteConfig change, gate Hostinfo.AllowsUpdate on
feature.IsRegistered("clientupdate") in addition to the
buildfeatures.HasClientUpdate build-tag const. tsnet binaries don't
import feature/clientupdate even though ts_omit_clientupdate isn't
set, so they shouldn't tell control they can be remotely updated.
Add the previously missing feature.Register call to
feature/clientupdate, document the binary-support requirement on
tailcfg.Hostinfo.AllowsUpdate, and make tsnet's dep test verify it
doesn't depend on feature/clientupdate.
Updates #12614
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I526ef11f2a4141f5fce161b1f77263324014b5c4
The netmap.NetworkMap type is deprecated and going away, and
reconfigAppConnectorLocked only needed its SelfNode field anyway.
Take a tailcfg.NodeView instead and check its validity in place of
the old nil netmap check.
Updates #12542
Change-Id: Id617845b67416404500cca438ce4ac0372cd8a8e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We had an internal Google doc about this (Tailscalars:
http://go/clientmod) but that doesn't help open source contributors or
agents.
So move the docs to git.
Updates #12614
Change-Id: I0b0e9f0286b23b4fb1b51ff3d41eba75edf62cdf
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
wgcfg.Config.NetworkLogging carried the network flow logging identity
inside the WireGuard config, where it was unrelated to WireGuard; it
lived there mainly so that identity changes would defeat Reconfig's
ErrNoChanges check and reach the netlog startup/shutdown logic.
Remove the field and move the whole netlog lifecycle into a new
feature/netlog package, installed on the engine via the new
wgengine.HookNewNetLogger hook, like other feature/* packages. The
logging identity now comes from LocalBackend's current netmap via the
widened NetLogSource interface (replacing Engine.SetNetLogNodeSource),
so nmcfg no longer parses audit log IDs into the config. The engine
still calls the hook before its ErrNoChanges return and before
router.Set (to capture initial packets), and again after router.Set
(to capture final packets), preserving the previous ordering.
Core wgengine no longer imports wgengine/netlog, so minimal builds
drop it entirely. tailscaled keeps netlog via feature/condregister,
and tsnet imports feature/condregister/netlog explicitly to keep
netlog enabled by default in tsnet-based binaries (tsidp,
k8s-operator).
This is pulled out of a future change that removes wgcfg.Config.Peers,
to make that PR smaller.
Updates #12542
Updates #12614
Change-Id: I41ca7dfe43c51e977c41b5f8e934bd1f0e6e6e24
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Nothing uses them. DNS and MTU are handled elsewhere.
This is pulled out of a future change that removes wgcfg.Config.Peers,
to make that PR smaller.
Updates #12542
Change-Id: I2ec8ae38dc6cce08bcc44e6c1f9177311202af89
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This is a variant of "tailscale configure flash-appliance" but for running
on Proxmox PVE hosts to make a Proxmox VM running the experimental
Tailscale Appliance.
This also makes the "Esc" key make the fbstatus GUI open up a terminal,
instead of Control-Alt-F2 which is hard to type over NoVNC.
And make gafpush unidirectional, to not require a local port be opened locally,
which I hit while working on this.
And make fbstatus included in all appliance variants, but bail out early
and stop respawing if the machine has no framebuffer (e.g. AWS VMs).
Updates #1866
Change-Id: I18ec2a16e4d5ff5574e16fe55c0e8d06cf4fab7f
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new Prefs.RemoteConfig bool. When true, a c2n endpoint at
/remoteapi/localapi/* proxies into this node's LocalAPI at
/localapi/* with full read/write permission, giving the tailnet
admin the same API surface a local root/admin user has via the
tailscale CLI. All LocalAPI versions (v0, v1, ...) proxy through.
RemoteConfig is an alternative to Tailscale's default per-feature
double opt-in, in which both the tailnet admin and the local machine
owner must consent to each individual setting change. It is a single
client-side "I trust the tailnet admin" switch that, once on, hands
over full remote management of this node's settings and LocalAPI
without any further local prompt or confirmation.
This is only appropriate when the tailnet admin already owns the
machine (e.g. a corporate fleet device) or the local user has
explicitly delegated full control. It should never be enabled on a
personal/BYOD device with an untrusted tailnet admin. The trust
model is documented on the pref, on the hidden --remote-config CLI
flag, and on the feature/remoteconfig package.
The node advertises its RemoteConfig state to the control plane via
a new Hostinfo.RemoteConfig bool. This is only true when the feature
is both compiled in (buildfeatures.HasRemoteConfig) and its init
actually ran (feature.IsRegistered("remoteconfig")); tsnet builds
have the former but not the latter and correctly report false.
The handler lives in feature/remoteconfig and can be omitted with the
ts_omit_remoteconfig build tag. tsnet's TestDeps guards against
accidentally pulling it in.
Updates tailscale/corp#18043
Change-Id: I72ce10a90a0e4e738c72c940af3af64c986160b2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The netstack GRO receive path validates L4 checksums before marking
packets as RX checksum validated for gVisor. That validation is invalid
for IPv4 fragments because TCP and UDP checksums cover the complete
reassembled transport packet, not an individual fragment.
Keep validating the IPv4 header checksum, but let IPv4 fragments through
to gVisor for reassembly without pre-validating TCP or UDP.
Fixes#20320
Change-Id: I779363a5e0ac5abee6a8e2a2a44b418fbc5f5e27
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Several improvements to the gokrazy appliance build and flash workflow:
gokrazy/build.go:
- Round Pi image size up to a power of 2 (QEMU raspi3b requires it)
- Use monogok's mkfs.Perm for the /perm ext4 partition (pure Go,
cross-platform, no e2fsprogs dependency)
gokrazy/mkfs:
- Accept optional PermFile entries to include in the freshly-created
ext4 filesystem (used for breakglass authorized_keys)
- Use progresstracking.Ticker for flush progress reporting
gokrazy/tsapp*/config.json:
- Point breakglass at /perm/breakglass.authorized_keys (not ec2)
- Fix Pi SerialConsole to serial0,115200 (not ttyS0)
cmd/tailscale/cli/configure-flash-appliance.go:
- Add --add-ssh-authorized-keys flag to write an authorized_keys
file into /perm during flash (for breakglass SSH access)
- Use progresstracking.CountingWriter + Ticker for write progress
Makefile:
- tsapp-build-and-flash-pi: auto-include ~/.ssh/id_ed25519.pub
- tsapp-qemu-pi: use virt machine + UEFI + ramfb + e1000 (working
network + framebuffer), with DTB watchdog patch and
gokrazy.log_to_serial for debugging
- Auto-detect UEFI firmware path across Debian/Homebrew/Fedora
Updates #1866
Change-Id: Ifa97ad34c509a81e1637d9bce12a788037dfe5ec
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds a Linux-only framebuffer status display (cmd/fbstatus) that draws
to /dev/fb0 on the Tailscale gokrazy appliance. It shows:
- the Tailscale logo
- the current tailscaled state (starting, needs login, running)
- a QR code with the login URL when enrollment is needed (triggers
StartLoginInteractive automatically so the URL appears without
user action)
- the LAN IP or "Waiting for DHCP (MAC)" pinned at the bottom-left
- Tailscale IPs once connected
VT switching: Ctrl-Alt-F2 drops to a busybox text shell on VT2 (for
debugging with a USB keyboard), Ctrl-Alt-F1 returns to the GUI.
Rendering pauses while the text VT is active.
On boot, fbstatus pokes the gokrazy unix socket API to restart the
breakglass SSH service (which uses DontStartOnBoot by default). It
waits until DHCP assigns an IP so breakglass binds to the LAN address
rather than just localhost.
Included in the tsapp-pi.arm64 gokrazy build by default.
Updates #1866
Change-Id: Ifdce4ad8e8c2e1005c840f579e637974a0a266d3
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds gokrazy/gafpush and a 'make tsapp-push-pi PI=<ip>' Makefile
target for pushing a freshly-built GAF to a running appliance over
the network. See the gokrazy/gafpush package doc for details.
Updates #1866
Change-Id: Id7a0bad712fcf2eddae593f71d5feacee05c5234
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Update ts-gokrazy to b83088f which includes:
- Skip hardware watchdog when nowatchdog is on kernel cmdline
- gokrazy.log_to_serial=1 tees service logs to /dev/console
- Fix /etc/resolv.conf symlink (point at /tmp/resolv.conf where
userspace DHCP writes, not /proc/net/pnp which is always empty)
All these things are more emulating a Raspberry Pi in qemu when doing
local development of the appliance image.
Updates #1866
Change-Id: Iba7847e5deb237b1e485b74a4126e31fd118333a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add three new helpers to the existing progresstracking package:
- Ticker: spawns a 1 Hz goroutine that calls a report function with
the current value of an atomic counter and a total. Returns a stop
function (safe to call multiple times via sync.OnceFunc) that fires
one final report and blocks until the goroutine exits.
- NewWriter: wraps an io.Writer and calls onProgress at most once per
interval with the cumulative byte count.
- CountingWriter: an io.Writer that atomically counts bytes written,
for use with Ticker.
These will be used by the appliance flash and OTA update code in
subsequent commits.
Updates #1866
Change-Id: If353cea6506f5351b6fb19bfdb7bc9b78fe7855e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds a CLI subcommand that downloads a signed Tailscale appliance
image (Gokrazy archive format, GAF) from pkgs.tailscale.com,
constructs a fresh GPT-partitioned disk from it (mbr.img + a
synthesized partition table + boot.img + root.img), formats /perm
as ext4 in pure Go via go-diskfs, and ejects the disk so a user
running on a regular workstation can flash an SD card or homelab
VM disk in one command without installing e2fsprogs.
On macOS the target disk is auto-discovered via diskutil, skipping
the boot disk and anything bigger than 256 GB out of paranoia. On
Linux the user passes --disk=/dev/sdX explicitly. Windows is not
supported yet and the command returns an error.
The GPT layout matches monogok's full-disk layout via the new
public github.com/bradfitz/monogok/disklayout package; a drift-
guard test inside monogok asserts the two implementations stay
byte-identical so OTA updates against monogok-built images keep
working.
Behind a ts_omit_flashappliance build tag (on by default).
Updates #1866
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ic1a8cd185e7039edccb7702ab4104544fcb58d29
Tailscaled had no way to seed device-scope syspolicy settings short of
environment variables or a custom store wired up out of tree. Add a
--syspolicy-file flag whose default points at a well-known JSON file
that, when present, is parsed as a map[string]any and registered as a
device-scope policy source. The default path is
/etc/tailscale/syspolicy.json on every non-Windows platform (Linux, the
BSDs, illumos/Solaris, and tailscaled-without-the-GUI on macOS) and
%ProgramData%\Tailscale\syspolicy.json on Windows. The flag lets users
running tailscaled by hand (development, custom installs) point it at
an alternate file, and "" disables the load entirely.
JSON values map to setting types as expected: strings to
StringValue/PreferenceOptionValue/VisibilityValue/DurationValue (e.g.
"24h" parsed by time.ParseDuration), booleans to BooleanValue, numbers
to IntegerValue, and string arrays to StringListValue. The file is
validated against the registered setting definitions at load time so
unknown keys and value/type mismatches fail startup loudly rather than
producing surprising defaults at first read.
When HuJSON support is linked into the build (default; opt out with
ts_omit_hujsonconf), the file may use HuJSON (comments, trailing
commas). With ts_omit_hujsonconf it must be pure standard JSON. This
mirrors the pattern used by ipn/conffile.
On Windows the JSON file and the existing HKLM registry store both
register at DeviceScope. rsop merges later-registered same-scope
sources over earlier ones, so per-key values in the file override the
registry while keys absent from the file fall back to the registry.
The loader is registered via a feature.Hook from a file gated by
!ts_omit_syspolicy, and called from main after flag parsing. tsnet
still does not depend on the root syspolicy package, so embedders
don't pick this up implicitly.
Fixes#20305
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6326461c14efb226979ac162998a9c6373ce493
Builds on top of the unsigned URL-based GAF update flow added previously
(see referenced issue for context). The pkgs.tailscale.com server now
publishes signed GAFs for the unstable track, with detached ed25519
signatures produced by pkgsign's signdist path (the same distsign scheme
used for every other release artifact). This change consumes them.
The URL-based path (tailscale update --gokrazy-update-from-url=URL) now
verifies the signature by default using clientupdate/distsign.Client,
which fetches distsign.pub from the root of the host serving the GAF and
checks the .sig against the root keys embedded in this binary. The
--unsigned flag stays for TestGokrazyUpdatesItselfToSameImage, whose
in-test fileserver does not publish distsign.pub.
The bare tailscale update path is now wired up for the Tailscale
appliance image. It fetches <pkgs>/<track>/?mode=json, picks the GAF
whose key matches the local device (vm-amd64, vm-arm64, or pi-arm64,
where arm64 is split via /sys/firmware/devicetree/base/model), confirms
the version with the user, and reuses the verified download path above.
To avoid wiping a user's custom Gokrazy build that happens to include
tailscaled, the bare update path is gated on hostinfo.Package == "tsapp",
which is only set when the new ts_appliance build tag is present
(mirroring the existing ts_package_container tag). The
gokrazy/tsapp*/config.json files now pass GoBuildTags ["ts_appliance"]
for the tailscale and tailscaled packages so monogok bakes the tag into
the official appliance builds. The TS_FORCE_ALLOW_TSAPP_UPDATE env var
is an escape hatch for callers who want to force the appliance update
path on a non-appliance build. The URL-based path stays ungated since it
requires explicit user intent (and is exercised by the natlab vmtest).
Updates #20002
Change-Id: I7c7856a88bf3dffb9eb8d3e9111fad0b3906743c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds two Gokrazy-based vmtests covering the tailscaled web client at
port 5252:
* TestWebClientLocalAccess enables the web client on a single node
and exercises the canonical owner session flow against the node's
own Tailscale IP: an unauthenticated GET /api/auth that identifies
the caller, a GET /api/auth/session/new that issues a
TS-Web-Session cookie, and a final GET /api/auth that reports
authorized=true with the cookie.
* TestWebClientRemoteAccess runs the same session flow from a peer
node on the same tailnet against a second target node's web
client, exercising netstack interception of incoming :5252
traffic, cross-node WhoIs, and the same-user "owner" path. It
then flips the test control server's AllNodesSameUser off,
re-logs in the client under a fresh identity, and asserts that
GET /api/auth/session/new returns 401 with body "not-owner" --
exercising the cross-user rejection in client/web/auth.go.
To make the natlab test environment exercise the same code path
as production (check mode, where the web client posts to
/machine/webclient/init via Noise and waits on a control-issued
auth URL), this also:
* Allowlists the natlab fake control hostname "control.tailscale"
in client/web/auth.go's controlSupportsCheckMode so the web
client follows the check-mode branch rather than the
no-check-mode shortcut that immediately marks new sessions
authenticated.
* Adds /machine/webclient/{init,wait} handlers to testcontrol.
init returns a placeholder auth ID and URL; wait returns
Complete=true immediately, so the web client's awaitUserAuth
resolves on its first call. Together these let the tests drive
the full check-mode session lifecycle without a real
browser-click loop.
To support the multi-request HTTP flows from the test harness,
this also adds:
* vmtest.Env.HTTPGetStatus, a sister of HTTPGet that returns the
upstream status code, body, and Set-Cookie cookies (as a
vmtest.HTTPResponse) and accepts cookies on the outgoing
request, so tests can drive flows that depend on cookie
continuity.
* Cookie pass-through in cmd/tta's /http-get handler: it forwards
the Cookie request header upstream and surfaces upstream
Set-Cookie response headers downstream. This is what lets
HTTPGetStatus carry a session cookie across requests.
Previously the only tests of the web client were in-process
httptest-based handler tests in client/web/web_test.go; nothing
exercised the actual port 5252 listener wiring, the cross-node
auth path, cookie-driven session state transitions through the
check-mode control round-trip, or the not-owner rejection end
to end.
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Idb01486a89b53ac02c6ad3358bcfcceca90dbc36
f5eac39ea ("feature/acme, ipn/ipnlocal: start moving ACME/cert state
into an extension") started to move the cert code into feature/acme
but was meant as a baby step.
This goes further, moving almost everything, leaving only some hooks
in ipnlocal.
When we later move "serve" support out to feature/serve, this will
look a bit different in that the hooks currently in ipnlocal will move
to feature/serve (cert support already depends on serve).
As part of this, cert-related tests move to feaure/acme too, which
means some test infra from ipnlocal now moves to shared ipnlocaltest.
(it's not big at the moment, but I imagine it growing)
Updates #12614
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9ea89aa9754f12d54b81751b6bd830f2664241ff
This was missing in the earlier f5eac39ea7 and meant that tsnet users weren't
getting (all of) acme support.
Thanks to @ChaosInTheCRD and @BeckyPauley for debugging.
Updates #12614
Updates #20252
Change-Id: I176a7b179b2ad3726aca484057f0aae7cc3561c8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Move all the FooForTest methods on LocalBackend to instead be
methods on a new unexported forTest type which is then given out
to callers in other packages via an exported ForTest method
(panicking in non-test contexts) that returns that unexported type.
This is unusual style (exported returning unexported) but declutters
godoc and makes call sites both more explicit and easier to read
without the "ForTest" suffix polluting the symbols. Now FooForTest()
changes into ForTest().Foo().
This was motivated by a pending change moving a bunch of code out of
LocalBackend into other packages that required adding more ForTest
methods to LocalBackend to keep the tests (now in other packages)
working. Instead, do this refactor now so the future change is prettier.
Updates #12614
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib25e6d76d48dc8622ac3a955e0b1220d582e63a8
Add a .policy-tests.yml file with tests exercising the policy
that was just landed: the tailcfg/ control-protocol-owners gate,
the "policybot-override:" comment escape hatch (including
defaults-regression guards so the override rule does not
silently accept a normal review or a 👍 comment), and the
always-on "any tailscale/dev review" baseline.
Updates tailscale/corp#13972
Change-Id: I42afb06b0771658c803512cb5de4701450c8a704
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
prometheus/common v0.66/v0.67 introduced a mandatory
model.ValidationScheme on expfmt.TextParser as part of
prepping for UTF-8 metric/label names in Prometheus 3.0. The
zero value is intentionally UnsetValidation, which panics on
the first call to IsValidMetricName / IsValidLabelName with
Invalid name validation scheme requested: unset
so the long-standing "var parser expfmt.TextParser" pattern
crashes at runtime. Several big downstreams have hit the same
sharp edge:
https://github.com/thanos-io/thanos/issues/8823https://github.com/grafana/loki/pull/21401
Switch our two callers (parseMetrics in tsnet's
TestUserMetricsByteCounters and the client-metrics scraper in
tstest/natlab/vmtest) to the new expfmt.NewTextParser
constructor with model.LegacyValidation. LegacyValidation
matches the classic ASCII metric/label naming rules that
tailscaled's exporter uses today; if and when we ever emit a
metric with a UTF-8 name, we can revisit.
Goes to v0.69.0 (the latest at the time of writing) rather
than v0.67.5 so we pick up the unrelated security fixes for
cross-host redirects.
Done in advance so a follow-up change can pull in
github.com/tailscale/policybottest (which depends on
palantir/policy-bot, which transitively requires
prometheus/common at v0.67+) without dragging this debugging
into that PR.
Updates tailscale/corp#13972
Change-Id: I4b37db9ad3bebef1a32d9020bf6f8790bab25336
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The override comment didn't work as expected.
(I'll be updating the policytest package to handle this)
Updates tailscale/corp#13972
Change-Id: Ic5c16eed09c8cb5fa8dab37d43cf05f8dfa75d49
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
GitHub's built-in CODEOWNERS only supports a hard "block until a team
member reviews" rule, with no way to leave an audit trail when the
requirement is intentionally bypassed. Move review enforcement to
palantir/policy-bot (https://github.com/palantir/policy-bot) running
at https://policybot.corp.ts.net, which lets us express the same
tailcfg/ -> control-protocol-owners rule plus an explicit override:
any other @tailscale/dev member can post
policybot-override: <reason>
as a PR comment and that comment counts as their approval, with the
reason recorded in the PR conversation as a permanent audit trail.
CODEOWNERS is kept as a one-screen comment so anyone landing on it
expecting the old behavior is directed to .policy.yml.
Updates tailscale/corp#13972
Change-Id: I2dc3619c498d4c4a6decae29aa123f6d67905eed
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The ACME serialization mutex (acmeMu) was a package-level global, and
several ACME-related fields lived on LocalBackend even though the
cert code is conditional and not linked into every binary. With
multiple tsnet.Servers in one process (each its own LocalBackend),
a process-wide acmeMu also serialized unrelated backends.
Introduce a new feature/acme extension that owns the per-LocalBackend
ACME/cert state in an ipnlocal.CertState value:
- acmeMu, renewMu, renewCertAt (previously package globals)
- pendingACMETLSALPNCerts, pendingCertDomains{,Mu},
getCertForTest, certRefreshCancel (previously LocalBackend
fields, only meaningful when ACME was compiled in)
ipnlocal/cert.go now reaches the state through b.certState(), which
is routed by a feature.Hook installed at init by feature/acme. The
CertState type lives in ipnlocal so cert.go can access its fields
directly without a method explosion; the extension in feature/acme
constructs and owns it.
This is a baby step. The end goal is for the entire cert/ACME code
to live in feature/acme, with ipnlocal only retaining whatever thin
hooks the rest of LocalBackend needs to call into it. The current
split (CertState and most of cert.go in ipnlocal, extension wrapper
in feature/acme) is a deliberately temporary middle ground that
keeps this PR small while making the next moves mechanical.
The package is named feature/acme to match the existing HasACME /
ts_omit_acme naming. condregister/maybe_acme.go wires it in for
non-js builds.
Updates #12614
Updates #20248
Updates #20249
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I520909f24ad11a9622ef33c2290fe36ad44d6f71
setWebClientAtomicBoolLocked and setDebugLogsByCapabilityLocked
each only need the node capabilities to decide what to do, so
take a set.Set[tailcfg.NodeCapability] directly as part of
getting rid of netmap.NetworkMap.
Updates #12542
Change-Id: If7c30b6354fd42dfe82ed6d2e2fe3439de401315
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The engine only used the netmap to look up self addresses and the
self node's primary routes, so pass it the self node directly
rather than the whole netmap.
Updates #12542
Change-Id: I13c0028eed65d2177baf4cf6c449f5e441845a18
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
tsdial.Dialer.SetNetMap rebuilt an O(n peers) map of MagicDNS names on
every netmap change. As we move toward per-peer incremental deltas,
this becomes quadratic. This removes it and replaces it with
SetResolveMagicDNS, a callback into LocalBackend that looks up
hostnames from nodeBackend's new nodeByName index (populated alongside
nodeByAddr/nodeByKey on both full and delta paths). The index stores
both FQDNs and short names as keys.
This is the same treatment applied to netlog (8f210454d), wglog
(988b0905b), and drive (1d6989408): stop pushing *netmap.NetworkMap
into subsystems and instead have them pull from LocalBackend's live
data via callbacks.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I24557ab0c8a27636e08e4779bcfd3ec633db0a78
Env.Start boots all VM nodes in parallel; each calls
createCloudInitISO -> ensureDebugSSHKey concurrently. When
/tmp/vmtest_key doesn't yet exist, the first goroutine creates it
with os.WriteFile, which opens with O_CREATE|O_TRUNC and briefly
leaves the file existing-but-empty between the open and the
subsequent write. A concurrent goroutine that hits that window
sees ReadFile succeed with zero bytes, then fails ssh.ParsePrivateKey
with "ssh: no key found", causing boot to fail with:
boot: creating cloud-init ISO: parse /tmp/vmtest_key: ssh: no key found
Observed in CI on TestSiteToSite (3 nodes). Wrap the function in
a package-level Mutex so the first caller fully writes the key
before any other caller reads it.
Updates #20228
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6399dcba0c397bb8041931d3de1c6063a11c568
Add a retry loop with BatchMode=yes to absorb the race window
between Env.Start() returning (when tta reports the tailscale
backend as Running) and cloud-init finishing the user/SSH-key
setup. In CI, the second VM's tta agent has been observed
connecting only a few hundred milliseconds before the test SSHes
in, which is inside the window where /root/.ssh/authorized_keys
hasn't fully landed yet. SSH key auth then fails and ssh(1) falls
back to interactive password prompts (3x), wasting time and
producing a confusing "Permission denied (publickey,password)"
error.
BatchMode=yes makes the client fail fast on auth failure instead
of prompting, and the retry loop handles SSH transport-level
errors (exit code 255) for up to 30 seconds with 500ms backoff.
Remote command non-zero exits still pass through unchanged.
Fixes#20228
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I17f7422e9e27bf7b995f505c0184cbb2b230ed81
userspaceEngine.PeerForIP read from e.netMap.Peers and
e.lastCfgFull.Peers, both of which go stale when peers arrive via
netmap deltas (which skip Engine.SetNetworkMap and Engine.Reconfig).
Every PeerForIP caller (Engine.Ping, the TSMP disco-key handler,
pendopen diagnostics, tsdial.Dialer.UseNetstackForIP, and
LocalBackend.GetPeerEndpointChanges) would report "no matching peer"
for freshly-added peers.
Fix it the same way SetPeerByIPPacketFunc fixed the outbound packet
hot path: have LocalBackend install a callback that reads the live
nodeBackend. nb.NodeByAddr is built from both SelfNode and Peers
(updateNodeByAddrLocked), so a single lookup covers the common case
with IsSelf set when the matched node ID is SelfNode's. The subnet-
route / exit-node-default-route slow path goes through a new
Engine.PeerKeyForIP that exposes the engine's AllowedIPs BART table
(the same table the outbound packet hot path already consults, with
exit-node selection honored), and resolves the matched key back to a
NodeView via the live nodeBackend.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0d4b0d8997c8e796b7367c46b49b61d4fdc717b0
Another baby step toward removing slices of peers from the engine.
getStatus iterated peerSequence (a key snapshot built in Reconfig
from cfg.Peers) and then asked wgdev for each peer's stats; peers
that weren't active in wgdev silently fell out. Iterate active wgdev
peers directly via RemoveMatchingPeers(returnFalse) instead.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3abd348abc30db706db29b3a785179259e48abda
The watchdog (ipn/ipnlocal/watchdog.go) was abusing PeerForIP with an
invalid netip.Addr as a way to acquire and release the engine's
internal locks for deadlock detection. This does the TODO to break it out
into its own method like all the other similarly named methods.
Splitting this out as a prerequisite for a follow-up rewrite of
PeerForIP itself; not having to preserve the lock-probe overload in
the new implementation keeps that follow-up smaller.
Updates #12542
Updates #cleanup
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I25cbffd11aeb65600d9128845404c4918ef88ead
Detect Hetzner via /sys/class/dmi/id/sys_vendor == "Hetzner" and wire
up Hetzner's public recursive DNS resolvers (185.12.64.1, 185.12.64.2)
for use as a cloud host resolver.
Fixes#20217
Change-Id: I24a4c51956adfdd5731f62c937e3c7a4a733ffc7
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This applies the same treatment from PR #20162 (netlog) and
PR #20171 (wglog) to the local Taildrive filesystem wiring, ending the
per-netmap-update O(n) rebuild of the drive remotes list.
This moves the O(n peers) taildrive-remote list rebuild from every
peer change (which previously happened regardless of whether you were
even using taildrive) to instead happen only as needed.
That running on every netmap update and was a contributor to the
broader quadratic behavior we want to eliminate when a single peer is
added or removed.
Instead, this introduces drive.RemoteSource, a small interface the
Taildrive filesystem pulls from lazily on incoming WebDAV requests,
and caches by a generation counter. ipn/ipnlocal installs a
driveRemoteSource once at NewLocalBackend time and bumps
LocalBackend.driveGen on the three events that can actually flip the
drive-capable peer set: full netmap installs (domain + self caps),
UpdateNetmapDelta (peer add/remove or per-peer address changes), and
updatePacketFilter (since PeerCapability values are derived from the
packet filter rules, not from peer.CapMap).
The hook itself is kept but narrowed: it no longer takes a
*netmap.NetworkMap and its only remaining job is to re-notify IPN bus
listeners of the current local shares list on full installs.
This is a dependency to removing the netmap.NetworkMap type from
upstream callers, like wgengine.Engine in general.
(Also add a bunch more tests)
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7e3d2f5b4a9c8e1d6f0a3b7c9e2d4f8a1b6c5e9d
This applies the same treatment from 8f210454dd (netlog) to wglog,
ending use of netmap.NetworkMap and instead getting the canonical data
from LocalBackend/nodeBackend.
This is a dependency to removing the netmap.NetworkMap from
upstream callers, like wgengine.Engine in general.
Updates #12542
Change-Id: Icb5af0799322def048a6f594b49f7d11273f025d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test transferred only 64 KiB over loopback, which can complete
within a single clock tick on fast CI machines, causing
time.Since(start).Seconds() to return 0 and the
"transfer_time_seconds_total > 0" assertion to fail.
Increase the payload to 1 MiB so zero is genuinely implausible, and
retry up to 3 additional times. If the metric is still zero after 4
total attempts, fail hard — at that size it means the timing logic is
actually broken.
Fixes#20213
Change-Id: I3fab510ce8c567506fea5ad803d35acf40d65700
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
aa5da2e5f2 (in the 1.99.x dev series, unstable) introduced some bugs,
only some of which were later fixed. This fixed another. As of that
change, tkaFilterNetmapLocked ran only on full netmaps through
LocalBackend.setClientStatusLocked and not peer upserts via new or
changed peers. The later ae743642d9 fixed a regression in the
Engine layer but didn't fix the tkaFilter code from re-running on
upserts.
This add a tkaFilterDeltaMutsLocked pass before
nodeBackend.UpdateNetmapDelta. For each NodeMutationUpsert whose
peer fails the same signature check tkaFilterNetmapLocked applies,
rewrite the upsert in place into a NodeMutationRemove targeting the
same node ID, so magicsock's per-mutation dispatch and
nodeBackend.peers both drop the peer, matching the prior full-netmap
semantics.
New tsnet tests added:
- TestTailnetLockFiltersUnsignedDeltaPeer covers the new-peer
case.
- TestTailnetLockFiltersUnsignedDeltaPeerReplacement covers the
existing-peer-replacement case, to an empty signature.
- TestTailnetLockFiltersDeltaPeerWithInvalidSignature like above
but with a bogus signature.
Updates #12542
Updates tailscale/corp#43767
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib35d0391541fee654867c26489847dbc5b7e2ae8
Both tests started flaking after my 910735448 ("tstest/natlab/vnet:
send unsolicited IPv6 Router Advertisements") added background RA
traffic on v6-enabled networks.
TestPacketSideEffects races the periodic unsolicited-RA goroutine
against its synchronous packet-count assertions: when the multicast
RA fires after the test has registered its sinks, both sinks receive
it and "got 1 packet, want N" becomes "got N+2".
TestProtocolQEMU's reader was doing raw Read on the SOCK_STREAM unix
socket and comparing the whole result to the expected length-prefixed
packet. The kernel is free to coalesce the on-register RA frame and
the test packet into one Read, in which case bytes.Equal fails and
the entire chunk (including the test packet's bytes) gets discarded
as "unexpected", leading to a 5s i/o timeout. Parse the QEMU uint32
length-prefix framing with io.ReadFull instead so we read exactly one
frame per iteration regardless of how the kernel buffers them. The
SOCK_DGRAM path (TestProtocolUnixDgram) keeps the original raw Read
since datagram boundaries are preserved.
These where the top two flakes in oss on the flakes dashboards.
Updates #13038
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I32983656b692921a0f43a4a5e9a8a6ab2555ee49
Outbound packets produced by netstack (used by tailscaled with
--tun userspace-networking, by tsnet, and by the SOCKS5/HTTP proxies)
enter the wrapper via InjectOutbound{,PacketBuffer} and take the
injectedRead path, which bypasses Filter.RunOut.
RunOut's side effect for UDP/SCTP is to insert the reverse-flow tuple
into the connection-tracking LRU so that Filter.RunIn admits inbound
replies that no explicit ACL rule covers. Skipping it on the injected
path meant a netstack-side dial of UDP would send fine but the reply
would be dropped as "no matching rule". The kernel-TUN path was
already fine because it goes through RunOut.
Fixes#14229Fixes#20064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I816ef55c493a12ff4f561cd89c095559b5c2743b
The Logger previously took a *netmap.NetworkMap at Startup and on every
ReconfigNetworkMap call, denormalizing it into per-IP and self lookup
maps. That denormalization is O(n) over all peers and ran on every
netmap update, contributing to the broader quadratic behavior we want
to eliminate when a single peer is added or removed.
Instead, this makes netlog ask LocalBackend (well, nodeBackend) for
the info it needs, letting us remove the netmap.NetworkMap type
entirely from the netlog package.
This is a dependency to removing the netmap.NetworkMap type from
upstream callers, like wgengine.Engine in general.
Updates #12542
Change-Id: Ib5f2de96e788a667332c0a6f7ac833b3d0053b5c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The earlier aa5da2e5f2 made peer adds and removes through a netmap
delta path that mutates only nodeBackend, on the assumption that
PeerForIP, lookupPeerByIP, the engine's wireguard config
(e.lastCfgFull), the engine BART, wgdev's PeerLookupFunc closure, and
the engine's cached netmap (e.netMap) would all stay correct without
further updates. They don't. I'd totally forgotten that
Engine.PeerForIP has its own alternate IP-to-peer lookup codepath.
Concretely, all of these failed for a peer that arrived via
[tailcfg.MapResponse.PeersChanged] (and never via a full
[tailcfg.MapResponse.Peers] list):
- [wgengine.Engine.PeerForIP] read from e.netMap and e.lastCfgFull
(neither updated on the delta path) and so missed the new
peer. The rando non-data-plane callers (Ping, TSMP, pendopen,
debug endpoints, tsdial.Dialer.UseNetstackForIP for tsnet and
onlyNetstack tailscaled) all returned "no matching peer".
- The engine BART (built from e.lastCfgFull) missed the new peer's
subnet routes / exit-node default routes.
- wgdev's [device.PeerLookupFunc] closure (rebuilt only inside
wgcfg.ReconfigDevice) didn't have the new peer's noise key, so
outbound encryption to the new peer dropped the packet even when
SetPeerByIPPacketFunc returned the right NodePublic.
- And nothing in the delta path triggered NodeMutationRemove to
flow through to authReconfig either, so the same stale state
pointed at removed peers indefinitely.
So just (functionally) revert it for now, to have something easily
cherry-pickable to the 1.100 release branch. Proper fixes can come later
for the next release.
This also adds three new tests:
- TestPingPeerLearnedViaDelta runs disco and TSMP subtests over a
delta-added peer with only self addresses. disco exercises the
cold PeerForIP path (magicsock); TSMP exercises the full data path
through wgdev encryption. Both fail without this fix.
- TestPingSubnetRouteOfDeltaPeer exercises a subnet-router peer
arriving via delta. With s1 in --accept-routes mode, an IP
inside the advertised CIDR must resolve to s2 and a TSMP ping
must round-trip. Hits the BART + lastCfgFull + wgdev staleness
in one go.
- TestPingSelfReturnsIsLocalIP is a regression guard for the
IsSelf early-out in Engine.Ping. Passes on main today; included
here so future refactors of PeerForIP can't regress self
handling without test breakage.
Updates tailscale/corp#43394
Change-Id: I7a049271359bd73e7147ae9e2554e85614c2b8d2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
I previously (in #20096) had only considered the tailscaled deps
and forgot about the CLI deps. This does the CLI ones too.
containerboot and k8s-operator aren't applicable because they build
from oss already.
Updates tailscale/corp#43243
Updates #20067
Change-Id: I66790f822b5d040e7fcf90feabca24669f69cf61
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We aren't supposed to be using CODEOWNERS as blocking
reviews, blocking global cleanups.
(This is why we want to move to go/policybot)
Updates tailscale/corp#13972
Change-Id: I380258e2d4ffd0720d57d891adab06c8ca388617
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add NotifyInProcessNoDisconnect for in-process IPN bus subscribers that
must apply every bus update. When such a subscriber falls behind, block
Notify production instead of sending the terminal fell-behind message and
closing the watch.
This is intentionally not available over LocalAPI, where a slow or stuck
out-of-process client should still be disconnected rather than allowed to
stall tailscaled. In-process callers that use the bit must keep their
callbacks fast and must not call back into LocalBackend from the callback.
Updates #20062
Change-Id: I730ad61a07475243bb226fba2262c1a3ded211ae
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
New-style IPN bus subscribers consume stateful delta streams. Reject
NotifyRateLimit when it is combined with those subscription bits so
tailscaled cannot merge or delay messages that clients need to apply in
order.
Also stop silently dropping notifications when a watcher falls behind.
Remove the watcher, replace its stale queue with one terminal ErrMessage
notification, and close the watch.
Updates #20062
Change-Id: Id9d402ea76f4011cd23f122adf62f30dd4b6f90b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds a fake vnet ACME service, TXT-backed SetDNS support, and a
VM test that fetches a certificate with tailscale cert, serves it with
tailscale serve, and verifies HTTPS from a second node.
This adds coverage motivated by #19915.
Updates #13038
Change-Id: Ie1e53409509337d81c8fbceb63f59f3dfbd48207
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We previously had two GOARCH variants for natlab testing
(gokrazy/natlabapp and gokrazy/natlabapp.arm64), but we didn't have
those variants for the "tsapp" (production, not testing) variant.
This adds arm64 for tsapp too. But there are two major users of arm64
for gokrazy: Raspberry Pis, and cloud VMs. So make two variants.
Updates #1866
Change-Id: Ib9efaf1255101e3cdbc31d49c76fd86dcba21bb1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds support for Gokrazy GAF (Gokrazy Archive Format) zip
auto-updates, starting to wire up Tailscale's clientupdate mechanism
to Gokrazy's update mechanism.
Currently there's just a CLI command to update from a GAF URL,
with an --unsigned flag for use in a new natlab vmtest.
Next step would be publishing unstable track GAF files on
pkgs.tailscale.com, with detached signatures, and then making the
clientupdate mechanism also download those and check signatures.
Updates #20002
Change-Id: Ib03c56f17a57f8a4638398ef83549dac4813323d
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Per chat with an SSH client author who wanted a URL in the output.
And then make it more clear what parts are banners.
Updates #cleanup
Change-Id: If5033ad9dc0dba3d833f24ea39e117a455010492
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The test was asserting that a tailnet ping between two nodes traversed
DERP rather than going direct. But that wasn't really the point of the test,
and I kept forgetting ways that magicsock could find direct paths and
thus break this test.
So loosen it.
We really just want to see whether DERP worked at all and was used in the process
of getting a ping through, whether it was direct or not.
And that "tailscale debug derp" worked at all, which was what the bug
was about to begin with.
No need for all the "must be over DERP" stuff.
Updates #15579
Change-Id: I70ca63dc10919efa3d193b7af1d31a4a3b9d3950
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
vnet only ever sent IPv6 RAs in response to a Router Solicitation. In
practice this meant gokrazy VMs running with a dual-stack LAN never
installed vnet's IPv6 default route: gokrazy brings the link up via
DHCPv4 and the kernel never emits an RS on its own under that init
path. Off-link IPv6 destinations like the fake DERP servers were
therefore unreachable from any gokrazy test node that also had v4
on the same interface. (Pure-v6 nodes happened to work because the
kernel sends an RS as part of v6-only autoconf.)
Fix this in two complementary ways:
- Send an unsolicited RA every 5s to the link-local all-nodes group
on every v6-enabled network. This matches what real routers do
(RFC 4861 §6.2.1, MaxRtrAdvInterval; we use a much shorter
interval than the spec's 200s default so short-lived tests don't
have to wait).
- Send a unicast RA to a newly-registered MAC as soon as a client
first transmits on the wire. Without this the first periodic RA
can land before any VM has connected and the next one isn't
until the next tick, which can be longer than the test runs.
Factor the RA serialization out into buildIPv6RouterAdvertisement so
the solicited, periodic, and per-client paths all share one body.
Update TestSelfSignedDERPHashPinning to use a dual-stack hard-NAT
builder and assert zero errors from DebugDERPRegion (instead of
filtering "over IPv6" errors as it had to before this change). The
new builder also sets TS_DEBUG_STRIP_ENDPOINTS=1 on tailscaled so
disco can't find a direct path: without endpoint stripping, the now-
working non-NATted IPv6 LAN gives the two hard-NAT'd nodes a direct
route, defeating the test's "must traverse DERP" assertion. (Hard
NAT alone was enough before this change because v6 routing was
broken.) Also update sendBetweenClients in the vnet unit tests to
tolerate the new on-register RA noise on its read path.
Updates #13038
Updates #19973
Change-Id: Ic281dc53702a25fa773c46313f453837814233e8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
serveDebugDERPRegion built its TLS config with
ServerName: cmp.Or(derpNode.CertName, derpNode.HostName), which for a
"sha256-raw:<hex>" CertName passed the raw fingerprint to Go's stock
verifier as a hostname; the handshake always failed with a hostname
mismatch. This is the second half of #15579; the first half (tailscaled
itself failing with "unexpected multiple certs presented") was fixed in
Extract a tlsConfigForNode helper that mirrors derphttp.Client.tlsClient
so that sha256-raw and domain-fronting CertName values are dispatched
to tlsdial.SetConfigExpectedCertHash and tlsdial.SetConfigExpectedCert
respectively, falling back to HostName when CertName is empty.
The core fix here was originally written by @imnuke in #19965; that PR
also added a unit test in ipn/localapi/debugderp_test.go which is
replaced in this commit by a new vmtest that exercises the whole stack:
vnet now serves a self-signed cert valid for each fake DERP node's
HostName and exposes its SHA-256 fingerprint, and vmtest grows a new
SelfSignedDERPCertPinning EnvOption that swaps the test DERP map's
nodes to CertName="sha256-raw:<hex>" with InsecureForTests cleared.
TestSelfSignedDERPHashPinning then stands up two hard-NAT'd nodes, has
them communicate over DERP, and calls DebugDERPRegion on each. Before
this fix the test fails with the exact x509 hostname-mismatch error
from the original bug; after, it passes.
Updates #15579
Change-Id: I61f38ffebc7ac5abc962639db1ae88f5cd8633b1
Co-authored-by: Nuke <nuke@imnuke.dev>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Commit 2b338dd6a8 removed watchdogEngine because it was weird
(so many methods) and increasingly unnecessary after we'd cleaned up
and simplified so much of the locking.
This adds back a watchdog, but an easier to maintain one that's more
idiomatic.
Updates #19759
Change-Id: I86c458473e126c0809f37696446ce7acf4cc4eb9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The Tailscale daemon only refreshed TLS certs as a side effect of inbound
TLS handshakes or "tailscale cert" CLI calls. A node that doesn't see
inbound traffic during the renewal window silently rolls past expiry.
Add a once-per-hour background loop on LocalBackend that enumerates Serve
and Funnel HTTPS hostnames (filtered against the netmap's CertDomains so
we don't poke ACME for other nodes' service hostnames) and calls the
existing GetCertPEM path. The renewal decision (ARI window, then 2/3
expiry fallback) is unchanged; the loop just guarantees it runs.
For visibility during initial issuance or restart with a long-expired
cached cert, add a "tls-cert-pending" health Warnable that's set while
ACME is in flight and no usable cached cert exists. Async renewal of a
still-valid cert intentionally doesn't fire it. And then make the CLI "cert"
subcommand print out a warning if it's blocking due to a cert fetch
in flight, using that health info.
Fixes#19911Fixes#19912
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I144e46c40e957b2e879587decace32a523a6eade
Capture Auto.mapCtx while holding Auto.mu before using it for
incremental map update forwarding. Pause and restart paths can replace
the context under the same mutex, so using it after unlocking races
with those writers.
Add a race regression test for the UserProfiles path that repeatedly
cancels the map context while incremental profile updates are
forwarded.
Fixes#19953
Change-Id: Icc55c4a0dffbc16d6507a2b446b3909d4d0a0278
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds @alexwlchan's proposed "tailscale get" command that reads
current preference values, complementing "tailscale set". It uses the
same flag names as set.
tailscale get # show all settings as a table
tailscale get all # same
tailscale get accept-dns # show a single value
tailscale get --json # output as JSON object
tailscale get --set-flags # output as tailscale set argv
Fixes#11389Fixestailscale/corp#38702
Change-Id: Ie366f27f11ccc56c76fff9a94ed8a9de9c835bd0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, testwrapper only retried tests explicitly annotated with
flakytest.Mark. Authors don't pre-emptively mark tests that haven't
flaked yet, so the first flake of a brand-new test failed CI even
when a re-run would have passed.
testwrapper now retries every failing test within a per-test wall-clock
budget (default: 5 minute per-attempt timeout capped at 1.5x the first
failure duration, 10 minute total). A test that fails and then passes
on retry is reported as flaky; a test that never passes within the
budget remains a real failure (exit non-zero).
For flakeapp's existing log scraping, the wire format is preserved:
the "flakytest failures JSON:" line is now emitted only for tests
that ultimately flaked (passed on retry). Unmarked tests get a fake
issue URL of the form https://github.com/{owner}/{repo}/issues/UNKNOWN
where owner/repo is detected from GITHUB_REPOSITORY, the local git
remote, or falls back to tailscale/tailscale. A new "permanent test
failures JSON:" line is emitted for tests that never passed; flakeapp
ignores it for now (a follow-up can teach it to record real failures
separately).
flakytest.Mark stays as an opt-in API: still useful for tracking a
known-flaky test against a real issue and for TS_SKIP_FLAKY_TESTS.
Updates tailscale/corp#38960
Change-Id: I56dfc9b023486d239f60793a53e9690578ce8017
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds tsnet.Server.ListenSSH which, if the SSH feature is linked,
returns a net.Listener whose Accept yields *tailssh.Session values (as
net.Conn). This lets tsnet apps accept incoming SSH connections to
implement custom TUI applications.
Basic apps can use net.Conn directly (Read/Write/Close). Rich apps
import ssh/tailssh and type-assert for peer identity, PTY, signals,
etc. If feature/ssh isn't imported, ListenSSH returns an error.
Includes a demo guess-the-number game in tsnet/example/ssh-game.
Updates tailscale/corp#37839
Change-Id: I4e7c3c96afb030cdf4da8f2d8b2253820628129a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
If a user explicitly adds a non-ts.net (not a CertDomain domain) domain
like "foo.com" to their serve config as a web target that's also an allowed
funnel domain (using raw "tailscale serve set-config"), then use the new
ALPN cert fetching (from b553969b) to get certs for that domain.
This is just plumbing; there's no new product functionality to
actually enable this easily client-side, and it also has no visible
product surface to enable it server-side.
Updates tailscale/corp#41736
Change-Id: Ie2e421ac9611bce64bba3de6a454b2d505ea0e8a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a "tailscale whoami" subcommand that is equivalent to running
"tailscale whois $(tailscale ip -4)" but more ergonomic. It supports
the --json flag just like whois, and shares the WhoIsResponse
rendering code with whois.
Fixes#19907
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I8f33ba7a5608bab7dffa8213303beb5f345936d3
Parallel subtests share *ipn.Notify pointers (e.g. runningNotify).
When multiple subtests reached the same phase concurrently, they
all wrote to the shared notify's InitialStatus field without
synchronization, triggering the race detector.
Fix by shallow-copying *ipn.Notify before setting InitialStatus,
so each test iteration works on its own copy.
Updates #19380
Change-Id: I9dd40037e02146166f006f4f7c1ddcc47adba191
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, sharding required tests to opt in by calling tstest.Shard,
which used a process-global counter to assign each test to a shard.
This had two problems: most tests didn't call it, so they ran on every
shard (defeating the purpose), and shard assignments were unstable
(depended on call order, so adding a test could reshuffle others).
Remove tstest.Shard and tstest.SkipOnUnshardedCI entirely. Instead,
have testwrapper implement sharding automatically for all tests: when
TS_TEST_SHARD=N/M is set, it uses "go list -json" (no compilation) to
find test source files, scans them for top-level Test/Benchmark/
Example/Fuzz function names, and filters by fnv32a(name) % M == N-1.
The filtered names are passed as an anchored -run regex to go test.
Using go list instead of "go test -list" avoids linking the test binary
twice (Go's build cache does not cache test binary linking).
Fixes#19886
Change-Id: I62ab7b3d757324d4c5fd0b5de50c1e3742681791
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Some tests in another repo were broken by tailscale/tailscale#19607.
This fixes them, by finishing off the rest of the migration away from
netmap.NetworkMap on the IPN bus in containerboot.
Containerboot used to rebuild a full NetworkMap-shaped view while
reacting to IPN bus notifications. Now it insteads has its own
netmapState type (immutable) of exactly what it needs to track, and
sends those immutable values around, making cheap edits of new
immutable values when an IPN bus edit arrives.
This should make cmd/containerboot scale to much larger tailnets now too.
Fixes#19852Fixestailscale/corp#42347
Updates #12542
Change-Id: I88adaf061f85f677f954a764935e6654329d75a6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Use TLS-ALPN-01 for Funnel certificate renewals only when the node
already has a cached certificate, and fall back to DNS-01 with a fresh
order if the ALPN path is unavailable or fails.
Dynamically advertise acme-tls/1 only while an ACME challenge
certificate is pending, and add client metrics for DNS-01 and
TLS-ALPN-01 start/success/failure paths.
Updates tailscale/corp#41736Fixestailscale/corp#42320
Change-Id: I5adc6ea129237f9ef592f84fc1a8953c80bc9d5c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
NodeMutationAdd was a misleading name: a PeersChanged entry in a
MapResponse can represent either a truly new peer or a full
replacement for an existing peer that couldn't be expressed as a
PeerChangedPatch. Calling it "Add" implied it was always a completely
new node, which is wrong. (I'd changed my mind on the design of
mapping add/delete events to NodeMutations halfway through #19607 and
forgot to update the name, even though I'd updated half the docs)
Rename it to NodeMutationUpsert to reflect the actual semantics: the
node should be inserted or replaced in the peer map regardless of
whether it already existed.
Updates #19607
Updates #12542
Change-Id: Iebd3daddb3318cba02e115a1b184fcb3ee8f83d6
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The prior aa5da2e5f2 ("process node adds/removes in constant
time") commit missed a bus notification case, where new-style
subscribers set NotifyNoNetmap and then the controlclient map routing
sends a full update (rather than a delta). Those profiles + peers
need to be put on the bus too.
I noticed this only when porting the Android app over to use the
new bus stuff.
Updates #19607
Updates #12542
Change-Id: I82c35011d2c532222ca27f7d4e790522c31bd156
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
In aa5da2e5f2 we made the IPN bus include deltas, including the
PeersRemoved, sending a slice of integer NodeIDs that were
removed. But when updating xcode, I realized there was no way to map
those integers to the stable node IDs used in other places.
I was consdering changing the just-added ipn.Notify.PeersRemoved from
an IntID to a string StableID, but then it doesn't match the MapResponse
wire protocol, which we've tried to match so far.
Instead, just add the integer ID as well. Callers can use whichever
world they want, having both. It's a little regrettable that we still
have two worlds of IDs, but oh well. Neither is really suitable to a
hypothetical future fully federated world of control servers anyway,
so we'll need a third type later anyway, so just live with the two we
have for now.
Updates #12542
Change-Id: Ib8fd48a265e1da1f8779152f141f624a7f7260e9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Brad Fitzpatrick
