go.cmd used cmd.exe to invoke PowerShell, which mangled arguments:
cmd.exe treats ^ as an escape character (so -run "^$" became -run "$",
running all tests instead of none) and = signs also caused issues in
the PowerShell→cmd.exe argument passing layer.
Replace it with a tiny no_std Rust binary (19KB, 32-bit x86 for
universal Windows compat: x86/x64/ARM64) that directly invokes the
Tailscale Go toolchain via CreateProcessW. The raw command line from
GetCommandLineW is passed through to CreateProcessW with only argv[0]
replaced, so arguments are never parsed or re-escaped.
The binary also handles first-run toolchain download natively using
curl.exe and tar.exe (both ship with Windows 10+), so PowerShell is
no longer required for normal operation. The PowerShell fallback is
only used for the rare TS_USE_GOCROSS=1 path.
PowerShell prefers go.exe over go.cmd when resolving ./tool/go, so
this is a drop-in replacement.
With go.exe in place, the CI can use the natural -bench=. -benchtime=1x
-run="^$" flags directly.
Also removes tool/go-win.ps1 which is now unused.
Updates #19255
Change-Id: I80da23285b74796e7694b89cff29a9fa0eaa6281
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Multi-NIC support:
- Add nodeNIC type and node.extraNICs for secondary network interfaces
- Add netForMAC/macForNet to route packets to the correct network by MAC
- Update initFromConfig to allocate a MAC + LAN IP per network
- Fix handleEthernetFrameFromVM, ServeUnixConn to use netForMAC
- Fix MACOfIP, writeEth, WriteUDPPacketNoNAT, gVisor write path, and
createARPResponse to use macForNet (return the MAC actually on that
network, not the node's primary MAC)
- Fix createDHCPResponse for multi-NIC (correct client IP and subnet)
- Add nodeNICMac for secondary NIC MAC generation
- Add Node accessors: NumNICs, NICMac, Networks, LanIP
DHCP fixes:
- Include LeaseTime, SubnetMask, Router, DNS in DHCP Offer (not just
Ack). systemd-networkd requires these to accept an Offer.
- Fix DHCP response source IP: use gateway IP instead of echoing
the request's destination (which was 255.255.255.255 for discovers)
New VIPs:
- cloud-init.tailscale: serves per-node cloud-init meta-data, user-data,
and network-config for VMs booting with nocloud datasource
- files.tailscale: serves binary files (tta, tailscale, tailscaled)
registered via RegisterFile for cloud VM provisioning
- Add ControlServer() accessor for test control server
This is necessary for a three-VM natlab subnet router
integration test, coming later.
Updates #13038
Change-Id: I59f9f356bae9b5509c117265237983972dfdd5af
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
When SetSubnetRoutes is called, also send updatePeerChanged to all
other connected nodes so they re-fetch their MapResponse and learn
about the updated AllowedIPs. Without this, peers never see new
subnet routes until they happen to reconnect to the control server.
Discovered while working on a three-VM natlab subnet router
integration test, coming later.
Updates #13038
Change-Id: I20e7a2fda994a8ab0e7a24240e6eae536f4f5f15
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Instead of generating the full netmap, just fetch the peers out the the
existing peers map.
The extra usage was introduced with netmap caching, but there is no need
to call the netmap to get this information, rather the existing peermap
can be used.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Fixes UDP listeners on VIP Service addresses not receiving inbound traffic.
- Modified shouldProcessInbound to check for registered UDP transport endpoints when processing packets to service VIPs
- Uses FindTransportEndpoint to determine if a UDP listener exists for the destination VIP/port
- Supports both IPv4 and IPv6
The aim was to mirror the existing TCP logic, providing feature parity for UDP-based services on VIP Services.
Fixes#18971
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Add ExtraRootCAs *x509.CertPool to tsd.System and plumb it through
the control client, noise transport, DERP, and wgengine layers so
that platforms like Android can inject user-installed CA certificates
into Go's TLS verification.
tlsdial.Config now honors base.RootCAs as additional trusted roots,
tried after system roots and before the baked-in LetsEncrypt fallback.
SetConfigExpectedCert gets the same treatment for domain-fronted DERP.
The Android client will set sys.ExtraRootCAs with a pool built from
x509.SystemCertPool + user-installed certs obtained via the Android
KeyStore API, replacing the current SSL_CERT_DIR environment variable
approach.
Updates #8085
Change-Id: Iecce0fd140cd5aa0331b124e55a7045e24d8e0c2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
CPU profiling a containerboot subnet router on a large tailnet showed
roughly 40% of CPU spent in serveWatchIPNBus JSON-encoding the full
netmap on every update. containerboot only reads SelfNode fields from
those notifications (and does a peer lookup when TailnetTargetFQDN is
set), so it does not need every intermediate netmap delta.
Set ipn.NotifyRateLimit on all three WatchIPNBus calls so netmap
notifications are coalesced to one per 3s. Initial-state delivery is
unaffected since the rateLimitingBusSender flushes the first send
immediately.
Updates #cleanup
Signed-off-by: Doug Bryant <dougbryant@anthropic.com>
Add server-side per-client bandwidth enforcement using TCP backpressure.
When configured, the server calls WaitN after reading each DERP frame,
which delays the next read, fills the TCP receive buffer, shrinks
the TCP window, and naturally throttles the sender — no packets are dropped.
- Rate limiting is on the receive (inbound) side, which is what an abusive
client controls
- Mesh peers are exempt since they are trusted infrastructure
- The burst size is at least MaxPacketSize (64KB) to ensure a
single max-size frame can always be processed
Also refactors sclient to store a context.Context directly instead of a
done channel, which simplifies the rate limiter's WaitN call.
Flags added to cmd/derper:
--per-client-rate-limit (bytes/sec, default 0 = unlimited)
--per-client-rate-burst (bytes, default 0 = 2x rate limit)
Example for 10Mbps: --per-client-rate-limit=1250000
Updates #38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Consolidate the duplicated WebSocket frame-parsing logic from Read
and Write into a shared processFrames loop, fixing several bugs in
the process:
- Mixed control and data frames in a single Read/Write call buffer
were not handled: a control frame would cause merged data frames
to be skipped.
- Multiple data frames into one Write call weren't being correctly
parsed: only the first frame was processed, ignoring the rest in
the buffer.
- msg.isFinalized was being set before confirming the fragment was
complete, so an incomplete msg fragment, could've been sometimes
marked as finalized.
- Continuation frames without any payload were being treated as if
they didn't have stream ID, even thought the id is already known
from the initial fragment.
Fixestailscale/corp#39583
Signed-off-by: Fernando Serboncini <fserb@tailscale.com>
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Co-authored-by: chaosinthecrd <tom@tmlabs.co.uk>
Move the ipn/desktop blank import from cmd/tailscaled/tailscaled_windows.go
into feature/condregister/maybe_desktop_sessions.go, consistent with how
all other modular features are registered. tailscaled already imports
feature/condregister, so it still gets ipn/desktop on Windows.
Updates #12614
Change-Id: I92418c4bf0e67f0ab40542e47584762ac0ffa2b2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
GetMessage can call back into Go, triggering stack growth and causing the stack
to be copied to a new memory region, which invalidates the original stack pointer
passed to the syscall. Since GetMessage uses that pointer to write the message
before returning, this leads to memory corruption.
In this PR, we fix this by using runtime.Pinner, which requires the pointer to refer
to heap-allocated memory.
Fixes#19263Fixes#17832
Signed-off-by: Nick Khyl <nickk@tailscale.com>
Add a new "ipnbus" build feature tag so the watch-ipn-bus LocalAPI
endpoint can be independently controlled, rather than being gated
behind HasDebug || HasServe. Minimal/embedded builds that omit both
debug and serve were getting 404s on watch-ipn-bus, breaking
"tailscale up --authkey=..." and other CLI flows that depend on
WatchIPNBus.
In the CLI, check buildfeatures.HasIPNBus before attempting to watch
the IPN bus in "tailscale up"/"tailscale login", and exit early with
an informational message when the feature is omitted.
Also add the missing NewCounterFunc stub to clientmetric/omit.go,
which caused compilation errors when building with
ts_omit_clientmetrics and netstack enabled.
Fixes#19240
Change-Id: I2e3c69a72fc50fa02542b91b8a54859618a463d1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
If an entry in the tsmpLearnedDisco does not match the disco key of the
key currently being processed, overwrite the key, and leave the entry in
the map for later processing.
In reality, this should not happen, but is put in as a safety measure
with logging of the situation so we can replicate the behaviour and
correct it should it happen.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
After moving around locks in 4334dfa7d5,
a data race were made possible.
Introduce an RWlock to the mapSession itself for fetching peers.
Fixes#19260
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
When a recording upload fails mid-session, the recording goroutine
cancels the session context. This triggers two concurrent paths:
exec.CommandContext kills the process (causing cmd.Wait to return),
and killProcessOnContextDone tries to write the termination message
via exitOnce.Do. If cmd.Wait returns first, the main goroutine's
exitOnce.Do(func(){}) steals the once, and the termination message
is never written to the client.
Fix by waiting for killProcessOnContextDone to finish writing the
termination message (via <-ss.exitHandled) before claiming exitOnce,
when the context is already done.
Also fix the fallback path when launchProcess itself fails due to
context cancellation: use SSHTerminationMessage() with the correct
"\r\n\r\n" framing instead of fmt.Fprintf with the internal error
string.
Deflakes TestSSHRecordingCancelsSessionsOnUploadFailure, which was
failing consistently at a low rate due to the exitOnce race. After
this fix, flakestress passes with 8,668 runs, 0 failures.
Fixes#7707 (again. hopefully for good.)
Change-Id: I5ab911c71574db8d3f9d979fb839f273be51ecf9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Brings in a newer version of Gliderlabs SSH with added socket forwarding support.
Fixes#12409Fixes#5295
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Investigating battery costs on a busy tailnet I noticed a large number
of nodes regularly reconnecting to control and DERP. In one case I was
able to analyze closely `pmset` reported the every-minute wake-ups being
triggered by bluetooth. The node was by side effect reconnecting to
control constantly, and this was at times visible to peers as well.
Three changes here improve the situation:
- Short time jumps (less than 10 minutes) no longer produce "major
network change" events, and so do not trigger full rebind/reconnect.
- Many "incidental" fields on interfaces are ignored, like MTU, flags
and so on - if the route is still good, the rest should be manageable.
- Additional log output will provide more detail about the cause of
major network change events.
Updates #3363
Signed-off-by: James Tucker <james@tailscale.com>
Set csrf.Path("/") so the CSRF cookie is available across all routes,
not just the path where it was set.
Add helpers to expose the gorilla/csrf token for use.
Updates #19264
Signed-off-by: Fernando Serboncini <fserb@tailscale.com>
Commit f905871fb moved host key generation from the ipnLocalBackend
interface (GetSSH_HostKeys) to the standalone getHostKeys function,
which requires either system host keys in /etc/ssh/ or a valid
TailscaleVarRoot to generate keys into. The testBackend returned ""
for TailscaleVarRoot, and the Docker test containers only install
openssh-client (no server host keys), so getHostKeys always failed.
When getHostKeys fails, HandleSSHConn returns the error but never
closes the TCP connection, so SSH clients hang forever waiting for
the server hello.
Fix by creating a temp directory in TestMain and returning it from
testBackend.TailscaleVarRoot().
Regression from f905871fb#18949 ("ipn/ipnlocal, feature/ssh: move SSH code
out of LocalBackend to feature").
I was apparently too impatient to wait for the test to complete
and didn't connect the dots: https://github.com/tailscale/tailscale/actions/runs/22930275950
We should make that test faster (#19244) for the patience issue, but
also fail more nicely if this happens in the future.
Updates #19244
Change-Id: If82393b8f35413b04174e6f7d09a1ee3a2125a6b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The cloner and viewer code generators didn't handle named types
with basic underlying types (map/slice) that have their own Clone
or View methods. For example, a type like:
type Map map[string]any
func (m Map) Clone() Map { ... }
func (m Map) View() MapView { ... }
When used as a struct field, the cloner would descend into the
underlying map[string]any and fail because it can't clone the any
(interface{}) value type. Similarly, the viewer would try to create
a MapFnOf view and fail.
Fix the cloner to check for a Clone method on the named type
before falling through to the underlying type handling.
Fix the viewer to check for a View method on named map/slice types,
so the type author can provide a purpose-built safe view that
doesn't leak raw any values. Named map/slice types without a View
method fall through to normal handling, which correctly rejects
types like map[string]any as unsupported.
Updates tailscale/corp#39502 (needed by tailscale/corp#39594)
Change-Id: Iaef0192a221e02b4b8e409c99ef8398090327744
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
To denoise log output, to make it easier to find real failures.
Updates #19252
Change-Id: Iae64a9278c70de24a236c39e3d181a509a512a0b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The -run "^$" flag was being mangled by cmd.exe's argument processing.
The ^ character is cmd.exe's escape character, so go.cmd's cmd.exe layer
eats it, turning -run "^$" into -run "$" which matches all test names.
This caused the benchmark job to run every test, leading to timeouts
and Go runtime crashes.
Use -run XXXXNothingXXXX instead, which avoids special characters
entirely.
Updates #19252
Change-Id: I888c124254dd2767a40b61bcd68dbc9b22ad35a1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The upload-client-metrics handler called metricCapture without
checking if it was nil or if the metrics slice was empty. Most
tests pass nil for metricCapture, so if a metrics upload races
in during the test, it panics.
Fixes#19252
Change-Id: Ib904d1fe6779067dc2a153d1680b8f50cba9c773
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Add a new vet analyzer that checks t.Run subtest names don't contain
characters requiring quoting when re-running via "go test -run". This
enforces the style guide rule: don't use spaces or punctuation in
subtest names.
The analyzer flags:
- Direct t.Run calls with string literal names containing spaces,
regex metacharacters, quotes, or other problematic characters
- Table-driven t.Run(tt.name, ...) calls where tt ranges over a
slice/map literal with bad name field values
Also fix all 978 existing violations across 81 test files, replacing
spaces with hyphens and shortening long sentence-like names to concise
hyphenated forms.
Updates #19242
Change-Id: Ib0ad96a111bd8e764582d1d4902fe2599454ab65
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
TestGocrossWrapper will fail when run inside a git linked worktree
because Go 1.26 and earlier cannot get the current revision hash.
Since this will be fixed in Go 1.27, see golang/go#58218, this patch
skips this test until that release.
Fixes#19217
Signed-off-by: Simon Law <sfllaw@tailscale.com>
The test sets up an HTTP-over-Unix server and a reverse proxy pointed at
this server, but prior to this change did not round-trip anything to the
backing server. This change ensures that we test code paths which proxy
Unix sockets for serve.
Fixes#19232
Signed-off-by: Harry Harpham <harry@tailscale.com>
This is a follow-up to #19117, adding a debug CLI command allowing the operator
to explicitly discard cached netmap data, as a safety and recovery measure.
Updates #12639
Change-Id: I5c3c47c0204754b9c8e526a4ff8f69d6974db6d0
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
When getting a full map from control, disco keys for the nodes will also
be delivered. When communicating with a peer that is running without
being connected to control, and having that connection running based on
a TSMP learned disco key, we need to avoid overwriting the disco key for
that peer with the stale one control knows about.
Add a filter that filteres out keys from control, and replace them with
the TSMP learned disco keys.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
* cmd/k8s-operator/e2e: add L7 HA ingress test
Change-Id: Ic017e4a7e3affbc3e2a87b9b6b9c38afd65f32ed
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
* cmd/k8s-operator: add further E2E tests for Ingress (#34833)
This change adds E2E tests for L3 HA Ingress and L7 Ingress (Standalone and
HA). Updates the existing L3 Ingress test to use the Service's Magic DNS
name to test connectivity.
Also refactors test setup to set TS_DEBUG_ACME_DIRECTORY_URL only for tests
running against devcontrol, and updates the Kind node image from v1.30.0 to
v1.35.0.
Fixestailscale/corp#34833
Signed-off-by: Becky Pauley <becky@tailscale.com>
---------
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
Signed-off-by: Becky Pauley <becky@tailscale.com>
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
We have ~2.5k nodes running Void Linux, which report a version string
like `1.96.2_1 (Void Linux)`. Previously these versions would fail to
parse, because we only expect a hyphen and `extraCommits` after the
major/minor/patch numbers.
Fix the version parsing logic to handle this case.
Updates #19148
Change-Id: Ica4f172d080af266af7f0d69bb31483a095cd199
Signed-off-by: Alex Chan <alexc@tailscale.com>
Add a new tailcfg.NodeCapability (NodeAttrCacheNetworkMaps) to control whether
a node with support for caching network maps will attempt to do so. Update the
capability version to reflect this change (mainly as a safety measure, as the
control plane does not currently need to know about it).
Use the presence (or absence) of the node attribute to decide whether to create
and update a netmap cache for each profile. If caching is disabled, discard the
cached data; this allows us to use the presence of a cached netmap as an
indicator it should be used (unless explicitly overridden). Add a test that
verifies the attribute is respected. Reverse the sense of the environment knob
to be true by default, with an override to disable caching at the client
regardless what the node attribute says.
Move the creation/update of the netmap cache (when enabled) until after
successfully applying the network map, to reduce the possibility that we will
cache (and thus reuse after a restart) a network map that fails to correctly
configure the client.
Updates #12639
Change-Id: I1df4dd791fdb485c6472a9f741037db6ed20c47e
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
Instead of sending out disco keys via TSMP once, send them out in
intervals of 60+ seconds. The trigger is still callmemaaybe and the keys
will not be send if no direct connection needs to be established.
This fixes a case where a node can have stale keys but have communicated
with the other peer before, leading to an infinite DERP state.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
In #10057, @seigel pointed out an inconsistency in the help text for
`exit-node list` and `set --exit-node`:
1. Use `tailscale exit-node list`, which has a column titled "hostname"
and tells you that you can use a hostname with `set --exit-node`:
```console
$ tailscale exit-node list
IP HOSTNAME COUNTRY CITY STATUS
100.98.193.6 linode-vps.tailfa84dd.ts.net - - -
[…]
100.93.242.75 ua-iev-wg-001.mullvad.ts.net Ukraine Kyiv -
# To view the complete list of exit nodes for a country, use `tailscale exit-node list --filter=` followed by the country name.
# To use an exit node, use `tailscale set --exit-node=` followed by the hostname or IP.
# To have Tailscale suggest an exit node, use `tailscale exit-node suggest`.
```
(This is the same format hostnames are presented in the admin
console.)
2. Try copy/pasting a hostname into `set --exit-node`:
```console
$ tailscale set --exit-node=linode-vps.tailfa84dd.ts.net
invalid value "linode-vps.tailfa84dd.ts.net" for --exit-node; must be IP or unique node name
```
3. Note that the command allows some hostnames, if they're from nodes
in a different tailnet:
```console
$ tailscale set --exit-node= ua-iev-wg-001.mullvad.ts.net
$ echo $?
0
```
This patch addresses the inconsistency in two ways:
1. Allow using `tailscale set --exit-node=` with an FQDN that's missing
the trailing dot, matching the formatting used in `exit-node list`
and the admin console.
2. Make the description of valid exit nodes consistent across commands
("hostname or IP").
Updates #10057
Change-Id: If5d74f950cc1a9cc4b0ebc0c2f2d70689ffe4d73
Signed-off-by: Alex Chan <alexc@tailscale.com>
This avoids putting "DisablementSecrets" in the JSON output from
`tailscale lock log`, which is potentially scary to somebody who doesn't
understand the distinction.
AUMs are stored and transmitted in CBOR-encoded format, which uses an
integer rather than a string key, so this doesn't break already-created
TKAs.
Fixes#19189
Change-Id: I15b4e81a7cef724a450bafcfa0b938da223c78c9
Signed-off-by: Alex Chan <alexc@tailscale.com>
Reports whether the current binary was built with Tailscale's
custom Go toolchain (the "tailscale_go" build tag).
For https://github.com/tailscale/go/pull/165
Updates tailscale/corp#39430
Change-Id: Ica437582ddf55d7df48b1453bad03ce14b1c0949
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
* Refer to "tailnet-lock" instead of "network-lock" in log messages
* Log keys as `tlpub:<hex>` rather than as Go structs
Updates tailscale/corp#39455
Updates tailscale/corp#37904
Change-Id: I644407d1eda029ee11027bcc949897aa4ba52787
Signed-off-by: Alex Chan <alexc@tailscale.com>
Prior to this change, closing multiple ServiceListeners concurrently
could result in failures as the independent close operations vie for the
attention of the Server's LocalBackend. The close operations would each
obtain the current ETag of the serve config and try to write new serve
config using this ETag. When one write invalidated the ETag of another,
the latter would fail. Exacerbating the issue, ServiceListener.Close
cannot be retried.
This change resolves the bug by using Server.mu to synchronize across
all ServiceListener.Close operations, ensuring they happen serially.
Fixes#19169
Signed-off-by: Harry Harpham <harry@tailscale.com>
This is a regression test for #19166, in which it was discovered that
after calling Server.ListenService for multiple Services, only the
Service from the most recent call would be advertised.
The bug was fixed in 99f8039101
Updates #19166
Signed-off-by: Harry Harpham <harry@tailscale.com>
This makes the limits easier to find and change, rather than scattering
them across the TKA code.
Updates #cleanup
Change-Id: I2f9b3b83d293eebb2572fa7bb6de2ca1f3d9a192
Signed-off-by: Alex Chan <alexc@tailscale.com>
The disco key subscriber could deadlock in a scenario where a self node
update came through the control path into the mapSession after the disco
key subscriber had taken the lock, but before it had pushed the netmap
change, as both the subscriber and onSelfNodeChanged needs the
controlclient lock.
The subscriber can safely take the mapsession as the changequeue has its
own lock for inserting records, and also checks if the queue has been
closed before inserting.
Updates #12639
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Brad Fitzpatrick
Claus Lensbøl
Tom Meadows
Jordan Whited
Doug Bryant
Mike O'Driscoll
License Updater
Fernando Serboncini
Fran Bull
Nick Khyl
Kristoffer Dalby
James Tucker
Dylan Bargatze
Simon Law
Harry Harpham
M. J. Fromberger
Naman Sood
BeckyPauley
Alex Chan