Add HTTPBandwidth/HTTPBandwidthWithDialAddr probe classes that download a
fixed number of bytes and record transfer time and bytes transferred as
Prometheus counters for bandwidth measurement, plus HTTPWithDialAddr and
the shared NewProbeTransport and HTTPBandwidthMetrics helpers.
The dial-address override lets a probe target a specific backend (e.g. a
single Funnel ingress node) while SNI, the Host header, and TLS cert
validation continue to derive from the URL host. HTTPBandwidthMetrics is
exported so other bandwidth probes (e.g. a receiver-reported upload probe)
emit an identical metric set and compare under a shared direction label.
Updates tailscale/corp#41587
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add a vmtest that guards the fix in #20025: after an in-process control
client swap (profile switch / interactive re-login), magicsock's NetInfo
dedup cache (netInfoLast) must be cleared so the structurally-identical
post-switch NetInfo (same PreferredDERP, same NAT shape) is re-reported to
the new control session rather than suppressed as unchanged.
The test brings a node up, pins its home DERP so the reported NetInfo is
identical across the switch, records the home DERP the test control learned,
switches to a fresh login profile on the same control/network/NAT/DERP, and
asserts the control re-learns the same non-zero home DERP for the node's new
identity. Without ResetNetInfoLast the assertion times out at HomeDERP=0.
To support this, vnet now serves the test control on port 443 (TLS) in
addition to port 80: an immediate re-login makes a fresh noise dial, and
because the prior dial was recent the control client forces an HTTPS (443)
dial (controlhttp.Dialer.forceNoise443), which the harness previously did
not answer. The control endpoint gets its own self-signed cert (the existing
selfSignedDERPCert helper, renamed to the generic selfSignedCert); the cert
is not validated since control noise dials authenticate via the Noise
handshake, so it only needs a TLS peer to complete the forced 443 dial.
Add Env.ForcePreferredDERP and Env.Relogin helpers for the above.
Updates #20024
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
magicsock de-duplicates NetInfo callbacks against c.netInfoLast, a cache
that lives on the long-lived magicsock.Conn. That cache survives a control
client swap (interactive login or profile switch), where only the control
client (and its own per-client NetInfo dedup) is replaced. As a result, the
first netcheck after the swap produces a structurally-identical NetInfo
(same PreferredDERP, same NAT shape), magicsock suppresses it as unchanged,
and the new control session never learns our home DERP. Peers can't reach
the node over DERP until some unrelated NetInfo field happens to change.
Add Conn.ResetNetInfoLast to clear the dedup cache, and call it from
LocalBackend.setControlClientLocked whenever a control client is installed,
so the next netcheck re-reports the current NetInfo to the new client.
netInfoLast is only a dedup/optimization cache (all readers nil-guard, and
it is recomputed by every netcheck), so clearing it can only add a delivery,
never lose or misroute one; it is scoped to control-client lifecycle events,
not steady-state operation.
Updates #17887Fixes#20024
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Fix the following issues:
1. Endianness Bug: The nftables runner used hardcoded
big-endian byte arrays for firewall mark values (0xff0000, etc.), breaking
bitwise operations on little-endian systems (all x86/x64, ARM). This caused
connmark save/restore rules to silently fail. Fixed by using
binary.NativeEndian to generate correct byte order for the host system.
2. Connmark Restore Conditional Check: The connmark restore
mechanism unconditionally overwrote packet marks, even when Tailscale
hadn't set any mark bits in conntrack. This destroyed mark bits set by
other systems (VPNs, policy routing, vendor flags), breaking coexistence.
Fixed by adding a conditional check to only restore when (ct mark &
0xff0000) != 0, preventing the worst case of wiping all marks to zero.
Changes:
- util/linuxfw/linuxfw.go: Added nativeEndianUint32() helper and updated
all mask functions to use native byte order instead of hardcoded bytes
- util/linuxfw/nftables_runner.go: Added conditional check in
makeConnmarkRestoreExprs() to only restore when ct mark has Tailscale
bits set; added detailed comment about bit preservation limitations
- util/linuxfw/iptables_runner.go: Added conditional check using -m
connmark ! --mark to match nftables behavior
- Tests updated: Fixed byte-level regression tests to expect little-endian
byte sequences and verify the new conditional check
Note: Perfect bit preservation in nftables remains challenging
due to nftables expression VM limitations. The current implementation
prevents the critical case of wiping marks with zero.
Updates #3310Fixes#11803
Related to #8555
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
On systems where this sysctl defaults to 0 (including GCP VMs), rp_filter performs its lookup with fwmark=0, hits rule 5270 then table 52 and routes to 0.0.0.0/0 dev tailscale0, and drops every reply packet arriving on the physical interface as a martian. This breaks all connectivity when using an exit node: DERP, DNS, control plane, and even the cloud metadata service.
Set src_valid_mark=1 when enabling the connmark rules so the rp_filter workaround actually works in these cases.
Updates #3310
Updates tailscale/corp#37846
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add a --rate-config flag pointing to a JSON file for per-client receive
rate limits (bytes/sec and burst bytes). The config is reloaded on SIGHUP,
updating all existing client connections live. The --per-client-rate-limit
and --per-client-rate-burst flags are removed in favor of the config file.
In derpserver, rate limiting uses an atomic.Pointer[xrate.Limiter] per
client: nil when unlimited or mesh (zero overhead), non-nil when
rate-limited.
Document that clientSet.activeClient Store operations require Server.mu.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add server-side per-client bandwidth enforcement using TCP backpressure.
When configured, the server calls WaitN after reading each DERP frame,
which delays the next read, fills the TCP receive buffer, shrinks
the TCP window, and naturally throttles the sender — no packets are dropped.
- Rate limiting is on the receive (inbound) side, which is what an abusive
client controls
- Mesh peers are exempt since they are trusted infrastructure
- The burst size is at least MaxPacketSize (64KB) to ensure a
single max-size frame can always be processed
Also refactors sclient to store a context.Context directly instead of a
done channel, which simplifies the rate limiter's WaitN call.
Flags added to cmd/derper:
--per-client-rate-limit (bytes/sec, default 0 = unlimited)
--per-client-rate-burst (bytes, default 0 = 2x rate limit)
Example for 10Mbps: --per-client-rate-limit=1250000
Updates #38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use bufio.Writer.AvailableBuffer to write the 32-byte public key
directly into bufio's internal buffer as a single append+Write,
avoiding 32 separate WriteByte calls. Fall back to the existing
byte-at-a-time path when the buffer has insufficient space.
```
name old ns/op new ns/op speedup
NodeWriteRawWithoutAllocating-8 121 12.5 ~9.7x
(0 allocs/op in both)
```
Add BenchmarkNodeWriteRawWithoutAllocating and expand
TestNodeWriteRawWithoutAllocating to cover both fast (AvailableBuffer)
and slow (WriteByte fallback) paths with correctness and allocation
checks.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use bufio.Writer.AvailableBuffer to write the frame header directly
into bufio's internal buffer as a single append+Write, avoiding 5
separate WriteByte calls. Fall back to the existing writeUint32
byte-at-a-time path when the buffer has insufficient space.
```
name old ns/op new ns/op speedup
WriteFrameHeader-8 18.8 7.8 ~2.4x
(0 allocs/op in both)
```
Add TestWriteFrameHeader with correctness
checks, allocation assertions, and coverage of both fast and slow
write paths. Move BenchmarkReadFrameHeader from client_test.go to
derp_test.go alongside BenchmarkWriteFrameHeader, co-located with
the functions under test.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Replace byte-at-a-time ReadByte loops with Peek+Discard in the DERP
read path. Peek returns a slice into bufio's internal buffer without
allocating, and Discard advances the read pointer without copying.
Introduce util/bufiox with a BufferedReader interface and ReadFull
helper that uses Peek+copy+Discard as an allocation-free alternative
to io.ReadFull.
- derp.ReadFrameHeader: replace 5× ReadByte with Peek(5)+Discard(5),
reading the frame type and length directly from the peeked slice.
Remove now-unused readUint32 helper.
name old ns/op new ns/op speedup
ReadFrameHeader-8 24.2 12.4 ~2x
(0 allocs/op in both)
- key.NodePublic.ReadRawWithoutAllocating: replace 32× ReadByte with
bufiox.ReadFull. Addresses the "Dear future" comment about switching
away from byte-at-a-time reads once a non-escaping alternative exists.
name old ns/op new ns/op speedup
NodeReadRawWithoutAllocating-8 140 43.6 ~3.2x
(0 allocs/op in both)
- derpserver.handleFramePing: replace io.ReadFull with bufiox.ReadFull.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Currently IP forwarding health check is done on sending MapRequests.
Move ip forwarding to the health service to gain the benefits
of the health tracker and perodic monitoring out of band from
the MapRequest path. ipnlocal now provides a closure to
the health service to provide the check if forwarding is broken.
Removed `skipIPForwardingCheck` from controlclient/direct.go,
it wasn't being used as the comments describe it, that check
has moved to ipnlocal for the closure to the health tracker.
Updates #18976
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
When IPv6 is unavailable on a system, AddConnmarkSaveRule() and
DelConnmarkSaveRule() would panic with a nil pointer dereference.
Both methods directly iterated over []iptablesInterface{i.ipt4, i.ipt6}
without checking if ipt6 was nil.
Use `getTables()` instead to properly retrieve the available tables
on a given system
Fixes#3310
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The txRecords buffer had two compounding bugs that caused the
overflow guard to fire on every send tick under high DERP server load,
spamming logs at the full send rate (e.g. 100x/second).
First, int(packetTimeout.Seconds()) truncates fractional-second timeouts,
under-allocating the buffer. Second, the capacity was sized to exactly the
theoretical maximum number of in-flight records with no headroom,
and the expiry check used strict > rather than >=, so records at exactly
the timeout boundary were never evicted by applyTimeouts,
leaving len==cap on the very next tick.
Fixestailscale/corp#37696
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
PR #18860 adds firewall rules in the mangle table to save outbound packet
marks to conntrack and restore them on reply packets before the routing
decision. When reply packets have their marks restored, the kernel uses
the correct routing table (based on the mark) and the packets pass the
rp_filter check.
This makes the risk check and reverse path filtering warnings unnecessary.
Updates #3310Fixestailscale/corp#37846
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
When a Linux system acts as an exit node or subnet router with strict
reverse path filtering (rp_filter=1), reply packets may
be dropped because they fail the RPF check. Reply packets arrive on the
WAN interface but the routing table indicates they should have arrived
on the Tailscale interface, causing the kernel to drop them.
This adds firewall rules in the mangle table to save outbound packet
marks to conntrack and restore them on reply packets before the routing
decision. When reply packets have their marks restored, the kernel uses
the correct routing table (based on the mark) and the packets pass the
rp_filter check.
Implementation adds two rules per address family (IPv4/IPv6):
- mangle/OUTPUT: Save packet marks to conntrack for NEW connections
with non-zero marks in the Tailscale fwmark range (0xff0000)
- mangle/PREROUTING: Restore marks from conntrack to packets for
ESTABLISHED,RELATED connections before routing decision and rp_filter
check
The workaround is automatically enabled when UseConnmarkForRPFilter is
set in the router configuration, which happens when subnet routes are
advertised on Linux systems.
Both iptables and nftables implementations are provided, with automatic
backend detection.
Fixes#3310Fixes#14409Fixes#12022Fixes#15815Fixes#9612
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The test was making HTTP requests before waiting for probes to
complete their initial run in "once" mode. This created a race where
sometimes the probe's previous state was empty (0 results) and
sometimes it had one result, causing inconsistent RecentResults and
PreviousSuccessRatio values.
Fixed by waiting for all probes to complete via their stopped channels
before making HTTP requests, matching the pattern used in other tests
like TestProberRunHandler and TestRunAllHandler.
Fixes#18806
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update Nix flake to use go 1.25.2
Create the hash from the toolchain rev file automatically from
update-flake.sh
Updates tailscale/go#135
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update odic-funnel-clients.json to take a path, this
allows setting the location of the file and prevents
it from landing in the root directory or users home directory.
Move setting of rootPath until after tsnet has started.
Previously this was added for the lazy creation of the
oidc-key.json. It's now needed earlier in the flow.
Updates #16734Fixes#16844
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update the runall handler to be more generic with an
exclude param to exclude multiple probes as the requesters
definition.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Cleanup nix support, make flake easier to read with nix-systems.
This also harmonizes with golinks flake setup and reduces an input
dependency by 1.
Update deps test to ensure the vendor hash stays harmonized
with go.mod.
Update make tidy to ensure vendor hash stays current.
Overlay the current version of golang, tailscale runs
recent releases faster than nixpkgs can update them into
the unstable branch.
Updates #16637
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add a Run all probes handler that executes all
probes except those that are continuous or the derpmap
probe.
This is leveraged by other tooling to confirm DERP
stability after a deploy.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update some logging to help future failures.
Improve test shutdown concurrency issues.
Fixes#16722
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The tsidp oidc-key.json ended up in the root directory
or home dir of the user process running it.
Update this to store it in a known location respecting
the TS_STATE_DIR and flagDir options.
Fixes#16734
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update nixpkgs-unstable to include newer golang
to satisfy go.mod requirement of 1.24.4
Update vendor hash to current.
Updates #15015
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The caller of client.RunWatchConnectionLoop may need to be
aware of errors that occur within loop. Add a channel
that notifies of errors to the caller to allow for
decisions to be make as to the state of the client.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add mesh key support to derpprobe for
probing derpers with verify set to true.
Move MeshKey checking to central point for code reuse.
Fix a bad error fmt msg.
Fixestailscale/corp#27294Fixestailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Fix the wireshark lua dissector to support 0 bit position
and not throw modulo div by 0 errors.
Add new disco frame types to the decoder.
Updates tailscale/corp#29036
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use of the httptest client doesn't render header ordering
as expected.
Use http.DefaultClient for the test to ensure that
the header ordering test is valid.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Content-type was responding as test/plain for probes
accepting application/json. Set content type header
before setting the response code to correct this.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
OCSP has been removed from the LE certs.
Use CRL verification instead.
If a cert provides a CRL, check its revocation
status, if no CRL is provided and otherwise
is valid, pass the check.
Fixes#15912
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Co-authored-by: Simon Law <sfllaw@tailscale.com>
Add a label which differentiates the address family
for STUN checks.
Also initialize the derpprobe_attempts_total and
derpprobe_seconds_total metrics by adding 0 for
the alternate fail/ok case.
Updates tailscale/corp#27249
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Incorrect disabled support for not having a mesh key in
d5316a4fbb
Allow for no mesh key to be set.
Fixes#14928
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Since dynamic reload of setec is not supported
in derper at this time, close the server after
the secret is loaded.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
A previous PR accidentally logged the key as part
of an error. Remove logging of the key.
Add log print for Setec store steup.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add setec secret support for derper.
Support dev mode via env var, and setec via secrets URL.
For backwards compatibility use setec load from file also.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Remove "unexpected" labelling of PeerGoneReasonNotHere.
A peer being no longer connected to a DERP server
is not an unexpected case and causes confusion in looking at logs.
Fixestailscale/corp#25609
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use envknob to configure the per client send
queue depth for the derp server.
Fixestailscale/corp#24978
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Mike O'Driscoll
