Add a --rate-config flag pointing to a JSON file for per-client receive
rate limits (bytes/sec and burst bytes). The config is reloaded on SIGHUP,
updating all existing client connections live. The --per-client-rate-limit
and --per-client-rate-burst flags are removed in favor of the config file.
In derpserver, rate limiting uses an atomic.Pointer[xrate.Limiter] per
client: nil when unlimited or mesh (zero overhead), non-nil when
rate-limited.
Document that clientSet.activeClient Store operations require Server.mu.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add server-side per-client bandwidth enforcement using TCP backpressure.
When configured, the server calls WaitN after reading each DERP frame,
which delays the next read, fills the TCP receive buffer, shrinks
the TCP window, and naturally throttles the sender — no packets are dropped.
- Rate limiting is on the receive (inbound) side, which is what an abusive
client controls
- Mesh peers are exempt since they are trusted infrastructure
- The burst size is at least MaxPacketSize (64KB) to ensure a
single max-size frame can always be processed
Also refactors sclient to store a context.Context directly instead of a
done channel, which simplifies the rate limiter's WaitN call.
Flags added to cmd/derper:
--per-client-rate-limit (bytes/sec, default 0 = unlimited)
--per-client-rate-burst (bytes, default 0 = 2x rate limit)
Example for 10Mbps: --per-client-rate-limit=1250000
Updates #38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use bufio.Writer.AvailableBuffer to write the 32-byte public key
directly into bufio's internal buffer as a single append+Write,
avoiding 32 separate WriteByte calls. Fall back to the existing
byte-at-a-time path when the buffer has insufficient space.
```
name old ns/op new ns/op speedup
NodeWriteRawWithoutAllocating-8 121 12.5 ~9.7x
(0 allocs/op in both)
```
Add BenchmarkNodeWriteRawWithoutAllocating and expand
TestNodeWriteRawWithoutAllocating to cover both fast (AvailableBuffer)
and slow (WriteByte fallback) paths with correctness and allocation
checks.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use bufio.Writer.AvailableBuffer to write the frame header directly
into bufio's internal buffer as a single append+Write, avoiding 5
separate WriteByte calls. Fall back to the existing writeUint32
byte-at-a-time path when the buffer has insufficient space.
```
name old ns/op new ns/op speedup
WriteFrameHeader-8 18.8 7.8 ~2.4x
(0 allocs/op in both)
```
Add TestWriteFrameHeader with correctness
checks, allocation assertions, and coverage of both fast and slow
write paths. Move BenchmarkReadFrameHeader from client_test.go to
derp_test.go alongside BenchmarkWriteFrameHeader, co-located with
the functions under test.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Replace byte-at-a-time ReadByte loops with Peek+Discard in the DERP
read path. Peek returns a slice into bufio's internal buffer without
allocating, and Discard advances the read pointer without copying.
Introduce util/bufiox with a BufferedReader interface and ReadFull
helper that uses Peek+copy+Discard as an allocation-free alternative
to io.ReadFull.
- derp.ReadFrameHeader: replace 5× ReadByte with Peek(5)+Discard(5),
reading the frame type and length directly from the peeked slice.
Remove now-unused readUint32 helper.
name old ns/op new ns/op speedup
ReadFrameHeader-8 24.2 12.4 ~2x
(0 allocs/op in both)
- key.NodePublic.ReadRawWithoutAllocating: replace 32× ReadByte with
bufiox.ReadFull. Addresses the "Dear future" comment about switching
away from byte-at-a-time reads once a non-escaping alternative exists.
name old ns/op new ns/op speedup
NodeReadRawWithoutAllocating-8 140 43.6 ~3.2x
(0 allocs/op in both)
- derpserver.handleFramePing: replace io.ReadFull with bufiox.ReadFull.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Currently IP forwarding health check is done on sending MapRequests.
Move ip forwarding to the health service to gain the benefits
of the health tracker and perodic monitoring out of band from
the MapRequest path. ipnlocal now provides a closure to
the health service to provide the check if forwarding is broken.
Removed `skipIPForwardingCheck` from controlclient/direct.go,
it wasn't being used as the comments describe it, that check
has moved to ipnlocal for the closure to the health tracker.
Updates #18976
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
When IPv6 is unavailable on a system, AddConnmarkSaveRule() and
DelConnmarkSaveRule() would panic with a nil pointer dereference.
Both methods directly iterated over []iptablesInterface{i.ipt4, i.ipt6}
without checking if ipt6 was nil.
Use `getTables()` instead to properly retrieve the available tables
on a given system
Fixes#3310
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The txRecords buffer had two compounding bugs that caused the
overflow guard to fire on every send tick under high DERP server load,
spamming logs at the full send rate (e.g. 100x/second).
First, int(packetTimeout.Seconds()) truncates fractional-second timeouts,
under-allocating the buffer. Second, the capacity was sized to exactly the
theoretical maximum number of in-flight records with no headroom,
and the expiry check used strict > rather than >=, so records at exactly
the timeout boundary were never evicted by applyTimeouts,
leaving len==cap on the very next tick.
Fixestailscale/corp#37696
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
PR #18860 adds firewall rules in the mangle table to save outbound packet
marks to conntrack and restore them on reply packets before the routing
decision. When reply packets have their marks restored, the kernel uses
the correct routing table (based on the mark) and the packets pass the
rp_filter check.
This makes the risk check and reverse path filtering warnings unnecessary.
Updates #3310Fixestailscale/corp#37846
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
When a Linux system acts as an exit node or subnet router with strict
reverse path filtering (rp_filter=1), reply packets may
be dropped because they fail the RPF check. Reply packets arrive on the
WAN interface but the routing table indicates they should have arrived
on the Tailscale interface, causing the kernel to drop them.
This adds firewall rules in the mangle table to save outbound packet
marks to conntrack and restore them on reply packets before the routing
decision. When reply packets have their marks restored, the kernel uses
the correct routing table (based on the mark) and the packets pass the
rp_filter check.
Implementation adds two rules per address family (IPv4/IPv6):
- mangle/OUTPUT: Save packet marks to conntrack for NEW connections
with non-zero marks in the Tailscale fwmark range (0xff0000)
- mangle/PREROUTING: Restore marks from conntrack to packets for
ESTABLISHED,RELATED connections before routing decision and rp_filter
check
The workaround is automatically enabled when UseConnmarkForRPFilter is
set in the router configuration, which happens when subnet routes are
advertised on Linux systems.
Both iptables and nftables implementations are provided, with automatic
backend detection.
Fixes#3310Fixes#14409Fixes#12022Fixes#15815Fixes#9612
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The test was making HTTP requests before waiting for probes to
complete their initial run in "once" mode. This created a race where
sometimes the probe's previous state was empty (0 results) and
sometimes it had one result, causing inconsistent RecentResults and
PreviousSuccessRatio values.
Fixed by waiting for all probes to complete via their stopped channels
before making HTTP requests, matching the pattern used in other tests
like TestProberRunHandler and TestRunAllHandler.
Fixes#18806
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update Nix flake to use go 1.25.2
Create the hash from the toolchain rev file automatically from
update-flake.sh
Updates tailscale/go#135
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update odic-funnel-clients.json to take a path, this
allows setting the location of the file and prevents
it from landing in the root directory or users home directory.
Move setting of rootPath until after tsnet has started.
Previously this was added for the lazy creation of the
oidc-key.json. It's now needed earlier in the flow.
Updates #16734Fixes#16844
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update the runall handler to be more generic with an
exclude param to exclude multiple probes as the requesters
definition.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Cleanup nix support, make flake easier to read with nix-systems.
This also harmonizes with golinks flake setup and reduces an input
dependency by 1.
Update deps test to ensure the vendor hash stays harmonized
with go.mod.
Update make tidy to ensure vendor hash stays current.
Overlay the current version of golang, tailscale runs
recent releases faster than nixpkgs can update them into
the unstable branch.
Updates #16637
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add a Run all probes handler that executes all
probes except those that are continuous or the derpmap
probe.
This is leveraged by other tooling to confirm DERP
stability after a deploy.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update some logging to help future failures.
Improve test shutdown concurrency issues.
Fixes#16722
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The tsidp oidc-key.json ended up in the root directory
or home dir of the user process running it.
Update this to store it in a known location respecting
the TS_STATE_DIR and flagDir options.
Fixes#16734
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Update nixpkgs-unstable to include newer golang
to satisfy go.mod requirement of 1.24.4
Update vendor hash to current.
Updates #15015
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
The caller of client.RunWatchConnectionLoop may need to be
aware of errors that occur within loop. Add a channel
that notifies of errors to the caller to allow for
decisions to be make as to the state of the client.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add mesh key support to derpprobe for
probing derpers with verify set to true.
Move MeshKey checking to central point for code reuse.
Fix a bad error fmt msg.
Fixestailscale/corp#27294Fixestailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Fix the wireshark lua dissector to support 0 bit position
and not throw modulo div by 0 errors.
Add new disco frame types to the decoder.
Updates tailscale/corp#29036
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use of the httptest client doesn't render header ordering
as expected.
Use http.DefaultClient for the test to ensure that
the header ordering test is valid.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Content-type was responding as test/plain for probes
accepting application/json. Set content type header
before setting the response code to correct this.
Updates tailscale/corp#27370
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
OCSP has been removed from the LE certs.
Use CRL verification instead.
If a cert provides a CRL, check its revocation
status, if no CRL is provided and otherwise
is valid, pass the check.
Fixes#15912
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Co-authored-by: Simon Law <sfllaw@tailscale.com>
Add a label which differentiates the address family
for STUN checks.
Also initialize the derpprobe_attempts_total and
derpprobe_seconds_total metrics by adding 0 for
the alternate fail/ok case.
Updates tailscale/corp#27249
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Incorrect disabled support for not having a mesh key in
d5316a4fbb
Allow for no mesh key to be set.
Fixes#14928
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Since dynamic reload of setec is not supported
in derper at this time, close the server after
the secret is loaded.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
A previous PR accidentally logged the key as part
of an error. Remove logging of the key.
Add log print for Setec store steup.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add setec secret support for derper.
Support dev mode via env var, and setec via secrets URL.
For backwards compatibility use setec load from file also.
Updates tailscale/corp#25756
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Remove "unexpected" labelling of PeerGoneReasonNotHere.
A peer being no longer connected to a DERP server
is not an unexpected case and causes confusion in looking at logs.
Fixestailscale/corp#25609
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Use envknob to configure the per client send
queue depth for the derp server.
Fixestailscale/corp#24978
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Mike O'DriscollandGitHub
