This reverts commit 6a93b17c8c.
The reverted commit added more complexity than it was worth at the
current stage. Handling delta CapVer changes requires extensive changes
to relayManager datastructures in order to also support delta updates of
relay servers.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
The relay server now fetches IPs from local interfaces and external
perspective IP:port's via netcheck (STUN).
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This enables us to mark nodes as relay capable or not. We don't actually
do that yet, as we haven't established a relay CapVer.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
It might complete, interrupting it reduces the chances of establishing a
relay path.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This is simply for consistency with relayManagerInputEvent(), which
should be the sole launcher of runLoop().
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
relayManager can now hand endpoint a relay epAddr for it to consider
as bestAddr.
endpoint and Conn disco ping/pong handling are now VNI-aware.
Updates tailscale/corp#27502
Updates tailscale/corp#29422
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit adds a new type to magicsock, epAddr, which largely ends up
replacing netip.AddrPort in packet I/O paths throughout, enabling
Geneve encapsulation over UDP awareness.
The conn.ReceiveFunc for UDP has been revamped to fix and more clearly
distinguish the different classes of packets we expect to receive: naked
STUN binding messages, naked disco, naked WireGuard, Geneve-encapsulated
disco, and Geneve-encapsulated WireGuard.
Prior to this commit, STUN matching logic in the RX path could swallow
a naked WireGuard packet if the keypair index, which is randomly
generated, happened to overlap with a subset of the STUN magic cookie.
Updates tailscale/corp#27502
Updates tailscale/corp#29326
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Our conn.Bind implementation is updated to make Send() offset-aware for
future VXLAN/Geneve encapsulation support.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
relayManager is responsible for disco ping/pong probing of relay
endpoints once a handshake is complete.
Future work will enable relayManager to set a relay endpoint as the best
UDP path on an endpoint if appropriate.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This type improves code clarity and reduces the chance of heap alloc as
we pass it as a non-pointer. VNI being a 3-byte value enables us to
track set vs unset via the reserved/unused byte.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
CallMeMaybeVia reception and endpoint allocation have been collapsed to
a single event channel. discoInfo caching for active relay handshakes
is now implemented.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
The event loop removes the need for growing locking complexities and
synchronization. Now we simply use channels. The event loop only runs
while there is active work to do.
relayManager remains no-op inside magicsock for the time being.
endpoints are never 'relayCapable' and therefore endpoint & Conn will
not feed CallMeMaybeVia or allocation events into it.
A number of relayManager events remain unimplemented, e.g.
CallMeMaybeVia reception and relay handshaking.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
ServerEndpoint will be used within magicsock and potentially elsewhere,
which should be possible without needing to import the server
implementation itself.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
relayManager will eventually be responsible for handling the allocation
and handshaking of UDP relay server endpoints.
relay servers are endpoint-independent, and Conn must already maintain
handshake state for all endpoints. This justifies a new data structure
to fill these roles.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Conn.handleDiscoMessage() now makes a distinction between relay
handshake disco messages and peer disco messages.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Conn.sendDiscoMessage() now verifies if the destination disco key is
associated with any known peer(s) in a thread-safe manner.
Updates #15844
Signed-off-by: Jordan Whited <jordan@tailscale.com>
The previous strategy assumed clients maintained adequate state to
understand the relationship between endpoint allocation and the server
it was allocated on.
magicsock will not have awareness of the server's disco key
pre-allocation, it only understands peerAPI address at this point. The
second client to allocate on the same server could trigger
re-allocation, breaking a functional relay server endpoint.
If magicsock needs to force reallocation we can add opt-in behaviors
for this later.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
tstime.GoDuration JSON serializes with time.Duration.String(), which is
more human-friendly than nanoseconds.
ServerEndpoint is currently experimental, therefore breaking changes
are tolerable.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
The relay server is still permanently disabled until node attribute
changes are wired up in a future commit.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This feature is "registered" as an ipnlocal.Extension, and
conditionally linked depending on GOOS and ts_omit_relayserver build
tag.
The feature is not linked on iOS in attempt to limit the impact to
binary size and resulting effect of pushing up against NetworkExtension
limits. Eventually we will want to support the relay server on iOS,
specifically on the Apple TV. Apple TVs are well-fitted to act as
underlay relay servers as they are effectively always-on servers.
This skeleton begins to tie a PeerAPI endpoint to a net/udprelay.Server.
The PeerAPI endpoint is currently no-op as
extension.shouldRunRelayServer() always returns false. Follow-up commits
will implement extension.shouldRunRelayServer().
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Query for the const quad-100 reverse DNS name, for which a forward
record will also be served. This test was previously dependent on
search domain behavior, and now it is not.
Updates #15607
Signed-off-by: Jordan Whited <jordan@tailscale.com>
The http.StatusMethodNotAllowed status code was being erroneously
set instead of http.StatusBadRequest in multiple places.
Updates #cleanup
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This flag is currently no-op and hidden. The flag does round trip
through the related pref. Subsequent commits will tie them to
net/udprelay.Server. There is no corresponding "tailscale up" flag,
enabling/disabling of the relay server will only be supported via
"tailscale set".
This is a string flag in order to support disablement via empty string
as a port value of 0 means "enable the server and listen on a random
unused port". Disablement via empty string also follows existing flag
convention, e.g. advertise-routes.
Early internal discussions settled on "tailscale set --relay="<port>",
but the author felt this was too ambiguous around client vs server, and
may cause confusion in the future if we add related flags.
Updates tailscale/corp#27502
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit implements an experimental UDP relay server. The UDP relay
server leverages the Disco protocol for a 3-way handshake between
client and server, along with 3 new Disco message types for said
handshake. These new Disco message types are also considered
experimental, and are not yet tied to a capver.
The server expects, and imposes, a Geneve (Generic Network
Virtualization Encapsulation) header immediately following the underlay
UDP header. Geneve protocol field values have been defined for Disco
and WireGuard. The Geneve control bit must be set for the handshake
between client and server, and unset for messages relayed between
clients through the server.
Updates tailscale/corp#27101
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Add an explicit case for exercising preferred DERP hysteresis around
the branch that compares latencies on a percentage basis.
Updates #cleanup
Signed-off-by: Jordan Whited <jordan@tailscale.com>
GetReport() may have side effects when the caller enforces a deadline
that is shorter than ReportTimeout.
Updates #13783
Updates #13394
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Write timeouts can be indicative of stalled TCP streams. Understanding
changes in the rate of such events can be helpful in an ops context.
Updates tailscale/corp#23668
Signed-off-by: Jordan Whited <jordan@tailscale.com>
netcheck.Client.GetReport() applies its own deadlines. This 2s deadline
was causing GetReport() to never fall back to HTTPS/ICMP measurements
as it was shorter than netcheck.stunProbeTimeout, leaving no time
for fallbacks.
Updates #13394
Updates #6187
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Disable TCP & UDP GRO if the probe fails.
torvalds/linux@e269d79c7d broke virtio_net
TCP & UDP GRO causing GRO writes to return EINVAL. The bug was then
resolved later in
torvalds/linux@89add40066. The offending
commit was pulled into various LTS releases.
Updates #13041
Signed-off-by: Jordan Whited <jordan@tailscale.com>
We've added more probe targets recently which has resulted in more
timeouts behind restrictive NATs in localized testing that don't
like how many flows we are creating at once. Not so much an issue
for datacenter or cloud-hosted deployments.
Updates tailscale/corp#22114
Signed-off-by: Jordan Whited <jordan@tailscale.com>
When the TS_DEBUG_NETSTACK_LOOPBACK_PORT environment variable is set,
netstack will loop back (dnat to addressFamilyLoopback:loopbackPort)
TCP & UDP flows originally destined to localServicesIP:loopbackPort.
localServicesIP is quad-100 or the IPv6 equivalent.
Updates tailscale/corp#22713
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This is the equivalent of quad-100, but for IPv6. This is technically
already contained in the Tailscale IPv6 ULA prefix, but that is only
installed when remote peers are visible via control with contained
addrs. The service addr should always be reachable.
Updates #1152
Signed-off-by: Jordan Whited <jordan@tailscale.com>
net/tstun.Wrapper.InjectInboundPacketBuffer is not GSO-aware, which can
break quad-100 TCP streams as a result. Linux is the only platform where
gVisor GSO was previously enabled.
Updates tailscale/corp#22511
Updates #13211
Signed-off-by: Jordan Whited <jordan@tailscale.com>
In df6014f1d7 we removed build tag
gating preventing importation, which tripped a NetworkExtension limit
test in corp. This was a reversal of
25f0a3fc8f which actually made the
situation worse, hence the simplification.
This commit goes back to the strategy in
25f0a3fc8f, and gets us back under the
limit in my local testing. Admittedly, we don't fully understand
the effects of importing or excluding importation of this package,
and have seen mixed results, but this commit allows us to move forward
again.
Updates tailscale/corp#22125
Signed-off-by: Jordan Whited <jordan@tailscale.com>
In 2f27319baf we disabled GRO due to a
data race around concurrent calls to tstun.Wrapper.Write(). This commit
refactors GRO to be thread-safe, and re-enables it on Linux.
This refactor now carries a GRO type across tstun and netstack APIs
with a lifetime that is scoped to a single tstun.Wrapper.Write() call.
In 25f0a3fc8f we used build tags to
prevent importation of gVisor's GRO package on iOS as at the time we
believed it was contributing to additional memory usage on that
platform. It wasn't, so this commit simplifies and removes those
build tags.
Updates tailscale/corp#22353
Updates tailscale/corp#22125
Updates #6816
Signed-off-by: Jordan Whited <jordan@tailscale.com>
In a93dc6cdb1 tryUpgradeToBatchingConn()
moved to build tag gated files, but the runtime.GOOS condition excluding
Android was removed unintentionally from batching_conn_linux.go. Add it
back.
Updates tailscale/corp#22348
Signed-off-by: Jordan Whited <jordan@tailscale.com>
getConns() is now responsible for returning both stable and unstable
conns. conn and measureFn are now passed together via connAndMeasureFn.
newConnAndMeasureFn() is responsible for constructing them.
TCP measurement timeouts are adjusted to more closely match netcheck.
Updates tailscale/corp#22114
Signed-off-by: Jordan Whited <jordan@tailscale.com>
'stun' has been removed from metric names and replaced with a protocol
label. This refactor is preparation work for HTTPS & ICMP support.
Updates tailscale/corp#22114
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit adds a batchingConn interface, and renames batchingUDPConn
to linuxBatchingConn. tryUpgradeToBatchingConn() may return a platform-
specific implementation of batchingConn. So far only a Linux
implementation of this interface exists, but this refactor is being
done in anticipation of a Windows implementation.
Updates tailscale/corp#21874
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit increases gVisor's TCP max send (4->6MiB) and receive
(4->8MiB) buffer sizes on all platforms except iOS. These values are
biased towards higher throughput on high bandwidth-delay product paths.
The iperf3 results below demonstrate the effect of this commit between
two Linux computers with i5-12400 CPUs. 100ms of RTT latency is
introduced via Linux's traffic control network emulator queue
discipline.
The first set of results are from commit f0230ce prior to TCP buffer
resizing.
gVisor write direction:
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 180 MBytes 151 Mbits/sec 0 sender
[ 5] 0.00-10.10 sec 179 MBytes 149 Mbits/sec receiver
gVisor read direction:
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.10 sec 337 MBytes 280 Mbits/sec 20 sender
[ 5] 0.00-10.00 sec 323 MBytes 271 Mbits/sec receiver
The second set of results are from this commit with increased TCP
buffer sizes.
gVisor write direction:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 297 MBytes 249 Mbits/sec 0 sender
[ 5] 0.00-10.10 sec 297 MBytes 247 Mbits/sec receiver
gVisor read direction:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.10 sec 501 MBytes 416 Mbits/sec 17 sender
[ 5] 0.00-10.00 sec 485 MBytes 407 Mbits/sec receiver
Updates #9707
Updates tailscale/corp#22119
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit implements TCP GRO for packets being written to gVisor on
Linux. Windows support will follow later. The wireguard-go dependency is
updated in order to make use of newly exported IP checksum functions.
gVisor is updated in order to make use of newly exported
stack.PacketBuffer GRO logic.
TCP throughput towards gVisor, i.e. TUN write direction, is dramatically
improved as a result of this commit. Benchmarks show substantial
improvement, sometimes as high as 2x. High bandwidth-delay product
paths remain receive window limited, bottlenecked by gVisor's default
TCP receive socket buffer size. This will be addressed in a follow-on
commit.
The iperf3 results below demonstrate the effect of this commit between
two Linux computers with i5-12400 CPUs. There is roughly ~13us of round
trip latency between them.
The first result is from commit 57856fc without TCP GRO.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.77 GBytes 4.10 Gbits/sec 20 sender
[ 5] 0.00-10.00 sec 4.77 GBytes 4.10 Gbits/sec receiver
The second result is from this commit with TCP GRO.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.6 GBytes 9.14 Gbits/sec 20 sender
[ 5] 0.00-10.00 sec 10.6 GBytes 9.14 Gbits/sec receiver
Updates #6816
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit implements TCP GSO for packets being read from gVisor on
Linux. Windows support will follow later. The wireguard-go dependency is
updated in order to make use of newly exported GSO logic from its tun
package.
A new gVisor stack.LinkEndpoint implementation has been established
(linkEndpoint) that is loosely modeled after its predecessor
(channel.Endpoint). This new implementation supports GSO of monster TCP
segments up to 64K in size, whereas channel.Endpoint only supports up to
32K. linkEndpoint will also be required for GRO, which will be
implemented in a follow-on commit.
TCP throughput from gVisor, i.e. TUN read direction, is dramatically
improved as a result of this commit. Benchmarks show substantial
improvement through a wide range of RTT and loss conditions, sometimes
as high as 5x.
The iperf3 results below demonstrate the effect of this commit between
two Linux computers with i5-12400 CPUs. There is roughly ~13us of round
trip latency between them.
The first result is from commit 57856fc without TCP GSO.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.51 GBytes 2.15 Gbits/sec 154 sender
[ 5] 0.00-10.00 sec 2.49 GBytes 2.14 Gbits/sec receiver
The second result is from this commit with TCP GSO.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec 6 sender
[ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec receiver
Updates #6816
Signed-off-by: Jordan Whited <jordan@tailscale.com>
stunstamp now sends data to Prometheus via remote write, and Prometheus
can serve the same data. Retaining and cleaning up old data in sqlite
leads to long probing pauses, and it's not worth investing more effort
to optimize the schema and/or concurrency model.
Updates tailscale/corp#20344
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Timeouts could already be identified as NaN values on
stunstamp_derp_stun_rtt_ns, but we can't use NaN effectively with
promql to visualize them. So, this commit adds a timeouts metric that
we can use with rate/delta/etc promql functions.
Updates tailscale/corp#20689
Signed-off-by: Jordan Whited <jordan@tailscale.com>
validate_udp_checksum was previously indeterminate (not zero) at
declaration, and IPv4 zero value UDP checksum packets were being passed
to the kernel.
Updates tailscale/corp#20689
Signed-off-by: Jordan Whited <jordan@tailscale.com>
This commit introduces a userspace program for managing an experimental
eBPF XDP STUN server program. derp/xdp contains the eBPF pseudo-C along
with a Go pkg for loading it and exporting its metrics.
cmd/xdpderper is a package main user of derp/xdp.
Updates tailscale/corp#20689
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Evaluation of remote write errors was using errors.Is() where it should
have been using errors.As().
Updates tailscale/corp#20344
Signed-off-by: Jordan Whited <jordan@tailscale.com>
stunstamp timestamping includes userspace and SO_TIMESTAMPING kernel
timestamping where available. Measurements are written locally to a
sqlite DB, exposed over an HTTP API, and written to prometheus
via remote-write protocol.
Updates tailscale/corp#20344
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Jordan WhitedandGitHub