natc itself can't immediately fix the problem, but it can more correctly
error that return bad addresses.
Updates tailscale/corp#26968
Signed-off-by: James Tucker <james@tailscale.com>
In order to improve latency tracking, we will use an exponentially
weighted moving average that will smooth change over time and suppress
large outlier values.
Updates tailscale/corp#26649
Signed-off-by: James Tucker <james@tailscale.com>
Bart has had some substantial improvements in internal representation,
update functions, and other optimizations to reduce memory usage and
improve runtime performance.
Updates tailscale/corp#26353
Signed-off-by: James Tucker <james@tailscale.com>
Cubic performs better than Reno in higher BDP scenarios, and enables the
use of the hystart++ implementation contributed by Coder. This improves
throughput on higher BDP links with a much faster ramp.
gVisor is bumped as well for some fixes related to send queue processing
and RTT tracking.
Updates #9707
Updates #10408
Updates #12393
Updates tailscale/corp#24483
Updates tailscale/corp#25169
Signed-off-by: James Tucker <james@tailscale.com>
Originally identified by Coder and documented in their blog post, this
implementation differs slightly as our link endpoint was introduced for
a different purpose, but the behavior is the same: apply backpressure
rather than dropping packets. This reduces the negative impact of large
packet count bursts substantially. An alternative would be to swell the
size of the channel buffer substantially, however that's largely just
moving where buffering occurs and may lead to reduced signalling back to
lower layer or upstream congestion controls.
Updates #9707
Updates #10408
Updates #12393
Updates tailscale/corp#24483
Updates tailscale/corp#25169
Signed-off-by: James Tucker <james@tailscale.com>
The gVisor RACK implementation appears to perfom badly, particularly in
scenarios with higher BDP. This may have gone poorly noticed as a result
of it being gated on SACK, which is not enabled by default in upstream
gVisor, but itself has a higher positive impact on performance. Both the
RACK and DACK implementations (which are now one) have overlapping
non-completion of tasks in their work streams on the public tracker.
Updates #9707
Signed-off-by: James Tucker <james@tailscale.com>
We once again have a report of a panic from ParseRIB. This panic guard
should probably remain permanent.
Updates #14201
This reverts commit de9d4b2f88.
Signed-off-by: James Tucker <james@tailscale.com>
Observed on some airlines (British Airways, WestJet), Squid is
configured to cache and transform these results, which is disruptive.
The server and client should both actively request that this is not done
by setting Cache-Control headers.
Send a timestamp parameter to further work against caches that do not
respect the cache-control headers.
Updates #14856
Signed-off-by: James Tucker <james@tailscale.com>
Some natc instances have been observed with excessive memory growth,
dominant in gvisor buffers. It is likely that the connection buffers are
sticking around for too long due to the default long segment time, and
uptuned buffer size applied by default in wgengine/netstack. Apply
configurations in natc specifically which are a better match for the
natc use case, most notably a 5s maximum segment lifetime.
Updates tailscale/corp#25169
Signed-off-by: James Tucker <james@tailscale.com>
We have observed some clients with extremely large lists of IPv6
endpoints, in some cases from subnets where the machine also has the
zero address for a whole /48 with then arbitrary addresses additionally
assigned within that /48. It is in general unnecessary for reachability
to report all of these addresses, typically only one will be necessary
for reachability. We report two, to cover some other common cases such
as some styles of IPv6 private address rotations.
Updates tailscale/corp#25850
Signed-off-by: James Tucker <james@tailscale.com>
Remove the platform specificity, it is unnecessary complexity.
Deduplicate repeated code as a result of reduced complexity.
Split out error identification code.
Update call-sites and tests.
Updates #14551
Updates tailscale/corp#25648
Signed-off-by: James Tucker <james@tailscale.com>
Observed in the wild some macOS machines gain broken sockets coming out
of sleep (we observe "time jumped", followed by EPIPE on sendto). The
cause of this in the platform is unclear, but the fix is clear: always
rebind if the socket is broken. This can also be created artificially on
Linux via `ss -K`, and other conditions or software on a system could
also lead to the same outcomes.
Updates tailscale/corp#25648
Signed-off-by: James Tucker <james@tailscale.com>
metrics.LabelMap grows slightly more heavy, needing a lock to ensure
proper ordering for newly initialized ShardedInt values. An Add method
enables callers to use .Add for both expvar.Int and syncs.ShardedInt
values, but retains the original behavior of defaulting to initializing
expvar.Int values.
Updates tailscale/corp#25450
Co-Authored-By: Andrew Dunham <andrew@du.nham.ca>
Signed-off-by: James Tucker <james@tailscale.com>
ShardedInt provides an int type expvar.Var that supports more efficient
writes at high frequencies (one order of magnigude on an M1 Max, much
more on NUMA systems).
There are two implementations of ShardValue, one that abuses sync.Pool
that will work on current public Go versions, and one that takes a
dependency on a runtime.TailscaleP function exposed in Tailscale's Go
fork. The sync.Pool variant has about 10x the throughput of a single
atomic integer on an M1 Max, and the runtime.TailscaleP variant is about
10x faster than the sync.Pool variant.
Neither variant have perfect distribution, or perfectly always avoid
cross-CPU sharing, as there is no locking or affinity to ensure that the
time of yield is on the same core as the time of core biasing, but in
the average case the distributions are enough to provide substantially
better performance.
See golang/go#18802 for a related upstream proposal.
Updates tailscale/go#109
Updates tailscale/corp#25450
Signed-off-by: James Tucker <james@tailscale.com>
The go-httpstat package has a data race when used with connections that
are performing happy-eyeballs connection setups as we are in the DERP
client. There is a long-stale PR upstream to address this, however
revisiting the purpose of this code suggests we don't really need
httpstat here.
The code populates a latency table that may be used to compare to STUN
latency, which is a lightweight RTT check. Switching out the reported
timing here to simply the request HTTP request RTT avoids the
problematic package.
Fixestailscale/corp#25095
Signed-off-by: James Tucker <james@tailscale.com>
This provides an interface for a user to force a preferred DERP outcome
for all future netchecks that will take precedence unless the forced
region is unreachable.
The option does not persist and will be lost when the daemon restarts.
Updates tailscale/corp#18997
Updates tailscale/corp#24755
Signed-off-by: James Tucker <james@tailscale.com>
I was hoping we'd catch an example input quickly, but the reporter had
rebooted their machine and it is no longer exhibiting the behavior. As
such this code may be sticking around quite a bit longer and we might
encounter other errors, so include the panic in the log entry.
Updates #14201
Updates #14202
Updates golang/go#70528
Signed-off-by: James Tucker <james@tailscale.com>
In an environment with unstable latency, such as upstream bufferbloat,
there are cases where a full netcheck could drop the prior preferred
DERP (likely home DERP) from future netcheck probe plans. This will then
likely result in a home DERP having a missing sample on the next
incremental netcheck, ultimately resulting in a home DERP move.
This change does not fix our overall response to highly unstable
latency, but it is an incremental improvement to prevent single spurious
samples during a full netcheck from alone triggering a flapping
condition, as now the prior changes to include historical latency will
still provide the desired resistance, and the home DERP should not move
unless latency is consistently worse over a 5 minute period.
Note that there is a nomenclature and semantics issue remaining in the
difference between a report preferred DERP and a home DERP. A report
preferred DERP is aspirational, it is what will be picked as a home DERP
if a home DERP connection needs to be established. A nodes home DERP may
be different than a recent preferred DERP, in which case a lot of
netcheck logic is fallible. In future enhancements much of the DERP move
logic should move to consider the home DERP, rather than recent report
preferred DERP.
Updates #8603
Updates #13969
Signed-off-by: James Tucker <james@tailscale.com>
trimpath can be inconvenient for IDEs and LSPs that do not always
correctly handle module relative paths, and can also contribute to
caching bugs taking effect. We rarely have a real need for trimpath of
test produced binaries, so avoiding it should be a net win.
Updates #2988
Signed-off-by: James Tucker <james@tailscale.com>
A simple implementation of latency and loss simulation, applied to
writes to the ethernet interface of the NIC. The latency implementation
could be optimized substantially later if necessary.
Updates #13355
Signed-off-by: James Tucker <james@tailscale.com>
Like Linux, macOS will reply to sendto(2) with EPERM if the firewall is
currently blocking writes, though this behavior is like Linux
undocumented. This is often caused by a faulting network extension or
content filter from EDR software.
Updates #11710
Updates #12891
Updates #13511
Signed-off-by: James Tucker <james@tailscale.com>
mdnsResponder at least as of macOS Sequoia does not find NXDOMAIN
responses to these dns-sd PTR queries acceptable unless they include the
question section in the response. This was found debugging #13511, once
we turned on additional diagnostic reporting from mdnsResponder we
witnessed:
```
Received unacceptable 12-byte response from 100.100.100.100 over UDP via utun6/27 -- id: 0x7F41 (32577), flags: 0x8183 (R/Query, RD, RA, NXDomain), counts: 0/0/0/0,
```
If the response includes a question section, the resposnes are
acceptable, e.g.:
```
Received acceptable 59-byte response from 8.8.8.8 over UDP via en0/17 -- id: 0x2E55 (11861), flags: 0x8183 (R/Query, RD, RA, NXDomain), counts: 1/0/0/0,
```
This may be contributing to an issue under diagnosis in #13511 wherein
some combination of conditions results in mdnsResponder no longer
answering DNS queries correctly to applications on the system for
extended periods of time (multiple minutes), while dig against quad-100
provides correct responses for those same domains. If additional debug
logging is enabled in mdnsResponder we see it reporting:
```
Penalizing server 100.100.100.100 for 60 seconds
```
It is also possible that the reason that macOS & iOS never "stopped
spamming" these queries is that they have never been replied to with
acceptable responses. It is not clear if this special case handling of
dns-sd PTR queries was ever beneficial, and given this evidence may have
always been harmful. If we subsequently observe that the queries settle
down now that they have acceptable responses, we should remove these
special cases - making upstream queries very occasionally isn't a lot of
battery, so we should be better off having to maintain less special
cases and avoid bugs of this class.
Updates #2442
Updates #3025
Updates #3363
Updates #3594
Updates #13511
Signed-off-by: James Tucker <james@tailscale.com>
On a major link change the LAN routes may change, so on linkChange where
ChangeDelta.Major, we need to call authReconfig to ensure that new
routes are observed and applied.
Updates tailscale/corp#22574
Signed-off-by: James Tucker <james@tailscale.com>
BPF links require that the owning FD remains open, this FD is embedded
into the RawLink returned by the attach function and must live for the
duration of the server.
Updates ENG-4274
Signed-off-by: James Tucker <james@tailscale.com>
A few other minor language updates.
Updates tailscale/corp#20844
Change-Id: Idba85941baa0e2714688cc8a4ec3e242e7d1a362
Signed-off-by: James Tucker <james@tailscale.com>
We do not intend to use this value for feature support communication in
the future, and have applied changes elsewhere that now fix the expected
value.
Updates tailscale/corp#19391
Updates tailscale/corp#20398
Signed-off-by: James Tucker <james@tailscale.com>
We don't build a lot of tools with CGO, but we do build some, and it's
extremely valuable for production services in particular to have symbols
included - for perf and so on.
I tested various other builds that could be affected negatively, in
particular macOS/iOS, but those use split-dwarf already as part of their
build path, and Android which does not currently use gocross.
One binary which is normally 120mb only grew to 123mb, so the trade-off
is definitely worthwhile in context.
Updates tailscale/corp#20296
Signed-off-by: James Tucker <james@tailscale.com>
Palo Alto reported interpreting hairpin probes as LAND attacks, and the
firewalls may be responding to this by shutting down otherwise in use NAT sessions
prematurely. We don't currently make use of the outcome of the hairpin
probes, and they contribute to other user confusion with e.g. the
AirPort Extreme hairpin session workaround. We decided in response to
remove the whole probe feature as a result.
Updates #188
Updates tailscale/corp#19106
Updates tailscale/corp#19116
Signed-off-by: James Tucker <james@tailscale.com>
Palo Alto firewalls have a typically hard NAT, but also have a mode
called Persistent DIPP that is supposed to provide consistent port
mapping suitable for STUN resolution of public ports. Persistent DIPP
works initially on most Palo Alto firewalls, but some models/software
versions have a bug which this works around.
The bug symptom presents as follows:
- STUN sessions resolve a consistent public IP:port to start with
- Much later netchecks report the same IP:Port for a subset of
sessions, most often the users active DERP, and/or the port related
to sustained traffic.
- The broader set of DERPs in a full netcheck will now consistently
observe a new IP:Port.
- After this point of observation, new inbound connections will only
succeed to the new IP:Port observed, and existing/old sessions will
only work to the old binding.
In this patch we now advertise the lowest latency global endpoint
discovered as we always have, but in addition any global endpoints that
are observed more than once in a single netcheck report. This should
provide viable endpoints for potential connection establishment across
a NAT with this behavior.
Updates tailscale/corp#19106
Signed-off-by: James Tucker <james@tailscale.com>
Tracking down the side effect can otherwise be a pain, for example on
Darwin an empty GOOS resulted in CGO being implicitly disabled. The user
intended for `export GOOS=` to act like unset, and while this is a
misunderstanding, the main toolchain would treat it this way.
Fixestailscale/corp#20059
Signed-off-by: James Tucker <james@tailscale.com>
Just because we don't have known endpoints for a peer does not mean that
the peer should become unreachable. If we know the peers key, it should
be able to call us, then we can talk back via whatever path it called us
on. First step - don't drop the packet in this context.
Updates tailscale/corp#19106
Signed-off-by: James Tucker <james@tailscale.com>
Request ID generation appears prominently in some services cumulative
allocation rate, and while this does not eradicate this issue (the API
still makes UUID objects), it does improve the overhead of this API and
reduce the amount of garbage that it produces.
Updates tailscale/corp#18266
Updates tailscale/corp#19054
Signed-off-by: James Tucker <james@tailscale.com>
This still generates github.com/google/uuid UUID objects, but does so
using a ChaCha8 CSPRNG from the stdlib rand/v2 package. The public API
is backed by a sync.Pool to provide good performance in highly
concurrent operation.
Under high load the read API produces a lot of extra garbage and
overhead by way of temporaries and syscalls. This implementation reduces
both to minimal levels, and avoids any long held global lock by
utilizing sync.Pool.
Updates tailscale/corp#18266
Updates tailscale/corp#19054
Signed-off-by: James Tucker <james@tailscale.com>
This removes a potentially increased boot delay for certain boot
topologies where they block on ExecStartPre that may have socket
activation dependencies on other system services (such as
systemd-resolved and NetworkManager).
Also rename cleanup to clean up in affected/immediately nearby places
per code review commentary.
Fixes#11599
Signed-off-by: James Tucker <james@tailscale.com>
The netcheck package and the magicksock package coordinate via the
health package, but both sides have time based heuristics through
indirect dependencies. These were misaligned, so the implemented
heuristic aimed at reducing DERP moves while there is active traffic
were non-operational about 3/5ths of the time.
It is problematic to setup a good test for this integration presently,
so instead I added comment breadcrumbs along with the initial fix.
Updates #8603
Signed-off-by: James Tucker <james@tailscale.com>
I originally came to update this to match the documented behavior, but
the code is deliberately avoiding this behavior currently, making it
hard to decide how to update this. For now just align the documentation
to the behavior.
Updates #cleanup
Signed-off-by: James Tucker <james@tailscale.com>
We now allow some more ICMP errors to flow, specifically:
- ICMP parameter problem in both IPv4 and IPv6 (corrupt headers)
- ICMP Packet Too Big (for IPv6 PMTU)
Updates #311
Updates #8102
Updates #11002
Signed-off-by: James Tucker <james@tailscale.com>
We're tracking down a new instance of memory usage, and excessive memory usage
from sockstats is definitely not going to help with debugging, so disable it by
default on mobile.
Updates tailscale/corp#18514
Signed-off-by: James Tucker <james@tailscale.com>
The initial control client request can get stuck in the event that a
connection is established but then lost part way through, without any
ICMP or RST. Ensure that the control client will be restarted by timing
out that initial request as well.
Fixes#11542
Signed-off-by: James Tucker <james@tailscale.com>
In control there are conditions where the leaf functions are not being
optimized away (i.e. At is not inlined), resulting in undesirable time
spent copying during SliceContains. This optimization is likely
irrelevant to simpler code or smaller structures.
Updates #optimization
Signed-off-by: James Tucker <james@tailscale.com>
There are container environments such as GitHub codespaces that have
partial IPv6 support - routing support is enabled at the kernel level,
but lacking IPv6 filter support in the iptables module.
In the specific example of the codespaces environment, this also has
pre-existing legacy iptables rules in the IPv4 tables, as such the
nascent firewall mode detection will always pick iptables.
We would previously fault trying to install rules to the filter table,
this catches that condition earlier, and disables IPv6 support under
these conditions.
Updates #5621
Updates #11344
Updates #11354
Signed-off-by: James Tucker <james@tailscale.com>
This server recently had a common ansible applied, which added a
periodic /tmp cleaner, as is needed on other CI machines to deal with
test tempfile leakage. The setting of $HOME to /tmp means that the go
toolchain in there was regularly getting pruned by the tmp cleaner, but
often incompletely, because it was also in use.
Move HOME to a runner owned directory.
Updates #11248
Signed-off-by: James Tucker <james@tailscale.com>
If a client socket is remotely lost but the client is not sent an RST in
response to the next request, the socket might sit in RTO for extended
lengths of time, resulting in "no internet" for users. Instead, timeout
after 10s, which will close the underlying socket, recovering from the
situation more promptly.
Updates #10967
Signed-off-by: James Tucker <james@tailscale.com>
I missed a case in the earlier patch, and so we're still sending 15s TCP
keepalive for TLS connections, now adjusted there too.
Updates tailscale/corp#17587
Updates #3363
Signed-off-by: James Tucker <james@tailscale.com>
This appears to be one of the contributors to this CI target regularly
entering a bad state with a partially written toolchain.
Updates #self
Signed-off-by: James Tucker <james@tailscale.com>
We don't need a log line every time defaultRoute is read in the good
case, and we now only log default interface updates that are actually
changes.
Updates #3363
Signed-off-by: James Tucker <james@tailscale.com>
An increasing number of users have very large subnet route
configurations, which can produce very large amounts of log data when
WireGuard is reconfigured. The logs don't contain the actual routes, so
they're largely useless for diagnostics, so we'll just suppress them.
Fixestailscale/corp#17532
Signed-off-by: James Tucker <james@tailscale.com>
The derper sends an in-protocol keepalive every 60-65s, so frequent TCP
keepalives are unnecessary. In this tuning TCP keepalives should never
occur for a DERP client connection, as they will send an L7 keepalive
often enough to always reset the TCP keepalive timer. If however a
connection does not receive an ACK promptly it will now be shutdown,
which happens sooner than it would with a normal TCP keepalive tuning.
This re-tuning reduces the frequency of network traffic from derp to
client, reducing battery cost.
Updates tailscale/corp#17587
Updates #3363
Signed-off-by: James Tucker <james@tailscale.com>
Setting a user timeout will be a more practical tuning knob for a number
of endpoints, this provides a way to set it.
Updates tailscale/corp#17587
Signed-off-by: James Tucker <james@tailscale.com>
From a packet trace we have seen log connections being closed
prematurely by the client, resulting in unnecessary extra TLS setup
traffic.
Updates #3363
Updates tailscale/corp#9230
Updates tailscale/corp#8564
Signed-off-by: James Tucker <james@tailscale.com>
The new math/rand/v2 package includes an m-local global random number
generator that can not be reseeded by the user, which is suitable for
most uses without the RNG pools we have in a number of areas of the code
base.
The new API still does not have an allocation-free way of performing a
seeded operations, due to the long term compiler bug around interface
parameter escapes, and the Source interface.
This change introduces the two APIs that math/rand/v2 can not yet
replace efficiently: seeded Perm() and Shuffle() operations. This
implementation chooses to use the PCG random source from math/rand/v2,
as with sufficient compiler optimization, this source should boil down
to only two on-stack registers for random state under ideal conditions.
Updates #17243
Signed-off-by: James Tucker <james@tailscale.com>
Advertise DNS discovered addresses as a single preference update rather
than one at a time.
Sort the list of observed addresses and use binary search to consult the
list.
Updates tailscale/corp#16636
Signed-off-by: James Tucker <james@tailscale.com>
If an app connector is also configured as an exit node, it should still
advertise discovered routes that are not covered by advertised routes,
excluding the exit node routes.
Updates tailscale/corp#16928
Signed-off-by: James Tucker <james@tailscale.com>
If any domain along a CNAME chain matches any of the routed domains, add
routes for the discovered domains.
Fixestailscale/corp#16928
Signed-off-by: James Tucker <james@tailscale.com>
The API on the DNS record parser is slightly subtle and requires
explicit handling of unhandled records. Failure to advance previously
resulted in an infinite loop in the pretty responder for any reply that
contains a record other than A/AAAA/TXT.
Updates tailscale/corp#16928
Signed-off-by: James Tucker <james@tailscale.com>
expvarx.SafeFunc wraps an expvar.Func with a time limit. On reaching the
time limit, calls to Value return nil, and no new concurrent calls to
the underlying expvar.Func will be started until the call completes.
Updates tailscale/corp#16999
Signed-off-by: James Tucker <james@tailscale.com>
If control advised the connector to advertise a route that had already
been discovered by DNS it would be incorrectly removed. Now those routes
are preserved.
Updates tailscale/corp#16833
Signed-off-by: James Tucker <james@tailscale.com>
Plan9 CI is disabled. 3p dependencies do not build for the target.
Contributor enthusiasm appears to have ceased again, and no usage has
been made.
Skipped gvisor, nfpm, and k8s.
Updates #5794
Updates #8043
Signed-off-by: James Tucker <james@tailscale.com>
If there are routes changes as a side effect of an app connector
configuration update, the connector configuration may want to reenter a
lock, so must be started asynchronously.
Updates tailscale/corp#16833
Signed-off-by: James Tucker <james@tailscale.com>
This is a useful primitive for asynchronous execution of ordered work I
want to use in another change.
Updates tailscale/corp#16833
Signed-off-by: James Tucker <james@tailscale.com>
Control can now send down a set of routes along with the domains, and
the routes will be advertised, with any newly overlapped routes being
removed to reduce the size of the routing table.
Fixestailscale/corp#16833
Signed-off-by: James Tucker <james@tailscale.com>
See the field alignment lints for more information.
Reductions are 64->24 and 64->32 respectively.
Updates #self
Signed-off-by: James Tucker <james@tailscale.com>
Observed on one busy derp node, there were 600 goroutines blocked
writing to this channel, which represents not only more blocked routines
than we need, but also excess wake-ups downstream as the latent
goroutines writes represent no new work.
Updates #self
Signed-off-by: James Tucker <james@tailscale.com>
Add a standalone server for STUN that can be hosted independently of the
derper, and factor that back into the derper.
Fixes#8434Closes#8435Closes#10745
Signed-off-by: James Tucker <james@tailscale.com>
Individual route advertisements that are covered by existing routes are
no longer advertised. If an upstream returns 0.0.0.0, 127.x, and other
common unwanted addresses those are also rejected.
Updates #16425
Signed-off-by: James Tucker <james@tailscale.com>
connector-gen can initially generate connector ACL snippets and
advertise-routes flags for Github and AWS based on their public IP /
domain data.
Updates ENG-2425
Signed-off-by: James Tucker <james@tailscale.com>
This will enable the runner to be replaced as a configuration side
effect in a later change.
Updates tailscale/corp#14029
Signed-off-by: James Tucker <james@tailscale.com>
The non-referential copy destination doesn't extend the map contents,
but also the read of a non-key is returning a zero value not bound to
the map contents in any way.
Updates tailscale/corp#15657
Signed-off-by: James Tucker <james@tailscale.com>
Some conditional paths may otherwise skip the hostinfo update, so kick
it off asynchronously as other code paths do.
Updates tailscale/corp#15437
Signed-off-by: James Tucker <james@tailscale.com>
The app connector matches a configuration of "*.example.com" to mean any
sub-domain of example.com.
Updates #15437
Signed-off-by: James Tucker <james@tailscale.com>
There was pre-existing additional usage for Exit Node DNS resolution via
PeerAPI, as well as new usage just introduced for App Connectors.
Fixes ENG-2324
Updates #cleanup
Signed-off-by: James Tucker <james@tailscale.com>
The design changed during integration and testing, resulting in the
earlier implementation growing in the appc package to be intended now
only for the sniproxy implementation. That code is moved to it's final
location, and the current App Connector code is now renamed.
Updates tailscale/corp#15437
Signed-off-by: James Tucker <james@tailscale.com>
