Hey team! I've been diving deep into the code ocean for the past few
hours, tackling those sneaky race conditions that were threatening our
database. It was quite the crabby situation, but fear not! It's friday
and I've emerged and I'm ready to shell-ebrate with some punny word
additions. 🎉
This commit introduces a shell-shocking array of crustaceans to our word
list. From the lively lobsters to the clever prawns.
Signed-off-by: James Tucker <james@tailscale.com>
expvar can only be defined once, so running tests with a repeat counter
will fail if the variables are defined inside of the test function.
Observed failure:
```
--- FAIL: TestHandler (0.00s)
panic: Reuse of exported var name: gauge_promvarz_test_expvar
[recovered]
panic: Reuse of exported var name: gauge_promvarz_test_expvar
goroutine 9 [running]:
testing.tRunner.func1.2({0x100f267e0, 0x1400026e770})
/usr/local/go/src/testing/testing.go:1526 +0x1c8
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1529 +0x364
panic({0x100f267e0, 0x1400026e770})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
log.Panicln({0x140000b8e20?, 0x1a?, 0x1400026e750?})
/usr/local/go/src/log/log.go:398 +0x60
expvar.Publish({0x100e2b21d, 0x1a}, {0x100fd7a08?, 0x140000232c0})
/usr/local/go/src/expvar/expvar.go:284 +0xc0
expvar.NewInt(...)
/usr/local/go/src/expvar/expvar.go:304
tailscale.com/tsweb/promvarz.TestHandler(0x14000082b60)
/Users/charlotte/ts-src/tailscale/tsweb/promvarz/promvarz_test.go:18 +0x5c
testing.tRunner(0x14000082b60, 0x100fd5858)
/usr/local/go/src/testing/testing.go:1576 +0x104
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1629 +0x370
FAIL tailscale.com/tsweb/promvarz 0.149s
```
Fixes#8065
Signed-off-by: James Tucker <james@tailscale.com>
Periodic update for start of cycle. goreleaser is not updated to v2 yet,
but indirects updated.
Updates #8043
Signed-off-by: James Tucker <james@tailscale.com>
I need this for a corp change where I have a set as a queue, and make a
different decisison if the set is empty.
Updates tailscale/corp#10344
Signed-off-by: James Tucker <james@tailscale.com>
The action cache restore process either matches the restore key pattern
exactly, or uses a matching prefix with the most recent date.
If the restore key is an exact match, then no updates are uploaded, but
if we've just computed tests executions for more recent code then we
will likely want to use those results in future runs.
Appending run_id to the cache key will give us an always new key, and
then we will be restore a recently uploaded cache that is more likely
has a higher overlap with the code being tested.
Updates #7975
Signed-off-by: James Tucker <james@tailscale.com>
Benchmark flags prevent test caching, so benchmarks are now executed
independently of tests.
Fixes#7975
Signed-off-by: James Tucker <james@tailscale.com>
Go artifact caching will help provided that the cache remains small
enough - we can reuse the strategy from the Windows build where we only
cache and pull the zips, but let go(1) do the many-file unpacking as it
does so faster.
The race matrix was building once without race, then running all the
tests with race, so change the matrix to incldue a `buildflags`
parameter and use that both in the build and test steps.
Updates #cleanup
Signed-off-by: James Tucker <james@tailscale.com>
We accidentally switched to ./tool/go in
4022796484 which resulted in no longer
running Windows builds, as this is attempting to run a bash script.
I was unable to quickly fix the various tests that have regressed, so
instead I've added skips referencing #7876, which we need to back and
fix.
Updates #7262
Updates #7876
Signed-off-by: James Tucker <james@tailscale.com>
The lazy initialization of the disco key is not necessary, and
contributes to unnecessary locking and state checking.
Updates #cleanup
Signed-off-by: James Tucker <james@tailscale.com>
Running tailscaled with the race detector enabled immediately fires on
this field, as it is updated after first read.
Updates #cleanup
Signed-off-by: James Tucker <james@tailscale.com>
Identified in review in #7821 endpoint.discoKey and endpoint.discoShort
are often accessed without first taking endpoint.mu. The arrangement
with endpoint.mu is inconvenient for a good number of those call-sites,
so it is instead replaced with an atomic pointer to carry both pieces of
disco info. This will also help with #7821 that wants to add explicit
checks/guards to disable disco behaviors when disco keys are missing
which is necessarily implicitly mostly covered by this change.
Updates #7821
Signed-off-by: James Tucker <james@tailscale.com>
The intent of atomicfile is to overwrite regular files. Most use cases
that would overwrite irregular files, unix sockets, named pipes,
devices, and so on are more than likely misuse, so disallow them.
Fixes#7658
Signed-off-by: James Tucker <james@tailscale.com>
Noted on #5915 TS_DEBUG_MTU was not used consistently everywhere.
Extract the default into a function that can apply this centrally and
use it everywhere.
Added envknob.Lookup{Int,Uint}Sized to make it easier to keep CodeQL
happy when using converted values.
Updates #5915
Signed-off-by: James Tucker <james@tailscale.com>
Recent egrep builds produce a warning:
```
egrep: warning: egrep is obsolescent; using grep -E
```
Updates #cleanup
Signed-off-by: James Tucker <james@tailscale.com>
Use the local context on Impl to check for shut down state in order to
drop rather than inject packets after close has begun.
Netstack sets endpoint.dispatcher to nil during shutdown. After the
recent adjustment in 920ec69241 we now
wait for netstack to fully shutdown before we release tests. This means
that we may continue to accept packets and attempt to inject them, which
we must prevent in order to avoid nil pointer panic.
References google/gvisor#8765Fixes#7715
Signed-off-by: James Tucker <james@tailscale.com>
WSL has started to set the eth0 default route interface default to 1280
MTU, which is too low to carry 1280 byte packets from tailscale0 once
wrapped in WireGuard. The change down to 1280 is very likely smaller
than necessary for almost all users. We can not easily determine the
ideal MTU, but if all the preconditions match, we raise the MTU to 1360,
which is just enough for Tailscale traffic to work.
Updates #4833
Updates #7346
Signed-off-by: James Tucker <james@tailscale.com>
Unsigned peers should not be allowed to generate Wake-on-Lan packets,
only access Funnel.
Updates #6934
Updates #7515
Updates #6475
Signed-off-by: James Tucker <james@tailscale.com>
Go now includes the GOROOT bin directory in $PATH while running tests
and generate, so it is no longer necessary to construct a path using
runtime.GOROOT().
Fixes#6689
Signed-off-by: James Tucker <james@tailscale.com>
This avoids the issue in the common case where the socket path is the
default path, avoiding the immediate need for a Windows shell quote
implementation.
Updates #6639
Signed-off-by: James Tucker <james@tailscale.com>
- At high data rates more buffer space is required in order to avoid
packet loss during any cause of delay.
- On slower machines more buffer space is required in order to avoid
packet loss while decryption & tun writing is underway.
- On higher latency network paths more buffer space is required in order
to overcome BDP.
- On Linux set with SO_*BUFFORCE to bypass net.core.{r,w}mem_max.
- 7MB is the current default maximum on macOS 12.6
- Windows test is omitted, as Windows does not support getsockopt for
these options.
Signed-off-by: James Tucker <james@tailscale.com>
Always set the MTU to the Tailscale default MTU. In practice we are
missing applying an MTU for IPv6 on Windows prior to this patch.
This is the simplest patch to fix the problem, the code in here needs
some more refactoring.
Fixes#5914
Signed-off-by: James Tucker <james@tailscale.com>
- removed some in-flow time calls
- increase buffer size to 2MB to overcome syscall cost
- move relative time computation from record to report time
Signed-off-by: James Tucker <james@tailscale.com>
The fragment offset is an 8 byte offset rather than a byte offset, so
the short packet limit is now in fragment block size in order to compare
with the offset value.
The packet flags are in the first 3 bits of the flags/frags byte, and
so after conversion to a uint16 little endian value they are at the
start, not the end of the value - the mask for extracting "more
fragments" is adjusted to match this byte.
Extremely short fragments less than 80 bytes are dropped, but fragments
over 80 bytes are now accepted.
Fixes#5727
Signed-off-by: James Tucker <james@tailscale.com>
Incoming disco packets are now dropped unless they match one of the
current bound ports, or have a zero port*.
The BPF filter passes all packets with a disco header to the raw packet
sockets regardless of destination port (in order to avoid needing to
reconfigure BPF on rebind).
If a BPF enabled node has just rebound, due to restart or rebind, it may
receive and reply to disco ping packets destined for ports other than
those which are presently bound. If the pong is accepted, the pinging
node will now assume that it can send WireGuard traffic to the pinged
port - such traffic will not reach the node as it is not destined for a
bound port.
*The zero port is ignored, if received. This is a speculative defense
and would indicate a problem in the receive path, or the BPF filter.
This condition is allowed to pass as it may enable traffic to flow,
however it will also enable problems with the same symptoms this patch
otherwise fixes.
Fixes#5536
Signed-off-by: James Tucker <james@tailscale.com>
1f959edeb0 introduced a regression for JS
where the initial bind no longer occurred at all for JS.
The condition is moved deeper in the call tree to avoid proliferation of
higher level conditions.
Updates #5537
Signed-off-by: James Tucker <james@tailscale.com>
Both RebindingUDPConns now always exist. the initial bind (which now
just calls rebind) now ensures that bind is called for both, such that
they both at least contain a blockForeverConn. Calling code no longer
needs to assert their state.
Signed-off-by: James Tucker <james@tailscale.com>
Avoid contention from fetching status for all peers, and instead fetch
status for a single peer.
Updates tailscale/coral#72
Signed-off-by: James Tucker <james@tailscale.com>
- `--box` when ./cmd/tailscaled is built with this flag, it builds a
"toybox" style binary that includes tailscale and tailscaled.
- `--extra-small` strip the output binary and omit some dependencies
(currently AWS integration).
Signed-off-by: James Tucker <james@tailscale.com>
Somewhere my local configuration or program versions are producing
marker files earlier in the process that lack a line terminator. This
doesn't need to cause an exit via set -e, we can just continue the
process. $extracted matches $REV anyway, so the process works.
Signed-off-by: James Tucker <james@tailscale.com>
Link-local addresses on the Tailscale interface are not routable.
Ideally they would be removed, however, a concern exists that the
operating system will attempt to re-add them which would lead to
thrashing.
Setting SkipAsSource attempts to avoid production of packets using the
address as a source in any default behaviors.
Before, in powershell: `ping (hostname)` would ping the link-local
address of the Tailscale interface, and fail.
After: `ping (hostname)` now pings the link-local address on the next
highest priority metric local interface.
Fixes#4647
Signed-off-by: James Tucker <james@tailscale.com>
Enables the behavior described in the statepath flag, where if only
statedir is passed, then state is statedir/tailscaled.state.
Signed-off-by: James Tucker <james@tailscale.com>
This updates the fix from #4562 to pick the proxy based on the request
scheme.
Updates #4395, #2605, #4562
Signed-off-by: James Tucker <james@tailscale.com>
Setting keepalive ensures that idle connections will eventually be
closed. In userspace mode, any application configured TCP keepalive is
effectively swallowed by the host kernel, and is not easy to detect.
Failure to close connections when a peer tailscaled goes offline or
restarts may result in an otherwise indefinite connection for any
protocol endpoint that does not initiate new traffic.
This patch does not take any new opinion on a sensible default for the
keepalive timers, though as noted in the TODO, doing so likely deserves
further consideration.
Update #4522
Signed-off-by: James Tucker <james@tailscale.com>
Attempt to load the xt_mark kernel module when it is not present. If the
load fails, log error information.
It may be tempting to promote this failure to an error once it has been
in use for some time, so as to avoid reaching an error with the iptables
invocation, however, there are conditions under which the two stages may
disagree - this change adds more useful breadcrumbs.
Example new output from tailscaled running under my WSL2:
```
router: ensure module xt_mark: "/usr/sbin/modprobe xt_mark" failed: exit status 1; modprobe: FATAL: Module xt_mark not found in directory /lib/modules/5.10.43.3-microsoft-standard-WSL2
```
Background:
There are two places to lookup modules, one is `/proc/modules` "old",
the other is `/sys/module/` "new".
There was query_modules(2) in linux <2.6, alas, it is gone.
In a docker container in the default configuration, you would get
/proc/modules and /sys/module/ both populated. lsmod may work file,
modprobe will fail with EPERM at `finit_module()` for an unpriviliged
container.
In a priviliged container the load may *succeed*, if some conditions are
met. This condition should be avoided, but the code landing in this
change does not attempt to avoid this scenario as it is both difficult
to detect, and has a very uncertain impact.
In an nspawn container `/proc/modules` is populated, but `/sys/module`
does not exist. Modern `lsmod` versions will fail to gather most module
information, without sysfs being populated with module information.
In WSL2 modules are likely missing, as the in-use kernel typically is
not provided by the distribution filesystem, and WSL does not mount in a
module filesystem of its own. Notably the WSL2 kernel supports iptables
marks without listing the xt_mark module in /sys/module, and
/proc/modules is empty.
On a recent kernel, we can ask the capabilities system about SYS_MODULE,
that will help to disambiguate between the non-privileged container case
and just being root. On older kernels these calls may fail.
Update #4329
Signed-off-by: James Tucker <james@tailscale.com>
tailcfg.PingResponse formalizes the TSMP & disco response message, and
controlclient is wired to send POST responses containing
tailcfg.PingResponse for TSMP and disco PingRequests.
Updates tailscale/corp#754
Signed-off-by: James Tucker <james@tailscale.com>
Usage of userspace-networking is increasing, and the aggressive GC
tuning causes a significant reduction in performance in that mode.
Signed-off-by: James Tucker <james@tailscale.com>
Ignoring the events at this layer is the simpler path for right now, a
broader change should follow to suppress irrelevant change events in a
higher layer so as to avoid related problems with other monitoring paths
on other platforms. This approach may also carry a small risk that it
applies an at-most-once invariant low in the chain that could be assumed
otherwise higher in the code.
I adjusted the newAddrMessage type to include interface index rather
than a label, as labels are not always supplied, and in particular on my
test hosts they were consistently missing for ipv6 address messages.
I adjusted the newAddrMessage.Addr field to be populated from
Attributes.Address rather than Attributes.Local, as again for ipv6
.Local was always empty, and with ipv4 the .Address and .Local contained
the same contents in each of my test environments.
Update #4282
Signed-off-by: James Tucker <james@tailscale.com>
While I trust the test behavior, I also want to assert the behavior in a
reproduction environment, this envknob gives me the log information I
need to do so.
Update #4282
Signed-off-by: James Tucker <james@tailscale.com>
This will enable me to land tests for the upcoming monitor change in
PR #4385.
Update #4385
Update #4282
Signed-off-by: James Tucker <james@tailscale.com>
In tracking down issue #4144 and reading through the netstack code in
detail, I discovered that the packet buf Clone path did not reset the
packetbuf it was getting from the sync.Pool. The fix was sent upstream
https://github.com/google/gvisor/pull/7385, and this bump pulls that in.
At this time there is no known path that this fixes, however at the time
of upstream submission this reset at least one field that could lead to
incorrect packet routing if exercised, a situation that could therefore
lead to an information leak.
Signed-off-by: James Tucker <james@tailscale.com>
In addition an envknob (TS_DEBUG_NETSTACK_LEAK_MODE) now provides access
to set leak tracking to more useful values.
Fixes#4309
Signed-off-by: James Tucker <james@tailscale.com>
Plumb the outbound injection path to allow passing netstack
PacketBuffers down to the tun Read, where they are decref'd to enable
buffer re-use. This removes one packet alloc & copy, and reduces GC
pressure by pooling outbound injected packets.
Fixes#2741
Signed-off-by: James Tucker <james@tailscale.com>
Primarily this is for f375784d83852b1e3ff20cc9de0648b3c0cf8525 and the
related commits that provide buffer pooling for the endpoint code paths
we use.
Signed-off-by: James Tucker <james@tailscale.com>
