Expand TestRedactNetmapPrivateKeys to cover all sub-structs of
NetworkMap and confirm that a) all fields are annotated as private or
public, and b) all private fields are getting redacted.
Updates tailscale/corp#32095
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
For debugging purposes, add a new C2N endpoint returning the current
netmap. Optionally, coordination server can send a new "candidate" map
response, which the client will generate a separate netmap for.
Coordination server can later compare two netmaps, detecting unexpected
changes to the client state.
Updates tailscale/corp#32095
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of a single hard-coded C2N handler, add support for calling
arbitrary C2N endpoints via a node roundtripper.
Updates tailscale/corp#32095
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
It was a bit confusing that provided history did not include the
current probe results.
Updates tailscale/corp#20583
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
We already present a health warning about this, but it is easy to miss
on a server when blackholing traffic makes it unreachable.
In addition to a health warning, present a risk message when exit node
is enabled.
Example:
```
$ tailscale up --exit-node=lizard
The following issues on your machine will likely make usage of exit nodes impossible:
- interface "ens4" has strict reverse-path filtering enabled
- interface "tailscale0" has strict reverse-path filtering enabled
Please set rp_filter=2 instead of rp_filter=1; see https://github.com/tailscale/tailscale/issues/3310
To skip this warning, use --accept-risk=linux-strict-rp-filter
$
```
Updates #3310
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This makes sure that the log target override is respected even if a
custom HTTP client is passed to logpolicy.
Updates tailscale/maple#29
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This adds support for using ACL Grants to configure a role for the
auto-provisioned user.
Fixestailscale/corp#14567
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Some probes might need to run for longer than their scheduling interval,
so this change relaxes the 1-at-a-time restriction, allowing us to
configure probe concurrency and timeout separately. The default values
remain the same (concurrency of 1; timeout of 80% of interval).
Updates tailscale/corp#25479
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Initial support for SrcCaps was added in 5ec01bf but it was not actually
working without this.
Updates #12542
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Perhaps I was too opimistic in #13323 thinking we won't need logs for
this. Let's log a summary of the response without logging specific
identifiers.
Updates tailscale/corp#24437
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This required sharing the dropped packet metric between two packages
(tstun and magicsock), so I've moved its definition to util/usermetric.
Updates tailscale/corp#22075
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
The user-facing metrics are intended to track data transmitted at
the overlay network level.
Updates tailscale/corp#22075
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
- `tailscale metrics print`: to show metric values in console
- `tailscale metrics write`: to write metrics to a file (with a tempfile
& rename dance, which is atomic on Unix).
Also, remove the `TS_DEBUG_USER_METRICS` envknob as we are getting
more confident in these metrics.
Updates tailscale/corp#22075
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Not confident this is the right way to expose this, so let's remote it
for now.
Updates tailscale/corp#22075
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
If the client cannot fetch a serial number, write a log message helping
the user understand what happened. Also, don't just return the error
immediately, since we still have a chance to collect network interface
addresses.
Updates #5902
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
When a rotation signature chain reaches a certain size, remove the
oldest rotation signature from the chain before wrapping it in a new
rotation signature.
Since all previous rotation signatures are signed by the same wrapping
pubkey (node's own tailnet lock key), the node can re-construct the
chain, re-signing previous rotation signatures. This will satisfy the
existing certificate validation logic.
Updates #13185
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Logging serial numbers every time they are read might have been useful
early on, but seems unnecessary now.
Updates #5902
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
In Tailnet Lock, there is an implicit limit on the number of rotation
signatures that can be chained before the signature becomes too long.
This program helps tailnet admins to identify nodes that have signatures
with long chains and prints commands to re-sign those node keys with a
fresh direct signature. It's a temporary mitigation measure, and we will
remove this tool as we design and implement a long-term approach for
rotation signatures.
Example output:
```
2024/08/20 18:25:03 Self: does not need re-signing
2024/08/20 18:25:03 Visible peers with valid signatures:
2024/08/20 18:25:03 Peer xxx2.yy.ts.net. (100.77.192.34) nodeid=nyDmhiZiGA11KTM59, current signature kind=direct: does not need re-signing
2024/08/20 18:25:03 Peer xxx3.yy.ts.net. (100.84.248.22) nodeid=ndQ64mDnaB11KTM59, current signature kind=direct: does not need re-signing
2024/08/20 18:25:03 Peer xxx4.yy.ts.net. (100.85.253.53) nodeid=nmZfVygzkB21KTM59, current signature kind=rotation: chain length 4, printing command to re-sign
tailscale lock sign nodekey:530bddbfbe69e91fe15758a1d6ead5337aa6307e55ac92dafad3794f8b3fc661 tlpub:4bf07597336703395f2149dce88e7c50dd8694ab5bbde3d7c2a1c7b3e231a3c2
```
To support this, the NetworkLockStatus localapi response now includes
information about signatures of all peers rather than just the invalid
ones. This is not displayed by default in `tailscale lock status`, but
will be surfaced in `tailscale lock status --json`.
Updates #13185
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
The same context we use for the HTTP request here might be re-used by
the dialer, which could result in `GotConn` being called multiple times.
We only care about the last one.
Fixes#13009
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This change adds an HTTP handler with a table showing a list of all
probes, their status, and a button that allows triggering a specific
probe.
Updates tailscale/corp#20583
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
- Keep track of the last 10 probe results and successful probe
latencies;
- Add an HTTP handler that triggers a given probe by name and returns it
result as a plaintext HTML page, showing recent probe results as a
baseline
Updates tailscale/corp#20583
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
I noticed a few places with custom http.Transport where we are not
closing idle connections when transport is no longer used.
Updates tailscale/corp#21609
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
During review of #8644 the `recover-compromised-key` command was renamed
to `revoke-key`, but the old name remained in some messages printed by
the command.
Fixestailscale/corp#19446
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
If an optional `hwaddrs` URL parameter is present, add network interface
hardware addresses to the posture identity response.
Just like with serial numbers, this requires client opt-in via MDM or
`tailscale set --posture-checking=true`
(https://tailscale.com/kb/1326/device-identity)
Updates tailscale/corp#21371
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Detection of duplicate Network Lock signature chains added in
01847e0123 failed to account for chains
originating with a SigCredential signature, which is used for wrapped
auth keys. This results in erroneous removal of signatures that
originate from the same re-usable auth key.
This change ensures that multiple nodes created by the same re-usable
auth key are not getting filtered out by the network lock.
Updates tailscale/corp#19764
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This change moves handling of wrapped auth keys to the `tka` package and
adds a test covering auth key originating signatures (SigCredential) in
netmap.
Updates tailscale/corp#19764
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
To complement the existing `onCompletion` callback, which is called
after request handler.
Updates tailscale/corp#17075
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
A non-signing node can be allowed to re-sign its new node keys following
key renewal/rotation (e.g. via `tailscale up --force-reauth`). To be
able to do this, node's TLK is written into WrappingPubkey field of the
initial SigDirect signature, signed by a signing node.
The intended use of this field implies that, for each WrappingPubkey, we
typically expect to have at most one active node with a signature
tracing back to that key. Multiple valid signatures referring to the
same WrappingPubkey can occur if a client's state has been cloned, but
it's something we explicitly discourage and don't support:
https://tailscale.com/s/clone
This change propagates rotation details (wrapping public key, a list
of previous node keys that have been rotated out) to netmap processing,
and adds tracking of obsolete node keys that, when found, will get
filtered out.
Updates tailscale/corp#19764
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
- Add current node signature to `ipnstate.NetworkLockStatus`;
- Print current node signature in a human-friendly format as part
of `tailscale lock status`.
Examples:
```
$ tailscale lock status
Tailnet lock is ENABLED.
This node is accessible under tailnet lock. Node signature:
SigKind: direct
Pubkey: [OTB3a]
KeyID: tlpub:44a0e23cd53a4b8acc02f6732813d8f5ba8b35d02d48bf94c9f1724ebe31c943
WrappingPubkey: tlpub:44a0e23cd53a4b8acc02f6732813d8f5ba8b35d02d48bf94c9f1724ebe31c943
This node's tailnet-lock key: tlpub:44a0e23cd53a4b8acc02f6732813d8f5ba8b35d02d48bf94c9f1724ebe31c943
Trusted signing keys:
tlpub:44a0e23cd53a4b8acc02f6732813d8f5ba8b35d02d48bf94c9f1724ebe31c943 1 (self)
tlpub:6fa21d242a202b290de85926ba3893a6861888679a73bc3a43f49539d67c9764 1 (pre-auth key kq3NzejWoS11KTM59)
```
For a node created via a signed auth key:
```
This node is accessible under tailnet lock. Node signature:
SigKind: rotation
Pubkey: [e3nAO]
Nested:
SigKind: credential
KeyID: tlpub:6fa21d242a202b290de85926ba3893a6861888679a73bc3a43f49539d67c9764
WrappingPubkey: tlpub:3623b0412cab0029cb1918806435709b5947ae03554050f20caf66629f21220a
```
For a node that rotated its key a few times:
```
This node is accessible under tailnet lock. Node signature:
SigKind: rotation
Pubkey: [DOzL4]
Nested:
SigKind: rotation
Pubkey: [S/9yU]
Nested:
SigKind: rotation
Pubkey: [9E9v4]
Nested:
SigKind: direct
Pubkey: [3QHTJ]
KeyID: tlpub:44a0e23cd53a4b8acc02f6732813d8f5ba8b35d02d48bf94c9f1724ebe31c943
WrappingPubkey: tlpub:2faa280025d3aba0884615f710d8c50590b052c01a004c2b4c2c9434702ae9d0
```
Updates tailscale/corp#19764
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
If seamless key renewal is enabled, we typically do not stop the engine
(deconfigure networking). However, if the node key has expired there is
no point in keeping the connection up, and it might actually prevent
key renewal if auth relies on endpoints routed via app connectors.
Fixestailscale/corp#5800
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
- Wrap each prober function into a probe class that allows associating
metric labels and custom metrics with a given probe;
- Make sure all existing probe classes set a `class` metric label;
- Move bandwidth probe size from being a metric label to a separate
gauge metric; this will make it possible to use it to calculate
average used bandwidth using a PromQL query;
- Also export transfer time for the bandwidth prober (more accurate than
the total probe time, since it excludes connection establishment
time).
Updates tailscale/corp#17912
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
I was running all tests while preparing a recent stable release, and
this was failing because my computer is connected to a fairly large
tailnet.
```
--- FAIL: TestGetRouteTable (0.01s)
routetable_linux_test.go:32: expected at least one default route;
...
```
```
$ ip route show table 52 | wc -l
1051
```
Updates #cleanup
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
When reverse path filtering is in strict mode on Linux, using an exit
node blocks all network connectivity. This change adds a warning about
this to `tailscale status` and the logs.
Example in `tailscale status`:
```
- not connected to home DERP region 22
- The following issues on your machine will likely make usage of exit nodes impossible: [interface "eth0" has strict reverse-path filtering enabled], please set rp_filter=2 instead of rp_filter=1; see https://github.com/tailscale/tailscale/issues/3310
```
Example in the logs:
```
2024/02/21 21:17:07 health("overall"): error: multiple errors:
not in map poll
The following issues on your machine will likely make usage of exit nodes impossible: [interface "eth0" has strict reverse-path filtering enabled], please set rp_filter=2 instead of rp_filter=1; see https://github.com/tailscale/tailscale/issues/3310
```
Updates #3310
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
- add a clientmetric with a counter of TCP forwarder drops due to the
max attempts;
- fix varz metric types, as they are all counters.
Updates #8210
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Having `127.0.0.53` is not the only way to use `systemd-resolved`. An
alternative way is to enable `libnss_resolve` module, which seems to now
be used by default on Debian 12 bookworm.
Fixes#8549
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of treating any lxcfs mount as an indicator that we're running
in a container, check for one of the mounts actually used by LXC
containers.
For reference, here's a list of mounts I am seeing in an LXC container:
```
$ grep lxcfs /proc/mounts
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
```
Fixes#8444
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This adds a new `SetAuthorized` method that allows setting device
authorization to true or false. I chose the method name to be consistent
with SetTags.
Updates https://github.com/tailscale/corp/issues/10160
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Installer script relies on pkgs.tailscale.com being reachable, both for
checking what Linux distros are supported, but also for actually
downloading repo configuration files, gpg keys and packages themselves.
This change adds a simple reachability check which will print an error
message when pkgs.tailscale.com is not reachable.
Fixes https://github.com/tailscale/corp/issues/8952
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Every time we change `installer.sh`, run it in a few docker
containers based on different Linux distros, just as a simple test.
Also includes a few changes to the installer script itself to make
installation work in docker:
- install dnf config-manager command before running it
- run zypper in non-interactive mode
- update pacman indexes before installing packages
Updates https://github.com/tailscale/corp/issues/8952
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This splits Prometheus metric handlers exposed by tsweb into two
modules:
- `varz.Handler` exposes Prometheus metrics generated by our expvar
converter;
- `promvarz.Handler` combines our expvar-converted metrics and native
Prometheus metrics.
By default, tsweb will use the promvarz handler, however users can keep
using only the expvar converter. Specifically, `tailscaled` now uses
`varz.Handler` explicitly, which avoids a dependency on the
(heavyweight) Prometheus client.
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This provides an example of using native Prometheus metrics with tsweb.
Prober library seems to be the only user of PrometheusVar, so I am
removing support for it in tsweb.
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This allows tracking packet flow via logs for prober clients. Note that
the new sclient.debug() function is called on every received packet, but
will do nothing for most clients.
I have adjusted sclient logging to print public keys in short format
rather than full. This takes effect even for existing non-debug logging
(mostly client disconnect messages).
Example logs for a packet being sent from client [SbsJn] (connected to
derper [dM2E3]) to client [10WOo] (connected to derper [AVxvv]):
```
derper [dM2E3]:
derp client 10.0.0.1:35470[SbsJn]: register single client mesh("10.0.1.1"): 4 peers
derp client 10.0.0.1:35470[SbsJn]: read frame type 4 len 40 err <nil>
derp client 10.0.0.1:35470[SbsJn]: SendPacket for [10WOo], forwarding via <derphttp_client.Client [AVxvv] url=https://10.0.1.1/derp>: <nil>
derp client 10.0.0.1:35470[SbsJn]: read frame type 0 len 0 err EOF
derp client 10.0.0.1:35470[SbsJn]: read EOF
derp client 10.0.0.1:35470[SbsJn]: sender failed: context canceled
derp client 10.0.0.1:35470[SbsJn]: removing connection
derper [AVxvv]:
derp client 10.0.1.1:50650[10WOo]: register single client
derp client 10.0.1.1:50650[10WOo]: received forwarded packet from [SbsJn] via [dM2E3]
derp client 10.0.1.1:50650[10WOo]: sendPkt attempt 0 enqueued
derp client 10.0.1.1:50650[10WOo]: sendPacket from [SbsJn]: <nil>
derp client 10.0.1.1:50650[10WOo]: read frame type 0 len 0 err EOF
derp client 10.0.1.1:50650[10WOo]: read EOF
derp client 10.0.1.1:50650[10WOo]: sender failed: context canceled
derp client 10.0.1.1:50650[10WOo]: removing connection
```
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
`prober.DERP` was created in #5988 based on derpprobe. Having used it
instead of derpprobe for a few months, I think we have enough confidence
that it works and can now migrate derpprobe to use the prober framework
and get rid of code duplication.
A few notable changes in behaviour:
- results of STUN probes over IPv4 and IPv6 are now reported separately;
- TLS probing now includes OCSP verification;
- probe names in the output have changed;
- ability to send Slack notification from the prober has been removed.
Instead, the prober now exports metrics in Expvar (/debug/vars) and
Prometheus (/debug/varz) formats.
Fixes https://github.com/tailscale/corp/issues/8497
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This will ensure that the `tailscale-archive-keyring` Debian package
gets installed by the installer script.
Updates #3151
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This ensures that we capture error returned by `Serve` and exit with a
non-zero exit code if it happens.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This documents the `If-Match: ts-default` header that can be set to only
overwrite the default ACL contents, and also briefly mentions a few of
the new top-level ACL fields.
Updates tailscale/terraform-provider-tailscale#182
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of iterating over the map to determine the preferred forwarder
on every packet (which could happen concurrently with map mutations),
store it separately in an atomic variable.
Fixes#6445
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Ideally we should strip other invalid characters too, but that would
call for a regexp replacement which increases the number of allocations
and makes `TestVarzHandlerSorting` fail.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of returning a custom error, use ErrGetBaseConfigNotSupported
that seems to be intended for this use case. This fixes DNS resolution
on macOS clients compiled from source.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This makes tags, creation time, exit node option and primary routes
for the current node exposed via `tailscale status --json`
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
By default all probes with the same probe interval that have been added
together will run on a synchronized schedule, which results in spiky
resource usage and potential throttling by third-party systems (for
example, OCSP servers used by the TLS probes).
To address this, prober can now run in "spread" mode that will
introduce a random delay before the first run of each probe.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This ensures that each DERP server is probed individually (TLS and STUN)
and also manages per-region mesh probing. Actual probing code has been
copied from cmd/derpprobe.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
