- Wrap each prober function into a probe class that allows associating
metric labels and custom metrics with a given probe;
- Make sure all existing probe classes set a `class` metric label;
- Move bandwidth probe size from being a metric label to a separate
gauge metric; this will make it possible to use it to calculate
average used bandwidth using a PromQL query;
- Also export transfer time for the bandwidth prober (more accurate than
the total probe time, since it excludes connection establishment
time).
Updates tailscale/corp#17912
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
I was running all tests while preparing a recent stable release, and
this was failing because my computer is connected to a fairly large
tailnet.
```
--- FAIL: TestGetRouteTable (0.01s)
routetable_linux_test.go:32: expected at least one default route;
...
```
```
$ ip route show table 52 | wc -l
1051
```
Updates #cleanup
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
When reverse path filtering is in strict mode on Linux, using an exit
node blocks all network connectivity. This change adds a warning about
this to `tailscale status` and the logs.
Example in `tailscale status`:
```
- not connected to home DERP region 22
- The following issues on your machine will likely make usage of exit nodes impossible: [interface "eth0" has strict reverse-path filtering enabled], please set rp_filter=2 instead of rp_filter=1; see https://github.com/tailscale/tailscale/issues/3310
```
Example in the logs:
```
2024/02/21 21:17:07 health("overall"): error: multiple errors:
not in map poll
The following issues on your machine will likely make usage of exit nodes impossible: [interface "eth0" has strict reverse-path filtering enabled], please set rp_filter=2 instead of rp_filter=1; see https://github.com/tailscale/tailscale/issues/3310
```
Updates #3310
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
- add a clientmetric with a counter of TCP forwarder drops due to the
max attempts;
- fix varz metric types, as they are all counters.
Updates #8210
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Having `127.0.0.53` is not the only way to use `systemd-resolved`. An
alternative way is to enable `libnss_resolve` module, which seems to now
be used by default on Debian 12 bookworm.
Fixes#8549
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of treating any lxcfs mount as an indicator that we're running
in a container, check for one of the mounts actually used by LXC
containers.
For reference, here's a list of mounts I am seeing in an LXC container:
```
$ grep lxcfs /proc/mounts
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
```
Fixes#8444
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This adds a new `SetAuthorized` method that allows setting device
authorization to true or false. I chose the method name to be consistent
with SetTags.
Updates https://github.com/tailscale/corp/issues/10160
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Installer script relies on pkgs.tailscale.com being reachable, both for
checking what Linux distros are supported, but also for actually
downloading repo configuration files, gpg keys and packages themselves.
This change adds a simple reachability check which will print an error
message when pkgs.tailscale.com is not reachable.
Fixes https://github.com/tailscale/corp/issues/8952
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Every time we change `installer.sh`, run it in a few docker
containers based on different Linux distros, just as a simple test.
Also includes a few changes to the installer script itself to make
installation work in docker:
- install dnf config-manager command before running it
- run zypper in non-interactive mode
- update pacman indexes before installing packages
Updates https://github.com/tailscale/corp/issues/8952
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This splits Prometheus metric handlers exposed by tsweb into two
modules:
- `varz.Handler` exposes Prometheus metrics generated by our expvar
converter;
- `promvarz.Handler` combines our expvar-converted metrics and native
Prometheus metrics.
By default, tsweb will use the promvarz handler, however users can keep
using only the expvar converter. Specifically, `tailscaled` now uses
`varz.Handler` explicitly, which avoids a dependency on the
(heavyweight) Prometheus client.
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This provides an example of using native Prometheus metrics with tsweb.
Prober library seems to be the only user of PrometheusVar, so I am
removing support for it in tsweb.
Updates https://github.com/tailscale/corp/issues/10205
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This allows tracking packet flow via logs for prober clients. Note that
the new sclient.debug() function is called on every received packet, but
will do nothing for most clients.
I have adjusted sclient logging to print public keys in short format
rather than full. This takes effect even for existing non-debug logging
(mostly client disconnect messages).
Example logs for a packet being sent from client [SbsJn] (connected to
derper [dM2E3]) to client [10WOo] (connected to derper [AVxvv]):
```
derper [dM2E3]:
derp client 10.0.0.1:35470[SbsJn]: register single client mesh("10.0.1.1"): 4 peers
derp client 10.0.0.1:35470[SbsJn]: read frame type 4 len 40 err <nil>
derp client 10.0.0.1:35470[SbsJn]: SendPacket for [10WOo], forwarding via <derphttp_client.Client [AVxvv] url=https://10.0.1.1/derp>: <nil>
derp client 10.0.0.1:35470[SbsJn]: read frame type 0 len 0 err EOF
derp client 10.0.0.1:35470[SbsJn]: read EOF
derp client 10.0.0.1:35470[SbsJn]: sender failed: context canceled
derp client 10.0.0.1:35470[SbsJn]: removing connection
derper [AVxvv]:
derp client 10.0.1.1:50650[10WOo]: register single client
derp client 10.0.1.1:50650[10WOo]: received forwarded packet from [SbsJn] via [dM2E3]
derp client 10.0.1.1:50650[10WOo]: sendPkt attempt 0 enqueued
derp client 10.0.1.1:50650[10WOo]: sendPacket from [SbsJn]: <nil>
derp client 10.0.1.1:50650[10WOo]: read frame type 0 len 0 err EOF
derp client 10.0.1.1:50650[10WOo]: read EOF
derp client 10.0.1.1:50650[10WOo]: sender failed: context canceled
derp client 10.0.1.1:50650[10WOo]: removing connection
```
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
`prober.DERP` was created in #5988 based on derpprobe. Having used it
instead of derpprobe for a few months, I think we have enough confidence
that it works and can now migrate derpprobe to use the prober framework
and get rid of code duplication.
A few notable changes in behaviour:
- results of STUN probes over IPv4 and IPv6 are now reported separately;
- TLS probing now includes OCSP verification;
- probe names in the output have changed;
- ability to send Slack notification from the prober has been removed.
Instead, the prober now exports metrics in Expvar (/debug/vars) and
Prometheus (/debug/varz) formats.
Fixes https://github.com/tailscale/corp/issues/8497
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This will ensure that the `tailscale-archive-keyring` Debian package
gets installed by the installer script.
Updates #3151
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This ensures that we capture error returned by `Serve` and exit with a
non-zero exit code if it happens.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This documents the `If-Match: ts-default` header that can be set to only
overwrite the default ACL contents, and also briefly mentions a few of
the new top-level ACL fields.
Updates tailscale/terraform-provider-tailscale#182
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of iterating over the map to determine the preferred forwarder
on every packet (which could happen concurrently with map mutations),
store it separately in an atomic variable.
Fixes#6445
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Ideally we should strip other invalid characters too, but that would
call for a regexp replacement which increases the number of allocations
and makes `TestVarzHandlerSorting` fail.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of returning a custom error, use ErrGetBaseConfigNotSupported
that seems to be intended for this use case. This fixes DNS resolution
on macOS clients compiled from source.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This makes tags, creation time, exit node option and primary routes
for the current node exposed via `tailscale status --json`
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
By default all probes with the same probe interval that have been added
together will run on a synchronized schedule, which results in spiky
resource usage and potential throttling by third-party systems (for
example, OCSP servers used by the TLS probes).
To address this, prober can now run in "spread" mode that will
introduce a random delay before the first run of each probe.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This ensures that each DERP server is probed individually (TLS and STUN)
and also manages per-region mesh probing. Actual probing code has been
copied from cmd/derpprobe.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Instead of treating any interface with a non-ifscope route as a
potential default gateway, now verify that a given route is
actually a default route (0.0.0.0/0 or ::/0).
Fixes#5879
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
TLS prober now checks validity period for all server certificates
and verifies OCSP revocation status for the leaf cert.
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
