Previously, the operator checked the ProxyGroup status fields for
information on how many of the proxies had successfully authed. Use
their state Secrets instead as a more reliable source of truth.
containerboot has written device_fqdn and device_ips keys to the
state Secret since inception, and pod_uid since 1.78.0, so there's
no need to use the API for that data. Read it from the state Secret
for consistency. However, to ensure we don't read data from a
previous run of containerboot, make sure we reset containerboot's
state keys on startup.
One other knock-on effect of that is ProxyGroups can briefly be
marked not Ready while a Pod is restarting. Introduce a new
ProxyGroupAvailable condition to more accurately reflect
when downstream controllers can implement flows that rely on a
ProxyGroup having at least 1 proxy Pod running.
Fixes#16327
Change-Id: I026c18e9d23e87109a471a87b8e4fb6271716a66
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
In 1.84 we made 'tailscale set'/'tailscale up' error out if duplicate
command line flags are passed.
This broke some container configurations as we have two env vars that
can be used to set --accept-dns flag:
- TS_ACCEPT_DNS- specifically for --accept-dns
- TS_EXTRA_ARGS- accepts any arbitrary 'tailscale up'/'tailscale set'
flag.
We default TS_ACCEPT_DNS to false (to make the container behaviour more
declarative), which with the new restrictive CLI behaviour resulted in
failure for users who had set --accept-dns via TS_EXTRA_ARGS as the flag would be
provided twice.
This PR re-instates the previous behaviour by checking if TS_EXTRA_ARGS
contains --accept-dns flag and if so using its value to override TS_ACCEPT_DNS.
Updates tailscale/tailscale#16108
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
The test suite had grown to about 20s on my machine, but it doesn't
do much taxing work so was a good candidate to parallelise. Now runs
in under 2s on my machine.
Updates #cleanup
Change-Id: I2fcc6be9ca226c74c0cb6c906778846e959492e4
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
tailscaled's ipn package writes a collection of keys to state after
authenticating to control, but one at a time. If containerboot happens
to send a SIGTERM signal to tailscaled in the middle of writing those
keys, it may shut down with an inconsistent state Secret and never
recover. While we can't durably fix this with our current single-use
auth keys (no atomic operation to auth + write state), we can reduce
the window for this race condition by checking for partial state
before sending SIGTERM to tailscaled. Best effort only.
Updates #14080
Change-Id: I0532d51b6f0b7d391e538468bd6a0a80dbe1d9f7
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
cmd/{containerboot,k8s-operator},kube: add preshutdown hook for egress PG proxies
This change is part of work towards minimizing downtime during update
rollouts of egress ProxyGroup replicas.
This change:
- updates the containerboot health check logic to return Pod IP in headers,
if set
- always runs the health check for egress PG proxies
- updates ClusterIP Services created for PG egress endpoints to include
the health check endpoint
- implements preshutdown endpoint in proxies. The preshutdown endpoint
logic waits till, for all currently configured egress services, the ClusterIP
Service health check endpoint is no longer returned by the shutting-down Pod
(by looking at the new Pod IP header).
- ensures that kubelet is configured to call the preshutdown endpoint
This reduces the possibility that, as replicas are terminated during an update,
a replica gets terminated to which cluster traffic is still being routed via
the ClusterIP Service because kube proxy has not yet updated routig rules.
This is not a perfect check as in practice, it only checks that the kube
proxy on the node on which the proxy runs has updated rules. However, overall
this might be good enough.
The preshutdown logic is disabled if users have configured a custom health check
port via TS_LOCAL_ADDR_PORT env var. This change throws a warnign if so and in
future setting of that env var for operator proxies might be disallowed (as users
shouldn't need to configure this for a Pod directly).
This is backwards compatible with earlier proxy versions.
Updates tailscale/tailscale#14326
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
cmd/containerboot,kube/kubetypes,cmd/k8s-operator: detect if Ingress is created in a tailnet that has no HTTPS
This attempts to make Kubernetes Operator L7 Ingress setup failures more explicit:
- the Ingress resource now only advertises HTTPS endpoint via status.ingress.loadBalancer.hostname when/if the proxy has succesfully loaded serve config
- the proxy attempts to catch cases where HTTPS is disabled for the tailnet and logs a warning
Updates tailscale/tailscale#12079
Updates tailscale/tailscale#10407
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
* cmd/containerboot: serve health on local endpoint
We introduced stable (user) metrics in #14035, and `TS_LOCAL_ADDR_PORT`
with it. Rather than requiring users to specify a new addr/port
combination for each new local endpoint they want the container to
serve, this combines the health check endpoint onto the local addr/port
used by metrics if `TS_ENABLE_HEALTH_CHECK` is used instead of
`TS_HEALTHCHECK_ADDR_PORT`.
`TS_LOCAL_ADDR_PORT` now defaults to binding to all interfaces on 9002
so that it works more seamlessly and with less configuration in
environments other than Kubernetes, where the operator always overrides
the default anyway. In particular, listening on localhost would not be
accessible from outside the container, and many scripted container
environments do not know the IP address of the container before it's
started. Listening on all interfaces allows users to just set one env
var (`TS_ENABLE_METRICS` or `TS_ENABLE_HEALTH_CHECK`) to get a fully
functioning local endpoint they can query from outside the container.
Updates #14035, #12898
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
cmd/containerboot,cmd/k8s-operator: enable IPv6 for fqdn egress proxies
Don't skip installing egress forwarding rules for IPv6 (as long as the host
supports IPv6), and set headless services `ipFamilyPolicy` to
`PreferDualStack` to optionally enable both IP families when possible. Note
that even with `PreferDualStack` set, testing a dual-stack GKE cluster with
the default DNS setup of kube-dns did not correctly set both A and
AAAA records for the headless service, and instead only did so when
switching the cluster DNS to Cloud DNS. For both IPv4 and IPv6 to work
simultaneously in a dual-stack cluster, we require headless services to
return both A and AAAA records.
If the host doesn't support IPv6 but the FQDN specified only has IPv6
addresses available, containerboot will exit with error code 1 and an
error message because there is no viable egress route.
Fixes#12215
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
Turn off stateful filtering for egress proxies to allow cluster
traffic to be forwarded to tailnet.
Allow configuring stateful filter via tailscaled config file.
Deprecate EXPERIMENTAL_TS_CONFIGFILE_PATH env var and introduce a new
TS_EXPERIMENTAL_VERSIONED_CONFIG env var that can be used to provide
containerboot a directory that should contain one or more
tailscaled config files named cap-<tailscaled-cap-version>.hujson.
Containerboot will pick the one with the newest capability version
that is not newer than its current capability version.
Proxies with this change will not work with older Tailscale
Kubernetes operator versions - users must ensure that
the deployed operator is at the same version or newer (up to
4 version skew) than the proxies.
Updates tailscale/tailscale#12061
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
Co-authored-by: Maisem Ali <maisem@tailscale.com>
* cmd/containerboot: optionally configure tailscaled with a configfile.
If EXPERIMENTAL_TS_CONFIGFILE_PATH env var is set,
only run tailscaled with the provided config file.
Do not run 'tailscale up' or 'tailscale set'.
* cmd/containerboot: store containerboot accept_dns val in bool pointer
So that we can distinguish between the value being set to
false explicitly bs being unset.
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
A Tailnet node can be told to stop advertise subnets by passing
an empty string to --advertise-routes flag.
Respect an explicitly passed empty value to TS_ROUTES env var
so that users have a way to stop containerboot acting as a subnet
router without recreating it.
Distinguish between TS_ROUTES being unset and empty.
Updates tailscale/tailscale#10708
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
This partially reverts commits a61a9ab087
and 7538f38671 and fully reverts
4823a7e591.
The goal of that commit was to reapply known config whenever the
container restarts. However, that already happens when TS_AUTH_ONCE was
false (the default back then). So we only had to selectively reapply the
config if TS_AUTH_ONCE is true, this does exactly that.
This is a little sad that we have to revert to `tailscale up`, but it
fixes the backwards incompatibility problem.
Updates tailscale/tailscale#9539
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This migrates containerboot to reuse the NetfilterRunner used
by tailscaled instead of manipulating iptables rule itself.
This has the added advantage of now working with nftables and
we can potentially drop the `iptables` command from the container
image in the future.
Updates #9310
Co-authored-by: Irbe Krumina <irbe@tailscale.com>
Signed-off-by: Maisem Ali <maisem@tailscale.com>
1.50.0 switched containerboot from using `tailscale up`
to `tailscale login`. A side-effect is that a re-usable
authkey is now re-applied on every boot by `tailscale login`,
where `tailscale up` would ignore an authkey if already
authenticated.
Though this looks like it is changing the default, in reality
it is setting the default to match what 1.48 and all
prior releases actually implemented.
Fixes https://github.com/tailscale/tailscale/issues/9539
Fixes https://github.com/tailscale/corp/issues/14953
Signed-off-by: Denton Gentry <dgentry@tailscale.com>
The test was sending SIGKILL to containerboot, which results in no
signal propagation down to the bash script that is running as a child
process, thus it leaks.
Minor changes to the test daemon script, so that it cleans up the socket
that it creates on exit, and spawns fewer processes.
Fixestailscale/corp#14833
Signed-off-by: James Tucker <james@tailscale.com>
In typical k8s setups, the MTU configured on the eth0 interfaces is typically 1500 which
results in packets being dropped when they make it to proxy pods as the tailscale0 interface
has a 1280 MTU.
As the primary use of this functionality is TCP, add iptables based MSS clamping to allow
connectivity.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Ensures that Statefulset reconciler config has only one of Cluster target IP or tailnet target IP.
Adds a test case for containerboot egress proxy mode.
Updates tailscale/tailscale#8184
Signed-off-by: irbekrm <irbekrm@gmail.com>
We had two implemenetations of the kube client, merge them.
containerboot was also using a raw http.Transport, this also has
the side effect of making it use a http.Client
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This updates all source files to use a new standard header for copyright
and license declaration. Notably, copyright no longer includes a date,
and we now use the standard SPDX-License-Identifier header.
This commit was done almost entirely mechanically with perl, and then
some minimal manual fixes.
Updates #6865
Signed-off-by: Will Norris <will@tailscale.com>
We still accept the previous TS_AUTH_KEY for backwards compatibility, but the documented option name is the spelling we use everywhere else.
Updates #6321
Signed-off-by: David Anderson <danderson@tailscale.com>
In some configurations, user explicitly do not want to store
tailscale state in k8s secrets, because doing that leads to
some annoying permission issues with sidecar containers.
With this change, TS_KUBE_SECRET="" and TS_STATE_DIR=/foo
will force storage to file when running in kubernetes.
Fixes#6704.
Signed-off-by: David Anderson <danderson@tailscale.com>
We still have to shell out to `tailscale up` because the container image's
API includes "arbitrary flags to tailscale up", unfortunately. But this
should still speed up startup a little, and also enables k8s-bound containers
to update their device information as new netmap updates come in.
Fixes#6657
Signed-off-by: David Anderson <danderson@tailscale.com>
Tom Proctor
Irbe Krumina
Matt Layher
Claire Wang
Maisem Ali
Denton Gentry
James Tucker
Irbe Krumina
Mike Beaumont
Brad Fitzpatrick
Will Norris
David Anderson
Anton Tolchanov