Introduce a per-tailnet shared ACME account key so that all ingress
ProxyGroup replicas on a tailnet present the same account identity to
Let's Encrypt. This lets renewals claim the ARI "replaces" exemption
from the 50-certs-per-week rate limit, surviving Pod restarts,
ProxyGroup recreation, and cluster migrations.
The operator provisions a "tailscale-acme-accounts" Secret in its
namespace, guarded by a finalizer and a deletion warning event, and
watched so it is recreated promptly if removed. Proxies migrate any
pre-existing per-pod key into the shared Secret on first boot, adopt
the shared key on subsequent boots, and restore it on cert writes if
the Secret was recreated empty. Certs are stamped with the fingerprint
of the issuing account so renewals skip the "replaces" claim when the
account doesn't match.
Opt-in per-ProxyGroup via the tailscale.com/share-acme-account
annotation, or operator-wide via OPERATOR_SHARED_ACME_ACCOUNT_KEY.
Updates #18251
Updates #20288
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Rate-limit responses from the CA now use the Retry-After hint (via
client/local.RateLimitRetryAfter) instead of walking the local retry
schedule.
Failures that never reached the CA -- context deadline/cancel,
ECONNREFUSED, ECONNRESET, EHOSTUNREACH, EPIPE, and net.Error
timeouts -- retry at retrySchedule[0] without advancing retryCount.
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The cert loop only stops when the domain leaves the ServeConfig.
Deleting the VIPService first left the loop hammering ACME for a
domain the control plane no longer recognised, burning retry slots.
Reorder to: remove from serve config, unadvertise, delete VIPService,
clean cert resources.
Updates #20288
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The extension's acmeMu was a single lock around getCertPEM. Any
in-flight ACME flow blocked every other domain. With many domains
(ProxyGroup ingress) the queue would back up and per-call timeouts
started firing while we were just waiting on the lock -- the cert
loop treated that as a failure.
Replace with one mutex per domain. Different domains run at the
same time. Same domain still queues so the first run fills the
cache and the rest read from it.
The old global lock also kept ACME account setup safe by accident.
Two goroutines could both find no account key, both generate one,
both write -- last one wins on disk but each carries on with its
own. Add acmeAccountMu around acmeKey and ensureACMEAccount to
keep that path single-file. Otherwise two first-time issuances for
different domains end up with separate accounts at LE.
Updates #20288
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* ipn/localapi,ipnlocal,feature/acme,client/local: honour Retry-After on cert rate-limit
serveCert now responds with 429 + Retry-After when the underlying ACME
error is a rate limit, instead of a generic 500. client/local surfaces
this as a typed RateLimitedError with the parsed hint so callers can
back off intelligently.
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* tsweb,feature/acme,ipn/localapi,ipnlocal: generalise cert error → HTTP mapping via tsweb.HTTPStatuser
Introduces a tsweb.HTTPStatuser interface, any error can implement
to describe its intended HTTP response (code, message, headers).
Moves CertRateLimitedError from ipnlocal to feature/acme where it's
constructed, and it now uses HTTPStatuser to return 429 + Retry-After.
serveCert now checks for tsweb.HTTPStatuser rather than the specific
error type, so it no longer needs to know about the ACME rate-limit
type.
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
---------
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
All issuances serialise through a single mutex in tailscaled. The old
300s timeout fired while a predecessor was legitimately mid-ACME,
causing the queued loop to advance retryCount on a non-failure. 30m
covers ~15 queued flows and works as a wedge detector against true
hangs.
Updates #20288
Updates #42164
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
364b952d6 switched containerboot to partial netmap fetching, but
stopped refreshing `DNS.ExtraRecords`, so Tailscale Services created
after pod boot were invisible to resolveTailnetFQDN. To fix we watch
for SelfChange ipn bus notifies, and refetch dns-config via LocalAPI
to get a fresh set of `DNS.ExtraRecords`.
Fixes#20233
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* kube/authkey,cmd/containerboot: extract shared auth key reissue package
Move auth key reissue logic (set marker, wait for new key, clear marker,
read config) into a shared kube/authkey package and update containerboot
to use it. No behaviour change.
Updates #14080
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* kube/authkey,kube/state,cmd/containerboot: preserve device_id across restarts
Stop clearing device_id, device_fqdn, and device_ips from state on startup.
These keys are now preserved across restarts so the operator can track
device identity. Expand ClearReissueAuthKey to clear device state and
tailscaled profile data when performing a full auth key reissue.
Updates #14080
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* cmd/containerboot: use root context for auth key reissue wait
Pass the root context instead of bootCtx to setAndWaitForAuthKeyReissue.
The 60-second bootCtx timeout was cancelling the reissue wait before the
operator had time to respond, causing the pod to crash-loop.
Updates #14080
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* cmd/k8s-proxy: add auth key renewal support
Add auth key reissue handling to k8s-proxy, mirroring containerboot.
When the proxy detects an auth failure (login-state health warning or
NeedsLogin state), it disconnects from control, signals the operator
via the state Secret, waits for a new key, clears stale state, and
exits so Kubernetes restarts the pod with the new key.
A health watcher goroutine runs alongside ts.Up() to short-circuit
the startup timeout on terminal auth failures.
Updates #14080
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
---------
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Fixes UDP listeners on VIP Service addresses not receiving inbound traffic.
- Modified shouldProcessInbound to check for registered UDP transport endpoints when processing packets to service VIPs
- Uses FindTransportEndpoint to determine if a UDP listener exists for the destination VIP/port
- Supports both IPv4 and IPv6
The aim was to mirror the existing TCP logic, providing feature parity for UDP-based services on VIP Services.
Fixes#18971
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* cmd/containerboot,kube/services: support the ability to automatically advertise services on startup
Updates #17769
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* cmd/containerboot: don't assume we want to use kube state store if in kubernetes
Fixes#8188
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
---------
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* k8s-operator,kube: removing enableSessionRecordings option. It seems
like it is going to create a confusing user experience and it's going to
be a very niche use case, so we have decided to defer this for now.
Updates tailscale/corp#35796
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* k8s-operator: adding metric for env var deprecation
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
---------
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
when tsrecorder receives events, it populates this field with
information about the node the request was sent to.
Updates #17141
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The hijacker on k8s-proxy's reverse proxy is used to stream recordings
to tsrecorder as they pass through the proxy to the kubernetes api
server. The connection to the recorder was using the client's
(e.g., kubectl) context, rather than a dedicated one. This was causing
the recording stream to get cut off in scenarios where the client
cancelled the context before streaming could be completed.
By using a dedicated context, we can continue streaming even if the
client cancels the context (for example if the client request
completes).
Fixes#17404
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
kubestore init function has now been moved to a more explicit path of
ipn/store/kubestore meaning we can now avoid the generic import of
feature/condregister.
Updates #12614
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
https://github.com/tailscale/tailscale/pull/17346 moved the kube and aws
arn store initializations to feature/condregister, under the assumption
that anything using it would use kubestore.New. Unfortunately,
cmd/k8s-proxy makes use of store.New, which compares the `<prefix>:`
supplied in the provided `path string` argument against known stores. If
it doesn't find it, it fallsback to using a FileStore.
Since cmd/k8s-proxy uses store.New to try and initialize a kube store in
some cases (without importing feature/condregister), it silently creates
a FileStore and that leads to misleading errors further along in
execution.
This fixes this issue by importing condregister, and successfully
initializes a kube store.
Updates #12614
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* cmd/k8s-operator: ProxyClass annotation for Services and Ingresses
Previously, the ProxyClass could only be configured for Services and
Ingresses via a Label. This adds the ability to set it via an
Annotation, but prioritizes the Label if both a Label and Annotation are
set.
Updates #14323
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* Update cmd/k8s-operator/operator.go
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Signed-off-by: Tom Meadows <tom@tmlabs.co.uk>
* Update cmd/k8s-operator/operator.go
Signed-off-by: Tom Meadows <tom@tmlabs.co.uk>
* cmd/k8s-operator: ProxyClass annotation for Services and Ingresses
Previously, the ProxyClass could only be configured for Services and
Ingresses via a Label. This adds the ability to set it via an
Annotation, but prioritizes the Label if both a Label and Annotation are
set.
Updates #14323
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
---------
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Signed-off-by: Tom Meadows <tom@tmlabs.co.uk>
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
This reconciler allows users to make applications highly available at L3 by
leveraging Tailscale Virtual Services. Many Kubernetes Service's
(irrespective of the cluster they reside in) can be mapped to a
Tailscale Virtual Service, allowing access to these Services at L3.
Updates #15895
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
chaosinthecrd
