Introduce a per-tailnet shared ACME account key so that all ingress
ProxyGroup replicas on a tailnet present the same account identity to
Let's Encrypt. This lets renewals claim the ARI "replaces" exemption
from the 50-certs-per-week rate limit, surviving Pod restarts,
ProxyGroup recreation, and cluster migrations.
The operator provisions a "tailscale-acme-accounts" Secret in its
namespace, guarded by a finalizer and a deletion warning event, and
watched so it is recreated promptly if removed. Proxies migrate any
pre-existing per-pod key into the shared Secret on first boot, adopt
the shared key on subsequent boots, and restore it on cert writes if
the Secret was recreated empty. Certs are stamped with the fingerprint
of the issuing account so renewals skip the "replaces" claim when the
account doesn't match.
Opt-in per-ProxyGroup via the tailscale.com/share-acme-account
annotation, or operator-wide via OPERATOR_SHARED_ACME_ACCOUNT_KEY.
Updates #18251
Updates #20288
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
An SNI ServerName with a trailing dot (e.g. "host.ts.net.") failed
cert lookup because stored cert names have no trailing dot. Per RFC
6066 section 3 the SNI HostName carries no trailing dot, but some
clients send a fully-qualified name with one.
Trim the trailing dot at the boundary in getCertPEMWithValidity so all
lookup paths (the GetCertificate hook, Serve, and the localapi) resolve
the dotted and dotless forms to the same certificate.
Fixes#10233
Signed-off-by: Saleh <root@lr0.org>
The extension's acmeMu was a single lock around getCertPEM. Any
in-flight ACME flow blocked every other domain. With many domains
(ProxyGroup ingress) the queue would back up and per-call timeouts
started firing while we were just waiting on the lock -- the cert
loop treated that as a failure.
Replace with one mutex per domain. Different domains run at the
same time. Same domain still queues so the first run fills the
cache and the rest read from it.
The old global lock also kept ACME account setup safe by accident.
Two goroutines could both find no account key, both generate one,
both write -- last one wins on disk but each carries on with its
own. Add acmeAccountMu around acmeKey and ensureACMEAccount to
keep that path single-file. Otherwise two first-time issuances for
different domains end up with separate accounts at LE.
Updates #20288
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* ipn/localapi,ipnlocal,feature/acme,client/local: honour Retry-After on cert rate-limit
serveCert now responds with 429 + Retry-After when the underlying ACME
error is a rate limit, instead of a generic 500. client/local surfaces
this as a typed RateLimitedError with the parsed hint so callers can
back off intelligently.
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
* tsweb,feature/acme,ipn/localapi,ipnlocal: generalise cert error → HTTP mapping via tsweb.HTTPStatuser
Introduces a tsweb.HTTPStatuser interface, any error can implement
to describe its intended HTTP response (code, message, headers).
Moves CertRateLimitedError from ipnlocal to feature/acme where it's
constructed, and it now uses HTTPStatuser to return 429 + Retry-After.
serveCert now checks for tsweb.HTTPStatuser rather than the specific
error type, so it no longer needs to know about the ACME rate-limit
type.
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
---------
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
f5eac39ea ("feature/acme, ipn/ipnlocal: start moving ACME/cert state
into an extension") started to move the cert code into feature/acme
but was meant as a baby step.
This goes further, moving almost everything, leaving only some hooks
in ipnlocal.
When we later move "serve" support out to feature/serve, this will
look a bit different in that the hooks currently in ipnlocal will move
to feature/serve (cert support already depends on serve).
As part of this, cert-related tests move to feaure/acme too, which
means some test infra from ipnlocal now moves to shared ipnlocaltest.
(it's not big at the moment, but I imagine it growing)
Updates #12614
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9ea89aa9754f12d54b81751b6bd830f2664241ff
The ACME serialization mutex (acmeMu) was a package-level global, and
several ACME-related fields lived on LocalBackend even though the
cert code is conditional and not linked into every binary. With
multiple tsnet.Servers in one process (each its own LocalBackend),
a process-wide acmeMu also serialized unrelated backends.
Introduce a new feature/acme extension that owns the per-LocalBackend
ACME/cert state in an ipnlocal.CertState value:
- acmeMu, renewMu, renewCertAt (previously package globals)
- pendingACMETLSALPNCerts, pendingCertDomains{,Mu},
getCertForTest, certRefreshCancel (previously LocalBackend
fields, only meaningful when ACME was compiled in)
ipnlocal/cert.go now reaches the state through b.certState(), which
is routed by a feature.Hook installed at init by feature/acme. The
CertState type lives in ipnlocal so cert.go can access its fields
directly without a method explosion; the extension in feature/acme
constructs and owns it.
This is a baby step. The end goal is for the entire cert/ACME code
to live in feature/acme, with ipnlocal only retaining whatever thin
hooks the rest of LocalBackend needs to call into it. The current
split (CertState and most of cert.go in ipnlocal, extension wrapper
in feature/acme) is a deliberately temporary middle ground that
keeps this PR small while making the next moves mechanical.
The package is named feature/acme to match the existing HasACME /
ts_omit_acme naming. condregister/maybe_acme.go wires it in for
non-js builds.
Updates #12614
Updates #20248
Updates #20249
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I520909f24ad11a9622ef33c2290fe36ad44d6f71
chaosinthecrd
SalehandBrad Fitzpatrick
Brad FitzpatrickandBrad Fitzpatrick