ipn/ipnlocal, feature/acme: move most remaining cert code into feature/acme

f5eac39ea ("feature/acme, ipn/ipnlocal: start moving ACME/cert state
into an extension") started to move the cert code into feature/acme
but was meant as a baby step.

This goes further, moving almost everything, leaving only some hooks
in ipnlocal.

When we later move "serve" support out to feature/serve, this will
look a bit different in that the hooks currently in ipnlocal will move
to feature/serve (cert support already depends on serve).

As part of this, cert-related tests move to feaure/acme too, which
means some test infra from ipnlocal now moves to shared ipnlocaltest.
(it's not big at the moment, but I imagine it growing)

Updates #12614

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9ea89aa9754f12d54b81751b6bd830f2664241ff

cmd/derper/depaware.txt

@@ -246,22 +246,22 @@ tailscale.com/cmd/derper dependencies: (generated by github.com/tailscale/depawa
         crypto/internal/boring/bbig                                  from crypto/ecdsa+
         crypto/internal/boring/sig                                   from crypto/internal/boring
         crypto/internal/constanttime                                 from crypto/internal/fips140/edwards25519+
-        crypto/internal/fips140                                      from crypto/internal/fips140/aes+
+        crypto/internal/fips140                                      from crypto/fips140+
         crypto/internal/fips140/aes                                  from crypto/aes+
         crypto/internal/fips140/aes/gcm                              from crypto/cipher+
         crypto/internal/fips140/alias                                from crypto/cipher+
         crypto/internal/fips140/bigmod                               from crypto/internal/fips140/ecdsa+
-        crypto/internal/fips140/check                                from crypto/internal/fips140/aes+
+        crypto/internal/fips140/check                                from crypto/fips140+
-        crypto/internal/fips140/drbg                                 from crypto/internal/fips140/aes/gcm+
+        crypto/internal/fips140/drbg                                 from crypto/hpke+
         crypto/internal/fips140/ecdh                                 from crypto/ecdh
         crypto/internal/fips140/ecdsa                                from crypto/ecdsa
         crypto/internal/fips140/ed25519                              from crypto/ed25519
         crypto/internal/fips140/edwards25519                         from crypto/internal/fips140/ed25519
         crypto/internal/fips140/edwards25519/field                   from crypto/ecdh+
-        crypto/internal/fips140/hkdf                                 from crypto/internal/fips140/tls13+
+        crypto/internal/fips140/hkdf                                 from crypto/hkdf+
         crypto/internal/fips140/hmac                                 from crypto/hmac+
         crypto/internal/fips140/mlkem                                from crypto/mlkem
-        crypto/internal/fips140/nistec                               from crypto/elliptic+
+        crypto/internal/fips140/nistec                               from crypto/ecdsa+
         crypto/internal/fips140/nistec/fiat                          from crypto/internal/fips140/nistec
         crypto/internal/fips140/rsa                                  from crypto/rsa
         crypto/internal/fips140/sha256                               from crypto/internal/fips140/check+
@@ -327,13 +327,13 @@ tailscale.com/cmd/derper dependencies: (generated by github.com/tailscale/depawa
         internal/filepathlite                                        from os+
         internal/fmtsort                                             from fmt+
         internal/goarch                                              from crypto/internal/fips140deps/cpu+
-        internal/godebug                                             from crypto/internal/fips140deps/godebug+
+        internal/godebug                                             from crypto/ed25519+
         internal/godebugs                                            from internal/godebug+
         internal/goexperiment                                        from net/http/pprof+
         internal/goos                                                from crypto/x509+
         internal/msan                                                from internal/runtime/maps+
         internal/nettrace                                            from net+
-        internal/oserror                                             from io/fs+
+        internal/oserror                                             from internal/syscall/windows+
         internal/poll                                                from net+
         internal/profile                                             from net/http/pprof
         internal/profilerecord                                       from runtime+
@@ -343,9 +343,9 @@ tailscale.com/cmd/derper dependencies: (generated by github.com/tailscale/depawa
         internal/runtime/atomic                                      from internal/runtime/exithook+
    L    internal/runtime/cgroup                                      from runtime
         internal/runtime/exithook                                    from runtime
-        internal/runtime/gc                                          from runtime+
+        internal/runtime/gc                                          from internal/runtime/gc/scan+
         internal/runtime/gc/scan                                     from runtime
-        internal/runtime/maps                                        from reflect+
+        internal/runtime/maps                                        from hash/maphash+
         internal/runtime/math                                        from internal/runtime/maps+
         internal/runtime/pprof/label                                 from runtime+
         internal/runtime/sys                                         from crypto/subtle+
@@ -359,7 +359,7 @@ tailscale.com/cmd/derper dependencies: (generated by github.com/tailscale/depawa
         internal/synctest                                            from sync
         internal/syscall/execenv                                     from os+
   LD    internal/syscall/unix                                        from crypto/internal/sysrand+
-   W    internal/syscall/windows                                     from crypto/internal/sysrand+
+   W    internal/syscall/windows                                     from crypto/internal/fips140deps/time+
    W    internal/syscall/windows/registry                            from mime+
    W    internal/syscall/windows/sysdll                              from internal/syscall/windows+
         internal/testlog                                             from os

cmd/k8s-operator/depaware.txt

@@ -752,7 +752,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/ipn/ipnlocal/netmapcache                       from tailscale.com/ipn/ipnlocal
         tailscale.com/ipn/ipnstate                                   from tailscale.com/client/local+
         tailscale.com/ipn/localapi                                   from tailscale.com/tsnet
-        tailscale.com/ipn/store                                      from tailscale.com/ipn/ipnlocal+
+        tailscale.com/ipn/store                                      from tailscale.com/ipn/store/kubestore+
         tailscale.com/ipn/store/kubestore                            from tailscale.com/cmd/k8s-operator
         tailscale.com/ipn/store/mem                                  from tailscale.com/ipn/ipnlocal+
         tailscale.com/k8s-operator                                   from tailscale.com/cmd/k8s-operator+
@@ -828,7 +828,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/sessionrecording                               from tailscale.com/k8s-operator/sessionrecording+
         tailscale.com/syncs                                          from tailscale.com/control/controlknobs+
         tailscale.com/tailcfg                                        from tailscale.com/client/local+
-        tailscale.com/tempfork/acme                                  from tailscale.com/ipn/ipnlocal
+        tailscale.com/tempfork/acme                                  from tailscale.com/feature/acme
         tailscale.com/tempfork/heap                                  from tailscale.com/wgengine/magicsock
         tailscale.com/tempfork/httprec                               from tailscale.com/feature/c2n
         tailscale.com/tka                                            from tailscale.com/client/local+
@@ -1020,22 +1020,22 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         crypto/internal/boring/bbig                                  from crypto/ecdsa+
         crypto/internal/boring/sig                                   from crypto/internal/boring
         crypto/internal/constanttime                                 from crypto/internal/fips140/edwards25519+
-        crypto/internal/fips140                                      from crypto/internal/fips140/aes+
+        crypto/internal/fips140                                      from crypto/fips140+
         crypto/internal/fips140/aes                                  from crypto/aes+
         crypto/internal/fips140/aes/gcm                              from crypto/cipher+
         crypto/internal/fips140/alias                                from crypto/cipher+
         crypto/internal/fips140/bigmod                               from crypto/internal/fips140/ecdsa+
-        crypto/internal/fips140/check                                from crypto/internal/fips140/aes+
+        crypto/internal/fips140/check                                from crypto/fips140+
-        crypto/internal/fips140/drbg                                 from crypto/internal/fips140/aes/gcm+
+        crypto/internal/fips140/drbg                                 from crypto/hpke+
         crypto/internal/fips140/ecdh                                 from crypto/ecdh
         crypto/internal/fips140/ecdsa                                from crypto/ecdsa
         crypto/internal/fips140/ed25519                              from crypto/ed25519
         crypto/internal/fips140/edwards25519                         from crypto/internal/fips140/ed25519
         crypto/internal/fips140/edwards25519/field                   from crypto/ecdh+
-        crypto/internal/fips140/hkdf                                 from crypto/internal/fips140/tls13+
+        crypto/internal/fips140/hkdf                                 from crypto/hkdf+
         crypto/internal/fips140/hmac                                 from crypto/hmac+
         crypto/internal/fips140/mlkem                                from crypto/mlkem
-        crypto/internal/fips140/nistec                               from crypto/elliptic+
+        crypto/internal/fips140/nistec                               from crypto/ecdsa+
         crypto/internal/fips140/nistec/fiat                          from crypto/internal/fips140/nistec
         crypto/internal/fips140/rsa                                  from crypto/rsa
         crypto/internal/fips140/sha256                               from crypto/internal/fips140/check+
@@ -1115,14 +1115,14 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         internal/filepathlite                                        from os+
         internal/fmtsort                                             from fmt+
         internal/goarch                                              from crypto/internal/fips140deps/cpu+
-        internal/godebug                                             from crypto/internal/fips140deps/godebug+
+        internal/godebug                                             from crypto/ed25519+
         internal/godebugs                                            from internal/godebug+
         internal/goexperiment                                        from net/http/pprof+
         internal/goos                                                from crypto/x509+
         internal/lazyregexp                                          from go/doc
         internal/msan                                                from internal/runtime/maps+
         internal/nettrace                                            from net+
-        internal/oserror                                             from io/fs+
+        internal/oserror                                             from internal/syscall/windows+
         internal/poll                                                from net+
         internal/profile                                             from net/http/pprof
         internal/profilerecord                                       from runtime+
@@ -1132,9 +1132,9 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         internal/runtime/atomic                                      from internal/runtime/exithook+
    L    internal/runtime/cgroup                                      from runtime
         internal/runtime/exithook                                    from runtime
-        internal/runtime/gc                                          from runtime+
+        internal/runtime/gc                                          from internal/runtime/gc/scan+
         internal/runtime/gc/scan                                     from runtime
-        internal/runtime/maps                                        from reflect+
+        internal/runtime/maps                                        from hash/maphash+
         internal/runtime/math                                        from internal/runtime/maps+
         internal/runtime/pprof/label                                 from runtime+
         internal/runtime/sys                                         from crypto/subtle+
@@ -1148,7 +1148,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         internal/synctest                                            from sync
         internal/syscall/execenv                                     from os+
   LD    internal/syscall/unix                                        from crypto/internal/sysrand+
-   W    internal/syscall/windows                                     from crypto/internal/sysrand+
+   W    internal/syscall/windows                                     from crypto/internal/fips140deps/time+
    W    internal/syscall/windows/registry                            from mime+
    W    internal/syscall/windows/sysdll                              from internal/syscall/windows+
         internal/testlog                                             from os

cmd/stund/depaware.txt

@@ -141,22 +141,22 @@ tailscale.com/cmd/stund dependencies: (generated by github.com/tailscale/depawar
         crypto/internal/boring/bbig                                  from crypto/ecdsa+
         crypto/internal/boring/sig                                   from crypto/internal/boring
         crypto/internal/constanttime                                 from crypto/internal/fips140/edwards25519+
-        crypto/internal/fips140                                      from crypto/internal/fips140/aes+
+        crypto/internal/fips140                                      from crypto/fips140+
         crypto/internal/fips140/aes                                  from crypto/aes+
         crypto/internal/fips140/aes/gcm                              from crypto/cipher+
         crypto/internal/fips140/alias                                from crypto/cipher+
         crypto/internal/fips140/bigmod                               from crypto/internal/fips140/ecdsa+
-        crypto/internal/fips140/check                                from crypto/internal/fips140/aes+
+        crypto/internal/fips140/check                                from crypto/fips140+
-        crypto/internal/fips140/drbg                                 from crypto/internal/fips140/aes/gcm+
+        crypto/internal/fips140/drbg                                 from crypto/hpke+
         crypto/internal/fips140/ecdh                                 from crypto/ecdh
         crypto/internal/fips140/ecdsa                                from crypto/ecdsa
         crypto/internal/fips140/ed25519                              from crypto/ed25519
         crypto/internal/fips140/edwards25519                         from crypto/internal/fips140/ed25519
         crypto/internal/fips140/edwards25519/field                   from crypto/ecdh+
-        crypto/internal/fips140/hkdf                                 from crypto/internal/fips140/tls13+
+        crypto/internal/fips140/hkdf                                 from crypto/hkdf+
         crypto/internal/fips140/hmac                                 from crypto/hmac+
         crypto/internal/fips140/mlkem                                from crypto/mlkem
-        crypto/internal/fips140/nistec                               from crypto/elliptic+
+        crypto/internal/fips140/nistec                               from crypto/ecdsa+
         crypto/internal/fips140/nistec/fiat                          from crypto/internal/fips140/nistec
         crypto/internal/fips140/rsa                                  from crypto/rsa
         crypto/internal/fips140/sha256                               from crypto/internal/fips140/check+
@@ -221,13 +221,13 @@ tailscale.com/cmd/stund dependencies: (generated by github.com/tailscale/depawar
         internal/filepathlite                                        from os+
         internal/fmtsort                                             from fmt
         internal/goarch                                              from crypto/internal/fips140deps/cpu+
-        internal/godebug                                             from crypto/internal/fips140deps/godebug+
+        internal/godebug                                             from crypto/ed25519+
         internal/godebugs                                            from internal/godebug+
         internal/goexperiment                                        from net/http/pprof+
         internal/goos                                                from crypto/x509+
         internal/msan                                                from internal/runtime/maps+
         internal/nettrace                                            from net+
-        internal/oserror                                             from io/fs+
+        internal/oserror                                             from internal/syscall/windows+
         internal/poll                                                from net+
         internal/profile                                             from net/http/pprof
         internal/profilerecord                                       from runtime+
@@ -237,9 +237,9 @@ tailscale.com/cmd/stund dependencies: (generated by github.com/tailscale/depawar
         internal/runtime/atomic                                      from internal/runtime/exithook+
    L    internal/runtime/cgroup                                      from runtime
         internal/runtime/exithook                                    from runtime
-        internal/runtime/gc                                          from runtime+
+        internal/runtime/gc                                          from internal/runtime/gc/scan+
         internal/runtime/gc/scan                                     from runtime
-        internal/runtime/maps                                        from reflect+
+        internal/runtime/maps                                        from hash/maphash+
         internal/runtime/math                                        from internal/runtime/maps+
         internal/runtime/pprof/label                                 from runtime+
         internal/runtime/sys                                         from crypto/subtle+
@@ -253,7 +253,7 @@ tailscale.com/cmd/stund dependencies: (generated by github.com/tailscale/depawar
         internal/synctest                                            from sync
         internal/syscall/execenv                                     from os
   LD    internal/syscall/unix                                        from crypto/internal/sysrand+
-   W    internal/syscall/windows                                     from crypto/internal/sysrand+
+   W    internal/syscall/windows                                     from crypto/internal/fips140deps/time+
    W    internal/syscall/windows/registry                            from mime+
    W    internal/syscall/windows/sysdll                              from internal/syscall/windows+
         internal/testlog                                             from os

cmd/tailscale/depaware.txt

@@ -403,22 +403,22 @@ tailscale.com/cmd/tailscale dependencies: (generated by github.com/tailscale/dep
         crypto/internal/boring/bbig                                  from crypto/ecdsa+
         crypto/internal/boring/sig                                   from crypto/internal/boring
         crypto/internal/constanttime                                 from crypto/internal/fips140/edwards25519+
-        crypto/internal/fips140                                      from crypto/internal/fips140/aes+
+        crypto/internal/fips140                                      from crypto/fips140+
         crypto/internal/fips140/aes                                  from crypto/aes+
         crypto/internal/fips140/aes/gcm                              from crypto/cipher+
         crypto/internal/fips140/alias                                from crypto/cipher+
         crypto/internal/fips140/bigmod                               from crypto/internal/fips140/ecdsa+
-        crypto/internal/fips140/check                                from crypto/internal/fips140/aes+
+        crypto/internal/fips140/check                                from crypto/fips140+
-        crypto/internal/fips140/drbg                                 from crypto/internal/fips140/aes/gcm+
+        crypto/internal/fips140/drbg                                 from crypto/hpke+
         crypto/internal/fips140/ecdh                                 from crypto/ecdh
         crypto/internal/fips140/ecdsa                                from crypto/ecdsa
         crypto/internal/fips140/ed25519                              from crypto/ed25519
         crypto/internal/fips140/edwards25519                         from crypto/internal/fips140/ed25519
         crypto/internal/fips140/edwards25519/field                   from crypto/ecdh+
-        crypto/internal/fips140/hkdf                                 from crypto/internal/fips140/tls13+
+        crypto/internal/fips140/hkdf                                 from crypto/hkdf+
         crypto/internal/fips140/hmac                                 from crypto/hmac+
         crypto/internal/fips140/mlkem                                from crypto/mlkem
-        crypto/internal/fips140/nistec                               from crypto/elliptic+
+        crypto/internal/fips140/nistec                               from crypto/ecdsa+
         crypto/internal/fips140/nistec/fiat                          from crypto/internal/fips140/nistec
         crypto/internal/fips140/pbkdf2                               from crypto/pbkdf2
         crypto/internal/fips140/rsa                                  from crypto/rsa
@@ -502,7 +502,7 @@ tailscale.com/cmd/tailscale dependencies: (generated by github.com/tailscale/dep
         internal/goos                                                from crypto/x509+
         internal/msan                                                from internal/runtime/maps+
         internal/nettrace                                            from net+
-        internal/oserror                                             from io/fs+
+        internal/oserror                                             from internal/syscall/windows+
         internal/poll                                                from net+
         internal/profile                                             from net/http/pprof
         internal/profilerecord                                       from runtime+
@@ -512,9 +512,9 @@ tailscale.com/cmd/tailscale dependencies: (generated by github.com/tailscale/dep
         internal/runtime/atomic                                      from internal/runtime/exithook+
    L    internal/runtime/cgroup                                      from runtime
         internal/runtime/exithook                                    from runtime
-        internal/runtime/gc                                          from runtime+
+        internal/runtime/gc                                          from internal/runtime/gc/scan+
         internal/runtime/gc/scan                                     from runtime
-        internal/runtime/maps                                        from reflect+
+        internal/runtime/maps                                        from hash/maphash+
         internal/runtime/math                                        from internal/runtime/maps+
         internal/runtime/pprof/label                                 from runtime+
         internal/runtime/sys                                         from crypto/subtle+
@@ -528,7 +528,7 @@ tailscale.com/cmd/tailscale dependencies: (generated by github.com/tailscale/dep
         internal/synctest                                            from sync
         internal/syscall/execenv                                     from os+
   LD    internal/syscall/unix                                        from crypto/internal/sysrand+
-   W    internal/syscall/windows                                     from crypto/internal/sysrand+
+   W    internal/syscall/windows                                     from crypto/internal/fips140deps/time+
    W    internal/syscall/windows/registry                            from mime+
    W    internal/syscall/windows/sysdll                              from internal/syscall/windows+
         internal/testlog                                             from os

cmd/tailscaled/depaware.txt

@@ -405,7 +405,7 @@ tailscale.com/cmd/tailscaled dependencies: (generated by github.com/tailscale/de
   LD 💣 tailscale.com/ssh/tailssh                                    from tailscale.com/feature/ssh
         tailscale.com/syncs                                          from tailscale.com/cmd/tailscaled+
         tailscale.com/tailcfg                                        from tailscale.com/client/local+
-        tailscale.com/tempfork/acme                                  from tailscale.com/ipn/ipnlocal
+        tailscale.com/tempfork/acme                                  from tailscale.com/feature/acme
         tailscale.com/tempfork/heap                                  from tailscale.com/wgengine/magicsock
         tailscale.com/tempfork/httprec                               from tailscale.com/feature/c2n
         tailscale.com/tka                                            from tailscale.com/client/local+

cmd/tsidp/depaware.txt

@@ -169,7 +169,7 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
         tailscale.com/ipn/ipnlocal/netmapcache                       from tailscale.com/ipn/ipnlocal
         tailscale.com/ipn/ipnstate                                   from tailscale.com/client/local+
         tailscale.com/ipn/localapi                                   from tailscale.com/tsnet
-        tailscale.com/ipn/store                                      from tailscale.com/ipn/ipnlocal+
+        tailscale.com/ipn/store                                      from tailscale.com/tsnet+
         tailscale.com/ipn/store/mem                                  from tailscale.com/ipn/ipnlocal+
         tailscale.com/kube/kubetypes                                 from tailscale.com/envknob
         tailscale.com/licenses                                       from tailscale.com/client/web
@@ -179,7 +179,7 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
         tailscale.com/logtail                                        from tailscale.com/control/controlclient+
         tailscale.com/logtail/filch                                  from tailscale.com/log/sockstatlog+
         tailscale.com/metrics                                        from tailscale.com/tsweb+
-        tailscale.com/net/bakedroots                                 from tailscale.com/ipn/ipnlocal+
+        tailscale.com/net/bakedroots                                 from tailscale.com/net/tlsdial+
      💣 tailscale.com/net/batching                                   from tailscale.com/wgengine/magicsock
         tailscale.com/net/captivedetection                           from tailscale.com/ipn/ipnlocal+
         tailscale.com/net/dns                                        from tailscale.com/ipn/ipnlocal+
@@ -226,7 +226,7 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
      💣 tailscale.com/safesocket                                     from tailscale.com/client/local+
         tailscale.com/syncs                                          from tailscale.com/control/controlhttp+
         tailscale.com/tailcfg                                        from tailscale.com/client/local+
-        tailscale.com/tempfork/acme                                  from tailscale.com/ipn/ipnlocal
+        tailscale.com/tempfork/acme                                  from tailscale.com/feature/acme
         tailscale.com/tempfork/heap                                  from tailscale.com/wgengine/magicsock
         tailscale.com/tempfork/httprec                               from tailscale.com/feature/c2n
         tailscale.com/tka                                            from tailscale.com/client/local+
@@ -414,22 +414,22 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
         crypto/internal/boring/bbig                                  from crypto/ecdsa+
         crypto/internal/boring/sig                                   from crypto/internal/boring
         crypto/internal/constanttime                                 from crypto/internal/fips140/edwards25519+
-        crypto/internal/fips140                                      from crypto/internal/fips140/aes+
+        crypto/internal/fips140                                      from crypto/fips140+
         crypto/internal/fips140/aes                                  from crypto/aes+
         crypto/internal/fips140/aes/gcm                              from crypto/cipher+
         crypto/internal/fips140/alias                                from crypto/cipher+
         crypto/internal/fips140/bigmod                               from crypto/internal/fips140/ecdsa+
-        crypto/internal/fips140/check                                from crypto/internal/fips140/aes+
+        crypto/internal/fips140/check                                from crypto/fips140+
-        crypto/internal/fips140/drbg                                 from crypto/internal/fips140/aes/gcm+
+        crypto/internal/fips140/drbg                                 from crypto/hpke+
         crypto/internal/fips140/ecdh                                 from crypto/ecdh
         crypto/internal/fips140/ecdsa                                from crypto/ecdsa
         crypto/internal/fips140/ed25519                              from crypto/ed25519
         crypto/internal/fips140/edwards25519                         from crypto/internal/fips140/ed25519
         crypto/internal/fips140/edwards25519/field                   from crypto/ecdh+
-        crypto/internal/fips140/hkdf                                 from crypto/internal/fips140/tls13+
+        crypto/internal/fips140/hkdf                                 from crypto/hkdf+
         crypto/internal/fips140/hmac                                 from crypto/hmac+
         crypto/internal/fips140/mlkem                                from crypto/mlkem
-        crypto/internal/fips140/nistec                               from crypto/elliptic+
+        crypto/internal/fips140/nistec                               from crypto/ecdsa+
         crypto/internal/fips140/nistec/fiat                          from crypto/internal/fips140/nistec
         crypto/internal/fips140/pbkdf2                               from crypto/pbkdf2
         crypto/internal/fips140/rsa                                  from crypto/rsa
@@ -500,13 +500,13 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
         internal/filepathlite                                        from os+
         internal/fmtsort                                             from fmt+
         internal/goarch                                              from crypto/internal/fips140deps/cpu+
-        internal/godebug                                             from crypto/internal/fips140deps/godebug+
+        internal/godebug                                             from crypto/ed25519+
         internal/godebugs                                            from internal/godebug+
         internal/goexperiment                                        from net/http/pprof+
         internal/goos                                                from crypto/x509+
         internal/msan                                                from internal/runtime/maps+
         internal/nettrace                                            from net+
-        internal/oserror                                             from io/fs+
+        internal/oserror                                             from internal/syscall/windows+
         internal/poll                                                from net+
         internal/profile                                             from net/http/pprof
         internal/profilerecord                                       from runtime+
@@ -516,9 +516,9 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
         internal/runtime/atomic                                      from internal/runtime/exithook+
    L    internal/runtime/cgroup                                      from runtime
         internal/runtime/exithook                                    from runtime
-        internal/runtime/gc                                          from runtime+
+        internal/runtime/gc                                          from internal/runtime/gc/scan+
         internal/runtime/gc/scan                                     from runtime
-        internal/runtime/maps                                        from reflect+
+        internal/runtime/maps                                        from hash/maphash+
         internal/runtime/math                                        from internal/runtime/maps+
         internal/runtime/pprof/label                                 from runtime+
         internal/runtime/sys                                         from crypto/subtle+
@@ -532,7 +532,7 @@ tailscale.com/cmd/tsidp dependencies: (generated by github.com/tailscale/depawar
         internal/synctest                                            from sync
         internal/syscall/execenv                                     from os+
   LD    internal/syscall/unix                                        from crypto/internal/sysrand+
-   W    internal/syscall/windows                                     from crypto/internal/sysrand+
+   W    internal/syscall/windows                                     from crypto/internal/fips140deps/time+
    W    internal/syscall/windows/registry                            from mime+
    W    internal/syscall/windows/sysdll                              from internal/syscall/windows+
         internal/testlog                                             from os

feature/acme/acme.go

@@ -3,19 +3,35 @@
 
 // Package acme registers the ACME/TLS-cert feature and implements its
 // associated [ipnext.Extension]. The extension owns the per-LocalBackend
-// state previously held in package-level globals and on LocalBackend
+// ACME serialization mutex, in-flight cert tracking, the refresh loop's
-// fields (ACME serialization mutex, in-flight cert tracking, etc.).
+// cancel func, and the test-only cert override; together with the cert
+// acquisition logic in this package, it is everything tailscaled needs
+// to obtain and renew TLS certificates via ACME.
 //
-// The cert code that runs against this state still lives in
+// In builds without ACME support (js or ts_omit_acme), this package is
-// [tailscale.com/ipn/ipnlocal]; this extension simply owns the state
+// not linked in; [ipn/ipnlocal] then exposes only stub wrappers that
-// and installs a hook so cert.go can find it from a *LocalBackend.
+// return errNoCerts or no-op.
 package acme
 
 import (
+	"context"
+	"crypto/tls"
+	"errors"
+	"net/http"
+	"sync"
+	"sync/atomic"
+	"time"
+
 	"tailscale.com/feature"
+	"tailscale.com/health"
+	"tailscale.com/ipn"
 	"tailscale.com/ipn/ipnext"
 	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/syncs"
+	"tailscale.com/tsconst"
 	"tailscale.com/types/logger"
+	"tailscale.com/util/clientmetric"
+	"tailscale.com/util/set"
 )
 
 // featureName is the name of the feature implemented by this package.
@@ -24,39 +40,211 @@ const featureName = "acme"
 func init() {
 	feature.Register(featureName)
 	ipnext.RegisterExtension(featureName, newExtension)
-	ipnlocal.HookCertState.Set(certStateFor)
+
+	ipnlocal.HookGetCertPEM.Set(getCertPEMHook)
+	ipnlocal.HookGetACMETLSALPNCert.Set(getACMETLSALPNCertHook)
+	ipnlocal.HookGetACMETLSALPNProto.Set(getACMETLSALPNProtoHook)
+	ipnlocal.HookUpdateCertRefreshLoop.Set(updateCertRefreshLoopHook)
+	ipnlocal.HookShutdownCertRefreshLoop.Set(shutdownCertRefreshLoopHook)
+	ipnlocal.HookConfigureCertsForTest.Set(configureCertsForTestHook)
+	ipnlocal.HookHandleC2NTLSCertStatus.Set(handleC2NTLSCertStatus)
 }
 
+// errNoExt is returned when a hook is invoked on a [*ipnlocal.LocalBackend]
+// that has no [extension] registered (shouldn't happen in practice, but
+// guards against misuse from tests that swap extension registrations).
+var errNoExt = errors.New("acme extension not registered on this LocalBackend")
+
+// extension is the ACME/cert [ipnext.Extension]. It owns the
+// per-[*ipnlocal.LocalBackend] state previously held in package-level
+// globals and in [*ipnlocal.LocalBackend] fields.
+//
+// All methods that take a [*ipnlocal.LocalBackend] argument operate on
+// the backend the extension was instantiated for; the argument is
+// passed through from the hook in [ipn/ipnlocal] rather than stored on
+// the extension, which keeps the extension's lifecycle independent of
+// any specific backend reference.
+type extension struct {
+	logf logger.Logf
+
+	// acmeMu serializes ACME operations so concurrent requests for
+	// certs don't slam ACME. The first goroutine through populates the
+	// on-disk cache and the rest reuse it.
+	acmeMu syncs.Mutex
+
+	// renewMu guards renewCertAt.
+	// Lock order: acmeMu before renewMu.
+	renewMu     syncs.Mutex
+	renewCertAt map[string]time.Time // lazily initialized under renewMu
+
+	// pendingACMETLSALPNCerts maps SNI names to short-lived ACME
+	// tls-alpn-01 challenge certificates while an ACME order is
+	// waiting for validation. Entries are deleted by the cleanup
+	// function returned from storeACMETLSALPNCert after the challenge
+	// validation path finishes, whether it succeeds or fails.
+	pendingACMETLSALPNCerts syncs.Map[string, *tls.Certificate]
+
+	// pendingCertDomains tracks the set of domains for which an ACME
+	// issuance is currently in flight with no usable cached cert. It
+	// backs the tls-cert-pending health Warnable.
+	// Guarded by pendingCertDomainsMu.
+	pendingCertDomainsMu sync.Mutex
+	pendingCertDomains   set.Set[string]
+
+	// wg tracks all background goroutines spawned by this extension
+	// (async cert renewals, the cert refresh loop and its per-domain
+	// workers). [extension.Shutdown] waits on it.
+	wg sync.WaitGroup
+
+	// goroutinesStarted counts goroutines started via [extension.Go].
+	// Tests use it to assert whether an operation kicked off async work.
+	goroutinesStarted atomic.Int64
+
+	// mu guards the test/lifecycle fields below.
+	mu sync.Mutex
+
+	// getCertForTest is used to retrieve TLS certificates in tests.
+	// See [LocalBackend.ConfigureCertsForTest].
+	getCertForTest func(hostname string) (*ipnlocal.TLSCertKeyPair, error)
+
+	// certRefreshCancel cancels the background TLS cert refresh loop
+	// that periodically pokes [LocalBackend.GetCertPEM] so renewals
+	// happen on idle nodes. Non-nil while the loop is running.
+	certRefreshCancel context.CancelFunc
+}
+
+// Go runs f in a new goroutine tracked by e.wg. [extension.Shutdown]
+// waits for all such goroutines to finish.
+func (e *extension) Go(f func()) {
+	e.wg.Add(1)
+	e.goroutinesStarted.Add(1)
+	go func() {
+		defer e.wg.Done()
+		f()
+	}()
+}
+
+// newExtension is the [ipnext.NewExtensionFn] registered for this
+// feature. It is called once per [*ipnlocal.LocalBackend].
 func newExtension(logf logger.Logf, _ ipnext.SafeBackend) (ipnext.Extension, error) {
 	return &extension{
-		state: new(ipnlocal.CertState),
+		logf: logger.WithPrefix(logf, featureName+": "),
-		logf:  logger.WithPrefix(logf, featureName+": "),
 	}, nil
 }
 
-// extension is an [ipnext.Extension] that owns the per-LocalBackend
-// ACME/cert state. Most of the cert logic still lives in ipnlocal;
-// this extension exists to give that state a non-global home.
-type extension struct {
-	state *ipnlocal.CertState
-	logf  logger.Logf
-}
-
 // Name implements [ipnext.Extension].
 func (e *extension) Name() string { return featureName }
 
 // Init implements [ipnext.Extension].
 func (e *extension) Init(ipnext.Host) error { return nil }
 
-// Shutdown implements [ipnext.Extension].
+// Shutdown implements [ipnext.Extension]. It cancels the cert refresh
-func (e *extension) Shutdown() error { return nil }
+// loop if it's running, then waits for all in-flight goroutines
+// (async renewals, refresh loop workers) to finish.
+func (e *extension) Shutdown() error {
+	e.mu.Lock()
+	if e.certRefreshCancel != nil {
+		e.certRefreshCancel()
+		e.certRefreshCancel = nil
+	}
+	e.mu.Unlock()
+	e.wg.Wait()
+	return nil
+}
 
-// certStateFor returns the [ipnlocal.CertState] owned by the acme
+// extFor returns the [*extension] for b, or an error if no acme
-// extension registered on b, or nil if none.
+// extension is registered on b.
-func certStateFor(b *ipnlocal.LocalBackend) *ipnlocal.CertState {
+func extFor(b *ipnlocal.LocalBackend) (*extension, error) {
 	e, ok := ipnlocal.GetExt[*extension](b)
 	if !ok {
-		return nil
+		return nil, errNoExt
 	}
-	return e.state
+	return e, nil
 }
+
+// Hook adapter funcs that thread (b *ipnlocal.LocalBackend) into the
+// extension's methods. These are what get installed in
+// [ipnlocal.Hook*] at init time.
+
+func getCertPEMHook(ctx context.Context, b *ipnlocal.LocalBackend, domain string, minValidity time.Duration) (*ipnlocal.TLSCertKeyPair, error) {
+	e, err := extFor(b)
+	if err != nil {
+		return nil, err
+	}
+	return e.getCertPEMWithValidity(ctx, b, domain, minValidity)
+}
+
+func getACMETLSALPNCertHook(b *ipnlocal.LocalBackend, hi *tls.ClientHelloInfo) (*tls.Certificate, bool) {
+	e, err := extFor(b)
+	if err != nil {
+		return nil, false
+	}
+	return e.getACMETLSALPNCert(hi)
+}
+
+func getACMETLSALPNProtoHook(b *ipnlocal.LocalBackend, hi *tls.ClientHelloInfo) (string, bool) {
+	e, err := extFor(b)
+	if err != nil {
+		return "", false
+	}
+	return e.getACMETLSALPNProto(hi)
+}
+
+func updateCertRefreshLoopHook(b *ipnlocal.LocalBackend, state ipn.State, sc ipn.ServeConfigView) {
+	e, err := extFor(b)
+	if err != nil {
+		return
+	}
+	e.updateCertRefreshLoop(b, state, sc)
+}
+
+func shutdownCertRefreshLoopHook(b *ipnlocal.LocalBackend) {
+	e, err := extFor(b)
+	if err != nil {
+		return
+	}
+	e.Shutdown()
+}
+
+func configureCertsForTestHook(b *ipnlocal.LocalBackend, getCert func(string) (*ipnlocal.TLSCertKeyPair, error)) {
+	e, err := extFor(b)
+	if err != nil {
+		panic(err)
+	}
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	e.getCertForTest = getCert
+}
+
+func handleC2NTLSCertStatus(b *ipnlocal.LocalBackend, w http.ResponseWriter, r *http.Request) {
+	e, err := extFor(b)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+	e.handleC2NTLSCertStatus(b, w, r)
+}
+
+// ACME / cert metrics. These are package-level (process-wide) because
+// they aggregate across all [*extension]s in the process.
+var (
+	metricACMEDNS01Start       = clientmetric.NewCounter("cert_acme_dns01_start")
+	metricACMEDNS01Success     = clientmetric.NewCounter("cert_acme_dns01_success")
+	metricACMEDNS01Failure     = clientmetric.NewCounter("cert_acme_dns01_failure")
+	metricACMETLSALPN01Start   = clientmetric.NewCounter("cert_acme_tls_alpn01_start")
+	metricACMETLSALPN01Success = clientmetric.NewCounter("cert_acme_tls_alpn01_success")
+	metricACMETLSALPN01Failure = clientmetric.NewCounter("cert_acme_tls_alpn01_failure")
+)
+
+// certPendingWarnable fires while ACME is fetching a TLS certificate
+// for which no usable cached copy exists (initial issuance or after
+// the cached cert has expired). Async renewal of a still-valid cert
+// does not fire it.
+var certPendingWarnable = health.Register(&health.Warnable{
+	Code:     tsconst.HealthWarnableTLSCertPending,
+	Title:    "Fetching TLS certificate",
+	Severity: health.SeverityLow,
+	Text: func(args health.Args) string {
+		return "Fetching TLS certificate via ACME for: " + args[health.ArgDomains]
+	},
+})

feature/acme/c2n.go

@@ -0,0 +1,75 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+package acme
+
+import (
+	"crypto/x509"
+	"encoding/json"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"net/http"
+	"time"
+
+	"tailscale.com/ipn"
+	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/tailcfg"
+)
+
+// handleC2NTLSCertStatus returns info about the last TLS certificate
+// issued for the provided domain. It is the implementation of
+// [ipnlocal.HookHandleC2NTLSCertStatus]; control calls it to clean up
+// DNS TXT records when they're no longer needed by LetsEncrypt.
+//
+// It does not kick off a cert fetch or async refresh. It only reports
+// anything that's already sitting on disk, and only reports metadata
+// about the public cert (stuff that'd be the in CT logs anyway).
+func (e *extension) handleC2NTLSCertStatus(b *ipnlocal.LocalBackend, w http.ResponseWriter, r *http.Request) {
+	cs, err := e.getCertStore(b)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+
+	domain := r.FormValue("domain")
+	if domain == "" {
+		http.Error(w, "no 'domain'", http.StatusBadRequest)
+		return
+	}
+
+	ret := &tailcfg.C2NTLSCertInfo{}
+	pair, err := getCertPEMCached(cs, domain, b.Clock().Now())
+	ret.Valid = err == nil
+	if err != nil {
+		ret.Error = err.Error()
+		if errors.Is(err, errCertExpired) {
+			ret.Expired = true
+		} else if errors.Is(err, ipn.ErrStateNotExist) {
+			ret.Missing = true
+			ret.Error = "no certificate"
+		}
+	} else {
+		block, _ := pem.Decode(pair.CertPEM)
+		if block == nil {
+			ret.Error = "invalid PEM"
+			ret.Valid = false
+		} else {
+			cert, err := x509.ParseCertificate(block.Bytes)
+			if err != nil {
+				ret.Error = fmt.Sprintf("invalid certificate: %v", err)
+				ret.Valid = false
+			} else {
+				ret.NotBefore = cert.NotBefore.UTC().Format(time.RFC3339)
+				ret.NotAfter = cert.NotAfter.UTC().Format(time.RFC3339)
+			}
+		}
+	}
+
+	writeJSON(w, ret)
+}
+
+func writeJSON(w http.ResponseWriter, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	json.NewEncoder(w).Encode(v)
+}

feature/acme/c2n_test.go

@@ -0,0 +1,135 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build !ios && !android && !js
+
+package acme
+
+import (
+	"cmp"
+	"crypto/x509"
+	"encoding/json"
+	"net/http/httptest"
+	"net/url"
+	"os"
+	"path/filepath"
+	"reflect"
+	"testing"
+	"time"
+
+	"tailscale.com/ipn/ipnlocal/ipnlocaltest"
+	"tailscale.com/tailcfg"
+	"tailscale.com/tstest"
+	"tailscale.com/types/logger"
+	"tailscale.com/util/must"
+)
+
+func TestHandleC2NTLSCertStatus(t *testing.T) {
+	b := ipnlocaltest.NewBackend(t)
+	b.SetVarRoot(t.TempDir())
+	e := extOf(t, b)
+
+	certDirPath, err := certDir(b)
+	if err != nil {
+		t.Fatalf("certDir error: %v", err)
+	}
+	if _, err := e.getCertStore(b); err != nil {
+		t.Fatalf("getCertStore error: %v", err)
+	}
+
+	testRoot, err := certTestFS.ReadFile("testdata/rootCA.pem")
+	if err != nil {
+		t.Fatal(err)
+	}
+	roots := x509.NewCertPool()
+	if !roots.AppendCertsFromPEM(testRoot) {
+		t.Fatal("Unable to add test CA to the cert pool")
+	}
+	testX509Roots = roots
+	defer func() { testX509Roots = nil }()
+
+	tests := []struct {
+		name       string
+		domain     string
+		copyFile   bool   // copy testdata/example.com.pem to the certDir
+		wantStatus int    // 0 means 200
+		wantError  string // wanted non-JSON non-200 error
+		now        time.Time
+		want       *tailcfg.C2NTLSCertInfo
+	}{
+		{
+			name:       "no-domain",
+			wantStatus: 400,
+			wantError:  "no 'domain'\n",
+		},
+		{
+			name:   "missing",
+			domain: "example.com",
+			want: &tailcfg.C2NTLSCertInfo{
+				Error:   "no certificate",
+				Missing: true,
+			},
+		},
+		{
+			name:     "valid",
+			domain:   "example.com",
+			now:      time.Date(2023, time.February, 20, 0, 0, 0, 0, time.UTC),
+			copyFile: true,
+			want: &tailcfg.C2NTLSCertInfo{
+				Valid:     true,
+				NotBefore: "2023-02-07T20:34:18Z",
+				NotAfter:  "2025-05-07T19:34:18Z",
+			},
+		},
+		{
+			name:     "expired",
+			domain:   "example.com",
+			now:      time.Date(2030, time.February, 20, 0, 0, 0, 0, time.UTC),
+			copyFile: true,
+			want: &tailcfg.C2NTLSCertInfo{
+				Error:   "cert expired",
+				Expired: true,
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			os.RemoveAll(certDirPath) // reset per test
+			if tt.copyFile {
+				os.MkdirAll(certDirPath, 0755)
+				if err := os.WriteFile(filepath.Join(certDirPath, "example.com.crt"),
+					must.Get(certTestFS.ReadFile("testdata/example.com.pem")), 0644); err != nil {
+					t.Fatal(err)
+				}
+				if err := os.WriteFile(filepath.Join(certDirPath, "example.com.key"),
+					must.Get(certTestFS.ReadFile("testdata/example.com-key.pem")), 0644); err != nil {
+					t.Fatal(err)
+				}
+			}
+			b.ForTest().SetClock(tstest.NewClock(tstest.ClockOpts{
+				Start: tt.now,
+			}))
+
+			rec := httptest.NewRecorder()
+			e.handleC2NTLSCertStatus(b, rec, httptest.NewRequest("GET", "/tls-cert-status?domain="+url.QueryEscape(tt.domain), nil))
+			res := rec.Result()
+			wantStatus := cmp.Or(tt.wantStatus, 200)
+			if res.StatusCode != wantStatus {
+				t.Fatalf("status code = %v; want %v. Body: %s", res.Status, wantStatus, rec.Body.Bytes())
+			}
+			if wantStatus == 200 {
+				var got tailcfg.C2NTLSCertInfo
+				if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
+					t.Fatalf("bad JSON: %v", err)
+				}
+				if !reflect.DeepEqual(&got, tt.want) {
+					t.Errorf("got %v; want %v", logger.AsJSON(got), logger.AsJSON(tt.want))
+				}
+			} else if tt.wantError != "" {
+				if got := rec.Body.String(); got != tt.wantError {
+					t.Errorf("body = %q; want %q", got, tt.wantError)
+				}
+			}
+		})
+	}
+}

feature/acme/cert.go

@@ -0,0 +1,706 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+package acme
+
+import (
+	"bytes"
+	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/json"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"log"
+	randv2 "math/rand/v2"
+	"net"
+	"slices"
+	"strings"
+	"time"
+
+	"tailscale.com/envknob"
+	"tailscale.com/health"
+	"tailscale.com/ipn"
+	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/tailcfg"
+	"tailscale.com/tempfork/acme"
+	"tailscale.com/types/logger"
+	"tailscale.com/util/mak"
+	"tailscale.com/util/set"
+	"tailscale.com/util/slicesx"
+	"tailscale.com/util/testenv"
+)
+
+type acmeChallengeType string
+
+const (
+	acmeChallengeDNS01     acmeChallengeType = "dns-01"
+	acmeChallengeTLSALPN01 acmeChallengeType = "tls-alpn-01"
+)
+
+var acmeDebug = envknob.RegisterBool("TS_DEBUG_ACME")
+
+// getACMETLSALPNCert returns the short-lived ACME challenge certificate
+// for hi.ServerName. The ok result reports whether hi offered acme-tls/1
+// and an ACME order is actively waiting on that challenge for
+// hi.ServerName.
+func (e *extension) getACMETLSALPNCert(hi *tls.ClientHelloInfo) (cert *tls.Certificate, ok bool) {
+	if hi == nil || hi.ServerName == "" || !slices.Contains(hi.SupportedProtos, acme.ALPNProto) {
+		return nil, false
+	}
+	cert, ok = e.pendingACMETLSALPNCerts.Load(hi.ServerName)
+	return cert, ok
+}
+
+// getACMETLSALPNProto reports whether serveTLSConfig should advertise
+// an ACME ALPN protocol for this ClientHello.
+func (e *extension) getACMETLSALPNProto(hi *tls.ClientHelloInfo) (proto string, ok bool) {
+	if _, ok := e.getACMETLSALPNCert(hi); !ok {
+		return "", false
+	}
+	return acme.ALPNProto, true
+}
+
+// storeACMETLSALPNCert publishes cert to Serve TLS handshakes for domain
+// until the returned cleanup function is called.
+func (e *extension) storeACMETLSALPNCert(domain string, cert *tls.Certificate) (cleanup func()) {
+	e.pendingACMETLSALPNCerts.Store(domain, cert)
+	return func() {
+		e.pendingACMETLSALPNCerts.Delete(domain)
+	}
+}
+
+// getCertPEMWithValidity gets the TLSCertKeyPair for domain, either
+// from cache or via ACME. ACME is used for new domain certs, existing
+// expired certs, or existing certs that should be renewed sooner than
+// minValidity.
+func (e *extension) getCertPEMWithValidity(ctx context.Context, b *ipnlocal.LocalBackend, domain string, minValidity time.Duration) (*ipnlocal.TLSCertKeyPair, error) {
+	e.mu.Lock()
+	getCertForTest := e.getCertForTest
+	e.mu.Unlock()
+
+	if getCertForTest != nil {
+		testenv.AssertInTest()
+		return getCertForTest(domain)
+	}
+
+	if !validLookingCertDomain(domain) {
+		return nil, errors.New("invalid domain")
+	}
+
+	certDomain, err := e.resolveCertDomain(b, domain)
+	if err != nil {
+		return nil, err
+	}
+	logf := logger.WithPrefix(b.Logger(), fmt.Sprintf("cert(%q): ", domain))
+	now := b.Clock().Now()
+	traceACME := func(v any) {
+		if !acmeDebug() {
+			return
+		}
+		j, _ := json.MarshalIndent(v, "", "\t")
+		log.Printf("acme %T: %s", v, j)
+	}
+
+	cs, err := e.getCertStore(b)
+	if err != nil {
+		return nil, err
+	}
+
+	pair, cacheErr := getCertPEMCached(cs, certDomain, now)
+	if cacheErr == nil {
+		if envknob.IsCertShareReadOnlyMode() {
+			return pair, nil
+		}
+		// If we got here, we have a valid unexpired cert.
+		// Check whether we should start an async renewal.
+		shouldRenew, err := e.shouldStartDomainRenewal(b, cs, certDomain, now, pair, minValidity)
+		if err != nil {
+			logf("error checking for certificate renewal: %v", err)
+			// Renewal check failed, but the current cert is valid and not
+			// expired, so it's safe to return.
+			return pair, nil
+		}
+		if !shouldRenew {
+			return pair, nil
+		}
+		if minValidity == 0 {
+			logf("starting async renewal")
+			// Start renewal in the background, return current valid cert.
+			e.Go(func() {
+				if _, err := getCertPEM(context.Background(), e, b, cs, logf, traceACME, certDomain, now, minValidity); err != nil {
+					logf("async renewal failed: getCertPem: %v", err)
+				}
+			})
+			return pair, nil
+		}
+		// If the caller requested a specific validity duration, fall through
+		// to synchronous renewal to fulfill that.
+		logf("starting sync renewal")
+	}
+
+	if envknob.IsCertShareReadOnlyMode() {
+		return nil, fmt.Errorf("retrieving cached TLS certificate failed and cert store is configured in read-only mode, not attempting to issue a new certificate: %w", cacheErr)
+	}
+
+	pair, err = getCertPEM(ctx, e, b, cs, logf, traceACME, certDomain, now, minValidity)
+	if err != nil {
+		logf("getCertPEM: %v", err)
+		return nil, err
+	}
+	return pair, nil
+}
+
+// shouldStartDomainRenewal reports whether the domain's cert should be
+// renewed based on the current time, the cert's expiry, and the ARI
+// check.
+func (e *extension) shouldStartDomainRenewal(b *ipnlocal.LocalBackend, cs certStore, domain string, now time.Time, pair *ipnlocal.TLSCertKeyPair, minValidity time.Duration) (bool, error) {
+	if minValidity != 0 {
+		cert, err := parseCertificate(pair)
+		if err != nil {
+			return false, fmt.Errorf("parsing certificate: %w", err)
+		}
+		return cert.NotAfter.Sub(now) < minValidity, nil
+	}
+	e.renewMu.Lock()
+	defer e.renewMu.Unlock()
+	if renewAt, ok := e.renewCertAt[domain]; ok {
+		return now.After(renewAt), nil
+	}
+
+	renewTime, err := e.domainRenewalTimeByARI(b, cs, pair)
+	if err != nil {
+		// Log any ARI failure and fall back to checking for renewal by expiry.
+		b.Logger()("acme: ARI check failed: %v; falling back to expiry-based check", err)
+		renewTime, err = domainRenewalTimeByExpiry(pair)
+		if err != nil {
+			return false, err
+		}
+	}
+
+	mak.Set(&e.renewCertAt, domain, renewTime)
+	return now.After(renewTime), nil
+}
+
+func (e *extension) domainRenewed(domain string) {
+	e.renewMu.Lock()
+	defer e.renewMu.Unlock()
+	delete(e.renewCertAt, domain)
+}
+
+func domainRenewalTimeByExpiry(pair *ipnlocal.TLSCertKeyPair) (time.Time, error) {
+	cert, err := parseCertificate(pair)
+	if err != nil {
+		return time.Time{}, fmt.Errorf("parsing certificate: %w", err)
+	}
+
+	certLifetime := cert.NotAfter.Sub(cert.NotBefore)
+	if certLifetime < 0 {
+		return time.Time{}, fmt.Errorf("negative certificate lifetime %v", certLifetime)
+	}
+
+	// Per https://github.com/tailscale/tailscale/issues/8204, check
+	// whether we're more than 2/3 of the way through the certificate's
+	// lifetime, which is the officially-recommended best practice by Let's
+	// Encrypt.
+	renewalDuration := certLifetime * 2 / 3
+	renewAt := cert.NotBefore.Add(renewalDuration)
+	return renewAt, nil
+}
+
+func (e *extension) shouldUseACMETLSALPN01(b *ipnlocal.LocalBackend, domain string, previous *ipnlocal.TLSCertKeyPair, logf logger.Logf) bool {
+	if isWildcardDomain(domain) {
+		logf("acme: using dns-01: tls-alpn-01 does not support wildcard certificates")
+		return false
+	}
+	if !b.HasFunnelForHostPort(domain, 443) {
+		logf("acme: using dns-01: Funnel is not enabled for %s:443", domain)
+		return false
+	}
+	if e.isBYOFunnelDomain(b, domain) {
+		// BYO Funnel domain: dns-01 is not a viable path because control
+		// does not own the user's DNS zone. Use tls-alpn-01 even on
+		// first issuance.
+		logf("acme: using tls-alpn-01 (BYO Funnel domain)")
+		return true
+	}
+	if previous == nil {
+		logf("acme: using dns-01: no cached certificate for Funnel renewal")
+		return false
+	}
+	logf("acme: using tls-alpn-01")
+	return true
+}
+
+// isBYOFunnelDomain reports whether domain is a "bring your own" Funnel
+// hostname: a domain that is not in the netmap's CertDomains but is
+// referenced as a Funnel target on :443 by the local serve config.
+// BYO domains can only be issued via tls-alpn-01 because control does
+// not own their DNS zone.
+func (e *extension) isBYOFunnelDomain(b *ipnlocal.LocalBackend, domain string) bool {
+	if domain == "" || isWildcardDomain(domain) {
+		return false
+	}
+	nm := b.NetMapNoPeers()
+	if nm != nil && slices.Contains(nm.DNS.CertDomains, domain) {
+		return false
+	}
+	return b.HasFunnelForHostPort(domain, 443)
+}
+
+func challengeByType(challenges []*acme.Challenge, typ string) *acme.Challenge {
+	for _, ch := range challenges {
+		if ch.Type == typ {
+			return ch
+		}
+	}
+	return nil
+}
+
+func isWildcardDomain(domain string) bool {
+	return strings.HasPrefix(domain, "*.")
+}
+
+func (e *extension) domainRenewalTimeByARI(b *ipnlocal.LocalBackend, cs certStore, pair *ipnlocal.TLSCertKeyPair) (time.Time, error) {
+	var blocks []*pem.Block
+	rest := pair.CertPEM
+	for len(rest) > 0 {
+		var block *pem.Block
+		block, rest = pem.Decode(rest)
+		if block == nil {
+			return time.Time{}, fmt.Errorf("parsing certificate PEM")
+		}
+		blocks = append(blocks, block)
+	}
+	if len(blocks) < 1 {
+		return time.Time{}, fmt.Errorf("could not parse certificate chain from certStore, got %d PEM block(s)", len(blocks))
+	}
+	ac, err := acmeClient(cs)
+	if err != nil {
+		return time.Time{}, err
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	ri, err := ac.FetchRenewalInfo(ctx, blocks[0].Bytes)
+	if err != nil {
+		return time.Time{}, fmt.Errorf("failed to fetch renewal info from ACME server: %w", err)
+	}
+	if acmeDebug() {
+		b.Logger()("acme: ARI response: %+v", ri)
+	}
+
+	// Select a random time in the suggested window and renew if that time has
+	// passed. Time is randomized per recommendation in
+	// https://datatracker.ietf.org/doc/draft-ietf-acme-ari/
+	start, end := ri.SuggestedWindow.Start, ri.SuggestedWindow.End
+	renewTime := start.Add(randv2.N(end.Sub(start)))
+	return renewTime, nil
+}
+
+// getCertPEM checks if a cert needs to be renewed and if so, renews it.
+// domain is the resolved cert domain (e.g., "*.node.ts.net" for
+// wildcards). It can be overridden in tests.
+var getCertPEM = func(ctx context.Context, e *extension, b *ipnlocal.LocalBackend, cs certStore, logf logger.Logf, traceACME func(any), domain string, now time.Time, minValidity time.Duration) (*ipnlocal.TLSCertKeyPair, error) {
+	e.acmeMu.Lock()
+	defer e.acmeMu.Unlock()
+
+	// In case this method was triggered multiple times in parallel (when
+	// serving incoming requests), check whether one of the other goroutines
+	// already renewed the cert before us.
+	previous, err := getCertPEMCached(cs, domain, now)
+	if err == nil {
+		// shouldStartDomainRenewal caches its result so it's OK to call this
+		// frequently.
+		shouldRenew, err := e.shouldStartDomainRenewal(b, cs, domain, now, previous, minValidity)
+		if err != nil {
+			logf("error checking for certificate renewal: %v", err)
+		} else if !shouldRenew {
+			return previous, nil
+		}
+	} else if !errors.Is(err, ipn.ErrStateNotExist) && !errors.Is(err, errCertExpired) {
+		return nil, err
+	}
+
+	// If we have no usable cached cert (either nothing on disk, or what is
+	// on disk has expired or otherwise failed verification), surface a
+	// health warning to the user for the duration of the ACME flow. We
+	// don't fire the warning when previous is non-nil because then we have
+	// a working cert and the renewal is happening behind the scenes.
+	if previous == nil {
+		e.setCertPending(b, domain, true)
+		defer e.setCertPending(b, domain, false)
+	}
+
+	ac, err := acmeClient(cs)
+	if err != nil {
+		return nil, err
+	}
+
+	if !isDefaultDirectoryURL(ac.DirectoryURL) {
+		logf("acme: using Directory URL %q", ac.DirectoryURL)
+	}
+
+	a, err := ac.GetReg(ctx, "" /* pre-RFC param */)
+	switch {
+	case err == nil:
+		// Great, already registered.
+		logf("already had ACME account.")
+	case err == acme.ErrNoAccount:
+		a, err = ac.Register(ctx, new(acme.Account), acme.AcceptTOS)
+		if err == acme.ErrAccountAlreadyExists {
+			// Potential race. Double check.
+			a, err = ac.GetReg(ctx, "" /* pre-RFC param */)
+		}
+		if err != nil {
+			return nil, fmt.Errorf("acme.Register: %w", err)
+		}
+		logf("registered ACME account.")
+		traceACME(a)
+	default:
+		return nil, fmt.Errorf("acme.GetReg: %w", err)
+
+	}
+	if a.Status != acme.StatusValid {
+		return nil, fmt.Errorf("unexpected ACME account status %q", a.Status)
+	}
+
+	// If we have a previous cert, include it in the order. Assuming we're
+	// within the ARI renewal window this should exclude us from LE rate
+	// limits.
+	// Note that this order extension will fail renewals if the ACME account key has changed
+	// since the last issuance, see
+	// https://github.com/tailscale/tailscale/issues/18251
+	var opts []acme.OrderOption
+	if previous != nil && !envknob.Bool("TS_DEBUG_ACME_FORCE_RENEWAL") {
+		prevCrt, err := parseCertificate(previous)
+		if err == nil {
+			opts = append(opts, acme.WithOrderReplacesCert(prevCrt))
+		}
+	}
+
+	issueArgs := acmeCertIssueArgs{
+		cs:        cs,
+		logf:      logf,
+		traceACME: traceACME,
+		domain:    domain,
+		opts:      opts,
+	}
+	if e.shouldUseACMETLSALPN01(b, domain, previous, logf) {
+		issueArgs.challengeType = acmeChallengeTLSALPN01
+		pair, err := e.issueACMECert(ctx, b, ac, issueArgs)
+		if err == nil {
+			return pair, nil
+		}
+		if ctx.Err() != nil {
+			return nil, ctx.Err()
+		}
+		if e.isBYOFunnelDomain(b, domain) {
+			// BYO domains have no working dns-01 path (control does not
+			// own the zone), so surface the tls-alpn-01 error instead of
+			// burning an ACME attempt on a guaranteed-to-fail fallback.
+			return nil, err
+		}
+		logf("acme: tls-alpn-01 failed; falling back to dns-01: %v", err)
+	}
+	issueArgs.challengeType = acmeChallengeDNS01
+	return e.issueACMECert(ctx, b, ac, issueArgs)
+}
+
+type acmeCertIssueArgs struct {
+	cs            certStore          // certificate and ACME account storage
+	logf          logger.Logf        // logs ACME progress and failures
+	traceACME     func(any)          // optional hook for logging ACME messages
+	domain        string             // certificate domain being issued
+	opts          []acme.OrderOption // ACME order options
+	challengeType acmeChallengeType  // challenge type to fulfill
+}
+
+func (args acmeCertIssueArgs) baseDomain() string { return strings.TrimPrefix(args.domain, "*.") }
+func (args acmeCertIssueArgs) isWildcard() bool   { return isWildcardDomain(args.domain) }
+
+func (e *extension) issueACMECert(ctx context.Context, b *ipnlocal.LocalBackend, ac *acme.Client, args acmeCertIssueArgs) (ret *ipnlocal.TLSCertKeyPair, err error) {
+	if args.traceACME == nil {
+		args.traceACME = func(any) {}
+	}
+
+	switch args.challengeType {
+	case acmeChallengeTLSALPN01:
+		metricACMETLSALPN01Start.Add(1)
+		defer func() {
+			if err == nil {
+				metricACMETLSALPN01Success.Add(1)
+			} else {
+				metricACMETLSALPN01Failure.Add(1)
+			}
+		}()
+	case acmeChallengeDNS01:
+		metricACMEDNS01Start.Add(1)
+		defer func() {
+			if err == nil {
+				metricACMEDNS01Success.Add(1)
+			} else {
+				metricACMEDNS01Failure.Add(1)
+			}
+		}()
+	default:
+		return nil, fmt.Errorf("unknown ACME challenge type %q", args.challengeType)
+	}
+
+	// For wildcards, we need to authorize both the wildcard and base domain.
+	var authzIDs []acme.AuthzID
+	if args.isWildcard() {
+		authzIDs = []acme.AuthzID{
+			{Type: "dns", Value: args.domain},
+			{Type: "dns", Value: args.baseDomain()},
+		}
+	} else {
+		authzIDs = []acme.AuthzID{{Type: "dns", Value: args.domain}}
+	}
+	order, err := ac.AuthorizeOrder(ctx, authzIDs, args.opts...)
+	if err != nil {
+		return nil, err
+	}
+	args.traceACME(order)
+
+	for _, aurl := range order.AuthzURLs {
+		az, err := ac.GetAuthorization(ctx, aurl)
+		if err != nil {
+			return nil, err
+		}
+		args.traceACME(az)
+		switch args.challengeType {
+		case acmeChallengeTLSALPN01:
+			ch := challengeByType(az.Challenges, string(acmeChallengeTLSALPN01))
+			if ch == nil {
+				return nil, errors.New("tls-alpn-01 challenge not offered")
+			}
+			cert, err := ac.TLSALPN01ChallengeCert(ch.Token, az.Identifier.Value)
+			if err != nil {
+				return nil, fmt.Errorf("TLSALPN01ChallengeCert: %w", err)
+			}
+			cleanup := e.storeACMETLSALPNCert(az.Identifier.Value, &cert)
+			defer cleanup()
+			chal, err := ac.Accept(ctx, ch)
+			if err != nil {
+				return nil, fmt.Errorf("Accept: %v", err)
+			}
+			args.traceACME(chal)
+		case acmeChallengeDNS01:
+			if err := fulfillACMEDNS01Challenge(ctx, b, ac, az, args.logf, args.traceACME); err != nil {
+				return nil, err
+			}
+		default:
+			return nil, fmt.Errorf("unknown ACME challenge type %q", args.challengeType)
+		}
+	}
+
+	orderURI := order.URI
+	order, err = ac.WaitOrder(ctx, orderURI)
+	if err != nil {
+		if ctx.Err() != nil {
+			return nil, ctx.Err()
+		}
+		if oe, ok := err.(*acme.OrderError); ok {
+			args.logf("acme: WaitOrder: OrderError status %q", oe.Status)
+		} else {
+			args.logf("acme: WaitOrder error: %v", err)
+		}
+		return nil, err
+	}
+	args.traceACME(order)
+
+	certPrivKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, err
+	}
+	var privPEM bytes.Buffer
+	if err := encodeECDSAKey(&privPEM, certPrivKey); err != nil {
+		return nil, err
+	}
+
+	csr, err := certRequest(certPrivKey, args.domain, nil)
+	if err != nil {
+		return nil, err
+	}
+
+	args.logf("requesting cert...")
+	args.traceACME(csr)
+	der, _, err := ac.CreateOrderCert(ctx, order.FinalizeURL, csr, true)
+	if err != nil {
+		return nil, fmt.Errorf("CreateOrder: %v", err)
+	}
+	args.logf("got cert")
+
+	var certPEM bytes.Buffer
+	for _, b := range der {
+		pb := &pem.Block{Type: "CERTIFICATE", Bytes: b}
+		if err := pem.Encode(&certPEM, pb); err != nil {
+			return nil, err
+		}
+	}
+	if err := args.cs.WriteTLSCertAndKey(args.domain, certPEM.Bytes(), privPEM.Bytes()); err != nil {
+		return nil, err
+	}
+	e.domainRenewed(args.domain)
+
+	return &ipnlocal.TLSCertKeyPair{CertPEM: certPEM.Bytes(), KeyPEM: privPEM.Bytes()}, nil
+}
+
+func fulfillACMEDNS01Challenge(ctx context.Context, b *ipnlocal.LocalBackend, ac *acme.Client, az *acme.Authorization, logf logger.Logf, traceACME func(any)) error {
+	for _, ch := range az.Challenges {
+		if ch.Type != string(acmeChallengeDNS01) {
+			continue
+		}
+		rec, err := ac.DNS01ChallengeRecord(ch.Token)
+		if err != nil {
+			return err
+		}
+		// For wildcards, the challenge is on the base domain.
+		// e.g., "*.node.ts.net" -> "_acme-challenge.node.ts.net"
+		key := "_acme-challenge." + strings.TrimPrefix(az.Identifier.Value, "*.")
+
+		// Do a best-effort lookup to see if we've already created this DNS name
+		// in a previous attempt. Don't burn too much time on it, though. Worst
+		// case we ask the server to create something that already exists.
+		var resolver net.Resolver
+		lookupCtx, lookupCancel := context.WithTimeout(ctx, 500*time.Millisecond)
+		txts, _ := resolver.LookupTXT(lookupCtx, key)
+		lookupCancel()
+		if slices.Contains(txts, rec) {
+			logf("TXT record already existed for %s", key)
+		} else {
+			logf("starting SetDNS call for %s...", key)
+			err = b.SetDNS(ctx, key, rec)
+			if err != nil {
+				return fmt.Errorf("SetDNS %q => %q: %w", key, rec, err)
+			}
+			logf("did SetDNS for %s", key)
+		}
+
+		chal, err := ac.Accept(ctx, ch)
+		if err != nil {
+			return fmt.Errorf("Accept: %v", err)
+		}
+		traceACME(chal)
+		return nil
+	}
+	return errors.New("dns-01 challenge not offered")
+}
+
+// validLookingCertDomain reports whether name looks like a valid domain
+// name that we might be able to get a cert for.
+//
+// It's a light check primarily for double checking before it's used as
+// part of a filesystem path. The actual validation happens in
+// resolveCertDomain.
+func validLookingCertDomain(name string) bool {
+	if name == "" ||
+		strings.Contains(name, "..") ||
+		strings.ContainsAny(name, ":/\\\x00") ||
+		!strings.Contains(name, ".") {
+		return false
+	}
+	// Only allow * as a wildcard prefix "*.domain.tld"
+	if rest, ok := strings.CutPrefix(name, "*."); ok {
+		if strings.Contains(rest, "*") || !strings.Contains(rest, ".") {
+			return false
+		}
+	} else if strings.Contains(name, "*") {
+		return false
+	}
+	return true
+}
+
+// resolveCertDomain validates a domain and returns the cert domain to use.
+//
+//   - "node.ts.net" -> "node.ts.net" (exact CertDomain match)
+//   - "*.node.ts.net" -> "*.node.ts.net" (explicit wildcard, requires NodeAttrDNSSubdomainResolve)
+//   - "foo.com" -> "foo.com" (bring-your-own Funnel domain referenced by the
+//     local serve config; issued via tls-alpn-01 in getCertPEM)
+//
+// Subdomain requests like "app.node.ts.net" are rejected; callers should
+// request "*.node.ts.net" explicitly for subdomain coverage.
+func (e *extension) resolveCertDomain(b *ipnlocal.LocalBackend, domain string) (string, error) {
+	if domain == "" {
+		return "", errors.New("missing domain name")
+	}
+
+	// Read the netmap once to get both CertDomains and capabilities atomically.
+	nm := b.NetMapNoPeers()
+	if nm == nil {
+		return "", errors.New("no netmap available")
+	}
+	certDomains := nm.DNS.CertDomains
+	if len(certDomains) == 0 && !e.isBYOFunnelDomain(b, domain) {
+		return "", errors.New("your Tailscale account does not support getting TLS certs")
+	}
+
+	// Wildcard request like "*.node.ts.net".
+	if base, ok := strings.CutPrefix(domain, "*."); ok {
+		if !nm.AllCaps.Contains(tailcfg.NodeAttrDNSSubdomainResolve) {
+			return "", fmt.Errorf("wildcard certificates are not enabled for this node")
+		}
+		if !slices.Contains(certDomains, base) {
+			return "", fmt.Errorf("invalid domain %q; wildcard certificates are not enabled for this domain", domain)
+		}
+		return domain, nil
+	}
+
+	// Exact CertDomain match.
+	if slices.Contains(certDomains, domain) {
+		return domain, nil
+	}
+
+	// Bring-your-own Funnel domain (e.g. "foo.com"). The serve config
+	// references the domain as a Funnel target on :443; cert acquisition
+	// happens via tls-alpn-01 in getCertPEM.
+	if e.isBYOFunnelDomain(b, domain) {
+		return domain, nil
+	}
+
+	return "", fmt.Errorf("invalid domain %q; must be one of %q", domain, certDomains)
+}
+
+// setCertPending sets or clears the in-flight ACME issuance state for
+// domain and updates the [certPendingWarnable] to reflect the current
+// set of pending domains.
+func (e *extension) setCertPending(b *ipnlocal.LocalBackend, domain string, pending bool) {
+	e.pendingCertDomainsMu.Lock()
+	defer e.pendingCertDomainsMu.Unlock()
+	if pending {
+		e.pendingCertDomains.Make()
+		e.pendingCertDomains.Add(domain)
+	} else {
+		e.pendingCertDomains.Delete(domain)
+	}
+	if e.pendingCertDomains.Len() == 0 {
+		b.HealthTracker().SetHealthy(certPendingWarnable)
+		return
+	}
+	b.HealthTracker().SetUnhealthy(certPendingWarnable, health.Args{
+		health.ArgDomains: joinedPendingCertDomainsLocked(e.pendingCertDomains),
+	})
+}
+
+func joinedPendingCertDomainsLocked(s set.Set[string]) string {
+	ds := slicesx.MapKeys(s)
+	slices.Sort(ds)
+	return strings.Join(ds, ", ")
+}
+
+// parseCertificate returns the leaf certificate from the given
+// TLSCertKeyPair's CertPEM.
+func parseCertificate(kp *ipnlocal.TLSCertKeyPair) (*x509.Certificate, error) {
+	block, _ := pem.Decode(kp.CertPEM)
+	if block == nil {
+		return nil, fmt.Errorf("error parsing certificate PEM")
+	}
+	if block.Type != "CERTIFICATE" {
+		return nil, fmt.Errorf("PEM block is %q, not a CERTIFICATE", block.Type)
+	}
+	return x509.ParseCertificate(block.Bytes)
+}

feature/acme/cert_test.go

@@ -3,7 +3,7 @@
 
 //go:build !ios && !android && !js
 
-package ipnlocal
+package acme
 
 import (
 	"context"
@@ -27,6 +27,8 @@ import (
 	"tailscale.com/envknob"
 	"tailscale.com/health"
 	"tailscale.com/ipn"
+	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/ipn/ipnlocal/ipnlocaltest"
 	"tailscale.com/ipn/store/mem"
 	"tailscale.com/tailcfg"
 	"tailscale.com/tempfork/acme"
@@ -38,6 +40,20 @@ import (
 	"tailscale.com/util/set"
 )
 
+//go:embed testdata/*
+var certTestFS embed.FS
+
+// extOf returns the [*extension] registered on b, failing the test if
+// it's not present.
+func extOf(t *testing.T, b *ipnlocal.LocalBackend) *extension {
+	t.Helper()
+	e, ok := ipnlocal.GetExt[*extension](b)
+	if !ok {
+		t.Fatal("acme extension not registered on backend")
+	}
+	return e
+}
+
 func TestCertRequest(t *testing.T) {
 	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
@@ -120,62 +136,46 @@ func TestResolveCertDomain(t *testing.T) {
 			name:        "wildcard_without_cap",
 			domain:      "*.node.ts.net",
 			certDomains: []string{"node.ts.net"},
-			hasCap:      false,
 			wantErr:     "wildcard certificates are not enabled for this node",
 		},
 		{
-			name:        "subdomain_with_cap_rejected",
+			name:        "wildcard_wrong_domain_with_cap",
-			domain:      "app.node.ts.net",
+			domain:      "*.other.com",
 			certDomains: []string{"node.ts.net"},
 			hasCap:      true,
-			wantErr:     `invalid domain "app.node.ts.net"; must be one of ["node.ts.net"]`,
+			wantErr:     `invalid domain "*.other.com"; wildcard certificates are not enabled for this domain`,
 		},
 		{
-			name:        "subdomain_without_cap_rejected",
+			name:        "missing_domain",
-			domain:      "app.node.ts.net",
+			domain:      "",
 			certDomains: []string{"node.ts.net"},
-			hasCap:      false,
+			wantErr:     "missing domain name",
-			wantErr:     `invalid domain "app.node.ts.net"; must be one of ["node.ts.net"]`,
 		},
 		{
-			name:        "multi_level_subdomain_rejected",
+			name:        "no_cert_domains_with_cap",
-			domain:      "a.b.node.ts.net",
+			domain:      "node.ts.net",
-			certDomains: []string{"node.ts.net"},
+			certDomains: nil,
 			hasCap:      true,
-			wantErr:     `invalid domain "a.b.node.ts.net"; must be one of ["node.ts.net"]`,
+			wantErr:     "your Tailscale account does not support getting TLS certs",
 		},
 		{
-			name:        "wildcard_no_matching_parent",
+			name:        "no_cert_domains_without_cap",
-			domain:      "*.unrelated.ts.net",
-			certDomains: []string{"node.ts.net"},
-			hasCap:      true,
-			wantErr:     `invalid domain "*.unrelated.ts.net"; wildcard certificates are not enabled for this domain`,
-		},
-		{
-			name:        "subdomain_unrelated_rejected",
-			domain:      "app.unrelated.ts.net",
-			certDomains: []string{"node.ts.net"},
-			hasCap:      true,
-			wantErr:     `invalid domain "app.unrelated.ts.net"; must be one of ["node.ts.net"]`,
-		},
-		{
-			name:        "no_cert_domains",
 			domain:      "node.ts.net",
 			certDomains: nil,
 			wantErr:     "your Tailscale account does not support getting TLS certs",
 		},
 		{
-			name:        "wildcard_no_cert_domains",
+			name:        "subdomain_request_rejected_without_cap",
-			domain:      "*.foo.ts.net",
+			domain:      "app.node.ts.net",
-			certDomains: nil,
+			certDomains: []string{"node.ts.net"},
-			hasCap:      true,
+			wantErr:     `invalid domain "app.node.ts.net"; must be one of ["node.ts.net"]`,
-			wantErr:     "your Tailscale account does not support getting TLS certs",
 		},
 		{
-			name:        "empty_domain",
+			name:        "subdomain_request_rejected_with_cap",
-			domain:      "",
+			domain:      "app.node.ts.net",
 			certDomains: []string{"node.ts.net"},
-			wantErr:     "missing domain name",
+			hasCap:      true,
+			wantErr:     `invalid domain "app.node.ts.net"; must be one of ["node.ts.net"]`,
 		},
 		{
 			name:       "nil_netmap",
@@ -187,26 +187,24 @@ func TestResolveCertDomain(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			b := newTestLocalBackend(t)
+			b := ipnlocaltest.NewBackend(t)
+			e := extOf(t, b)
 
 			if !tt.skipNetmap {
-				// Set up netmap with CertDomains and capability
 				var allCaps set.Set[tailcfg.NodeCapability]
 				if tt.hasCap {
 					allCaps = set.Of(tailcfg.NodeAttrDNSSubdomainResolve)
 				}
-				b.mu.Lock()
+				b.ForTest().SetNetMap(&netmap.NetworkMap{
-				b.currentNode().SetNetMap(&netmap.NetworkMap{
 					SelfNode: (&tailcfg.Node{}).View(),
 					DNS: tailcfg.DNSConfig{
 						CertDomains: tt.certDomains,
 					},
 					AllCaps: allCaps,
 				})
-				b.mu.Unlock()
 			}
 
-			got, err := b.resolveCertDomain(tt.domain)
+			got, err := e.resolveCertDomain(b, tt.domain)
 			if tt.wantErr != "" {
 				if err == nil {
 					t.Errorf("resolveCertDomain(%q) = %q, want error %q", tt.domain, got, tt.wantErr)
@@ -257,32 +255,33 @@ func TestValidLookingCertDomain(t *testing.T) {
 }
 
 func TestACMETLSALPNCertHook(t *testing.T) {
-	b := newTestLocalBackend(t)
+	b := ipnlocaltest.NewBackend(t)
+	e := extOf(t, b)
 	cert := &tls.Certificate{}
-	cleanup := b.storeACMETLSALPNCert("example.com", cert)
+	cleanup := e.storeACMETLSALPNCert("example.com", cert)
 	defer cleanup()
 
-	if got, ok := b.getACMETLSALPNCert(&tls.ClientHelloInfo{
+	if got, ok := b.ForTest().GetACMETLSALPNCert(&tls.ClientHelloInfo{
 		ServerName:      "example.com",
 		SupportedProtos: []string{acme.ALPNProto},
 	}); !ok || got != cert {
 		t.Fatalf("getACMETLSALPNCert = %v, %v; want stored cert, true", got, ok)
 	}
-	if _, ok := b.getACMETLSALPNCert(&tls.ClientHelloInfo{
+	if _, ok := b.ForTest().GetACMETLSALPNCert(&tls.ClientHelloInfo{
 		ServerName:      "example.com",
 		SupportedProtos: []string{"http/1.1"},
 	}); ok {
 		t.Fatal("getACMETLSALPNCert without acme ALPN = ok, want false")
 	}
-	if _, ok := b.getACMETLSALPNCert(&tls.ClientHelloInfo{
+	if _, ok := b.ForTest().GetACMETLSALPNCert(&tls.ClientHelloInfo{
 		ServerName:      "other.example.com",
 		SupportedProtos: []string{acme.ALPNProto},
 	}); ok {
 		t.Fatal("getACMETLSALPNCert for other name = ok, want false")
 	}
 
-	otherBackend := newTestLocalBackend(t)
+	otherBackend := ipnlocaltest.NewBackend(t)
-	if _, ok := otherBackend.getACMETLSALPNCert(&tls.ClientHelloInfo{
+	if _, ok := otherBackend.ForTest().GetACMETLSALPNCert(&tls.ClientHelloInfo{
 		ServerName:      "example.com",
 		SupportedProtos: []string{acme.ALPNProto},
 	}); ok {
@@ -290,52 +289,31 @@ func TestACMETLSALPNCertHook(t *testing.T) {
 	}
 }
 
-func TestServeTLSConfigNextProtos(t *testing.T) {
-	b := newTestLocalBackend(t)
-	getCert := func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
-		return nil, nil
-	}
-
-	httpsConfig := b.serveTLSConfig(getCert, serveTLSNextProtos())
-	if got, want := httpsConfig.NextProtos, []string{"h2", "http/1.1"}; !slices.Equal(got, want) {
-		t.Fatalf("HTTPS NextProtos = %q; want %q", got, want)
-	}
-
-	tcpForwardConfig := b.serveTLSConfig(getCert, nil)
-	if got := tcpForwardConfig.NextProtos; got != nil {
-		t.Fatalf("TLS-terminated TCP forward NextProtos = %q; want nil", got)
-	}
-}
-
 func TestShouldUseACMETLSALPN01(t *testing.T) {
 	const (
 		tsNetDomain = "node.ts.net"
 		byoDomain   = "foo.com"
 	)
-	previous := &TLSCertKeyPair{}
+	previous := &ipnlocal.TLSCertKeyPair{}
 
-	setFunnel := func(b *LocalBackend, hosts ...string) {
+	setFunnel := func(b *ipnlocal.LocalBackend, hosts ...string) {
 		funnel := map[ipn.HostPort]bool{}
 		for _, h := range hosts {
 			funnel[ipn.HostPort(h+":443")] = true
 		}
-		b.mu.Lock()
+		b.ForTest().SetServeConfig((&ipn.ServeConfig{AllowFunnel: funnel}).View())
-		b.serveConfig = (&ipn.ServeConfig{AllowFunnel: funnel}).View()
-		b.mu.Unlock()
 	}
-	setNetmap := func(b *LocalBackend, certDomains ...string) {
+	setNetmap := func(b *ipnlocal.LocalBackend, certDomains ...string) {
-		b.mu.Lock()
+		b.ForTest().SetNetMap(&netmap.NetworkMap{
-		b.currentNode().SetNetMap(&netmap.NetworkMap{
 			SelfNode: (&tailcfg.Node{}).View(),
 			DNS:      tailcfg.DNSConfig{CertDomains: certDomains},
 		})
-		b.mu.Unlock()
 	}
 
 	tests := []struct {
 		name     string
 		domain   string
-		previous *TLSCertKeyPair
+		previous *ipnlocal.TLSCertKeyPair
 		funnel   []string
 		netmap   []string // CertDomains; if nil, no netmap installed
 		want     bool
@@ -399,12 +377,13 @@ func TestShouldUseACMETLSALPN01(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			b := newTestLocalBackend(t)
+			b := ipnlocaltest.NewBackend(t)
+			e := extOf(t, b)
 			if tt.netmap != nil {
 				setNetmap(b, tt.netmap...)
 			}
 			setFunnel(b, tt.funnel...)
-			if got := b.shouldUseACMETLSALPN01(tt.domain, tt.previous, t.Logf); got != tt.want {
+			if got := e.shouldUseACMETLSALPN01(b, tt.domain, tt.previous, t.Logf); got != tt.want {
 				t.Errorf("shouldUseACMETLSALPN01(%q, previous=%v) = %v, want %v",
 					tt.domain, tt.previous != nil, got, tt.want)
 			}
@@ -413,22 +392,18 @@ func TestShouldUseACMETLSALPN01(t *testing.T) {
 }
 
 func TestIsBYOFunnelDomain(t *testing.T) {
-	setFunnel := func(b *LocalBackend, hosts ...string) {
+	setFunnel := func(b *ipnlocal.LocalBackend, hosts ...string) {
 		funnel := map[ipn.HostPort]bool{}
 		for _, h := range hosts {
 			funnel[ipn.HostPort(h+":443")] = true
 		}
-		b.mu.Lock()
+		b.ForTest().SetServeConfig((&ipn.ServeConfig{AllowFunnel: funnel}).View())
-		b.serveConfig = (&ipn.ServeConfig{AllowFunnel: funnel}).View()
-		b.mu.Unlock()
 	}
-	setNetmap := func(b *LocalBackend, certDomains ...string) {
+	setNetmap := func(b *ipnlocal.LocalBackend, certDomains ...string) {
-		b.mu.Lock()
+		b.ForTest().SetNetMap(&netmap.NetworkMap{
-		b.currentNode().SetNetMap(&netmap.NetworkMap{
 			SelfNode: (&tailcfg.Node{}).View(),
 			DNS:      tailcfg.DNSConfig{CertDomains: certDomains},
 		})
-		b.mu.Unlock()
 	}
 
 	tests := []struct {
@@ -446,10 +421,11 @@ func TestIsBYOFunnelDomain(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			b := newTestLocalBackend(t)
+			b := ipnlocaltest.NewBackend(t)
+			e := extOf(t, b)
 			setNetmap(b, tt.certDomains...)
 			setFunnel(b, tt.funnel...)
-			if got := b.isBYOFunnelDomain(tt.domain); got != tt.want {
+			if got := e.isBYOFunnelDomain(b, tt.domain); got != tt.want {
 				t.Errorf("isBYOFunnelDomain(%q) = %v, want %v", tt.domain, got, tt.want)
 			}
 		})
@@ -461,41 +437,36 @@ func TestResolveCertDomainBYO(t *testing.T) {
 		tsNetDomain = "node.ts.net"
 		byoDomain   = "foo.com"
 	)
-	b := newTestLocalBackend(t)
+	b := ipnlocaltest.NewBackend(t)
-	b.mu.Lock()
+	e := extOf(t, b)
-	b.currentNode().SetNetMap(&netmap.NetworkMap{
+	b.ForTest().SetNetMap(&netmap.NetworkMap{
 		SelfNode: (&tailcfg.Node{}).View(),
 		DNS:      tailcfg.DNSConfig{CertDomains: []string{tsNetDomain}},
 	})
-	b.mu.Unlock()
 
 	// Without a serve config, BYO is rejected.
-	if _, err := b.resolveCertDomain(byoDomain); err == nil {
+	if _, err := e.resolveCertDomain(b, byoDomain); err == nil {
 		t.Fatalf("resolveCertDomain(%q) without serve config: want error, got nil", byoDomain)
 	}
 
 	// Web entry alone (no AllowFunnel) is not enough; the gate is Funnel.
-	b.mu.Lock()
+	b.ForTest().SetServeConfig((&ipn.ServeConfig{
-	b.serveConfig = (&ipn.ServeConfig{
 		Web: map[ipn.HostPort]*ipn.WebServerConfig{
 			byoDomain + ":443": {Handlers: map[string]*ipn.HTTPHandler{"/": {Proxy: "http://127.0.0.1:8080"}}},
 		},
-	}).View()
+	}).View())
-	b.mu.Unlock()
+	if _, err := e.resolveCertDomain(b, byoDomain); err == nil {
-	if _, err := b.resolveCertDomain(byoDomain); err == nil {
 		t.Fatalf("resolveCertDomain(%q) with Web but no Funnel: want error, got nil", byoDomain)
 	}
 
 	// With AllowFunnel, BYO is accepted.
-	b.mu.Lock()
+	b.ForTest().SetServeConfig((&ipn.ServeConfig{
-	b.serveConfig = (&ipn.ServeConfig{
 		Web: map[ipn.HostPort]*ipn.WebServerConfig{
 			byoDomain + ":443": {Handlers: map[string]*ipn.HTTPHandler{"/": {Proxy: "http://127.0.0.1:8080"}}},
 		},
 		AllowFunnel: map[ipn.HostPort]bool{byoDomain + ":443": true},
-	}).View()
+	}).View())
-	b.mu.Unlock()
+	got, err := e.resolveCertDomain(b, byoDomain)
-	got, err := b.resolveCertDomain(byoDomain)
 	if err != nil {
 		t.Fatalf("resolveCertDomain(%q): %v", byoDomain, err)
 	}
@@ -504,7 +475,7 @@ func TestResolveCertDomainBYO(t *testing.T) {
 	}
 
 	// The ts.net path still works alongside BYO entries.
-	got, err = b.resolveCertDomain(tsNetDomain)
+	got, err = e.resolveCertDomain(b, tsNetDomain)
 	if err != nil {
 		t.Fatalf("resolveCertDomain(%q): %v", tsNetDomain, err)
 	}
@@ -513,9 +484,6 @@ func TestResolveCertDomainBYO(t *testing.T) {
 	}
 }
 
-//go:embed testdata/*
-var certTestFS embed.FS
-
 func TestCertStoreRoundTrip(t *testing.T) {
 	const testDomain = "example.com"
 
@@ -586,7 +554,7 @@ func TestCertStoreRoundTrip(t *testing.T) {
 }
 
 func TestShouldStartDomainRenewal(t *testing.T) {
-	mustMakePair := func(template *x509.Certificate) *TLSCertKeyPair {
+	mustMakePair := func(template *x509.Certificate) *ipnlocal.TLSCertKeyPair {
 		priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 		if err != nil {
 			panic(err)
@@ -601,7 +569,7 @@ func TestShouldStartDomainRenewal(t *testing.T) {
 			Bytes: b,
 		})
 
-		return &TLSCertKeyPair{
+		return &ipnlocal.TLSCertKeyPair{
 			Cached:  false,
 			CertPEM: certPEM,
 			KeyPEM:  []byte("unused"),
@@ -644,10 +612,9 @@ func TestShouldStartDomainRenewal(t *testing.T) {
 			want:      false,
 		},
 	}
-	b := new(LocalBackend)
 	for _, tt := range testCases {
 		t.Run(tt.name, func(t *testing.T) {
-			ret, err := b.domainRenewalTimeByExpiry(mustMakePair(&x509.Certificate{
+			ret, err := domainRenewalTimeByExpiry(mustMakePair(&x509.Certificate{
 				SerialNumber: big.NewInt(2019),
 				Subject:      subject,
 				NotBefore:    tt.notBefore,
@@ -688,24 +655,23 @@ func TestDebugACMEDirectoryURL(t *testing.T) {
 
 func TestGetCertPEMWithValidity(t *testing.T) {
 	const testDomain = "example.com"
-	b := newTestLocalBackend(t)
+	b := ipnlocaltest.NewBackend(t)
-	b.varRoot = t.TempDir()
+	b.SetVarRoot(t.TempDir())
+	e := extOf(t, b)
 
-	// Set up netmap with CertDomains so resolveCertDomain works
+	// Set up netmap with CertDomains so resolveCertDomain works.
-	b.mu.Lock()
+	b.ForTest().SetNetMap(&netmap.NetworkMap{
-	b.currentNode().SetNetMap(&netmap.NetworkMap{
 		SelfNode: (&tailcfg.Node{}).View(),
 		DNS: tailcfg.DNSConfig{
 			CertDomains: []string{testDomain},
 		},
 	})
-	b.mu.Unlock()
 
-	certDir, err := b.certDir()
+	certDirPath, err := certDir(b)
 	if err != nil {
 		t.Fatalf("certDir error: %v", err)
 	}
-	if _, err := b.getCertStore(); err != nil {
+	if _, err := e.getCertStore(b); err != nil {
 		t.Fatalf("getCertStore error: %v", err)
 	}
 	testRoot, err := certTestFS.ReadFile("testdata/rootCA.pem")
@@ -781,54 +747,46 @@ func TestGetCertPEMWithValidity(t *testing.T) {
 				envknob.Setenv("TS_CERT_SHARE_MODE", "")
 			}
 
-			os.RemoveAll(certDir)
+			os.RemoveAll(certDirPath)
 			if tt.storeCerts {
-				os.MkdirAll(certDir, 0755)
+				os.MkdirAll(certDirPath, 0755)
-				if err := os.WriteFile(filepath.Join(certDir, "example.com.crt"),
+				if err := os.WriteFile(filepath.Join(certDirPath, "example.com.crt"),
-					must.Get(os.ReadFile("testdata/example.com.pem")), 0644); err != nil {
+					must.Get(certTestFS.ReadFile("testdata/example.com.pem")), 0644); err != nil {
 					t.Fatal(err)
 				}
-				if err := os.WriteFile(filepath.Join(certDir, "example.com.key"),
+				if err := os.WriteFile(filepath.Join(certDirPath, "example.com.key"),
-					must.Get(os.ReadFile("testdata/example.com-key.pem")), 0644); err != nil {
+					must.Get(certTestFS.ReadFile("testdata/example.com-key.pem")), 0644); err != nil {
 					t.Fatal(err)
 				}
 			}
 
-			b.clock = tstest.NewClock(tstest.ClockOpts{Start: tt.now})
+			b.ForTest().SetClock(tstest.NewClock(tstest.ClockOpts{Start: tt.now}))
 
-			allDone := make(chan bool, 1)
+			// Set to true if getCertPEM is called. GetCertPEM can be called in
-			defer b.goTracker.AddDoneCallback(func() {
+			// a goroutine for async renewal or in the main goroutine if
-				b.mu.Lock()
+			// issuance is required to obtain valid TLS credentials.
-				defer b.mu.Unlock()
-				if b.goTracker.RunningGoroutines() > 0 {
-					return
-				}
-				select {
-				case allDone <- true:
-				default:
-				}
-			})()
-
-			// Set to true if get getCertPEM is called. GetCertPEM can be called in a goroutine for async
-			// renewal or in the main goroutine if issuance is required to obtain valid TLS credentials.
 			getCertPemWasCalled := false
-			getCertPEM = func(ctx context.Context, b *LocalBackend, cs certStore, logf logger.Logf, traceACME func(any), domain string, now time.Time, minValidity time.Duration) (*TLSCertKeyPair, error) {
+			orig := getCertPEM
+			getCertPEM = func(ctx context.Context, e *extension, b *ipnlocal.LocalBackend, cs certStore, logf logger.Logf, traceACME func(any), domain string, now time.Time, minValidity time.Duration) (*ipnlocal.TLSCertKeyPair, error) {
 				getCertPemWasCalled = true
 				return nil, nil
 			}
-			prevGoRoutines := b.goTracker.StartedGoroutines()
+			t.Cleanup(func() { getCertPEM = orig })
+			prevGo := e.goroutinesStarted.Load()
 			_, err = b.GetCertPEMWithValidity(context.Background(), testDomain, 0)
 			if (err != nil) != tt.wantErr {
 				t.Errorf("b.GetCertPemWithValidity got err %v, wants error: '%v'", err, tt.wantErr)
 			}
-			// GetCertPEMWithValidity calls getCertPEM in a goroutine if async renewal is needed. That's the
+			// GetCertPEMWithValidity spawns one tracked goroutine (via
-			// only goroutine it starts, so this can be used to test if async renewal was started.
+			// extension.Go) iff it kicked off async renewal.
-			gotAsyncRenewal := b.goTracker.StartedGoroutines()-prevGoRoutines != 0
+			gotAsyncRenewal := e.goroutinesStarted.Load()-prevGo != 0
 			if gotAsyncRenewal {
+				done := make(chan struct{})
+				go func() { e.wg.Wait(); close(done) }()
 				select {
 				case <-time.After(5 * time.Second):
-					t.Fatal("timed out waiting for goroutines to finish")
+					t.Fatal("timed out waiting for async renewal goroutine to finish")
-				case <-allDone:
+				case <-done:
 				}
 			}
 			// Verify that async renewal was triggered if expected.
@@ -845,24 +803,25 @@ func TestGetCertPEMWithValidity(t *testing.T) {
 }
 
 func TestCertPendingWarnable(t *testing.T) {
-	b := newTestLocalBackend(t)
+	b := ipnlocaltest.NewBackend(t)
+	e := extOf(t, b)
 
 	// currentWarning returns the pending warning's rendered text and
 	// domain-list arg, or "", "" if the warnable is currently healthy.
 	currentWarning := func() (text, domains string) {
-		ws, ok := b.health.CurrentState().Warnings[tsconst.HealthWarnableTLSCertPending]
+		ws, ok := b.HealthTracker().CurrentState().Warnings[tsconst.HealthWarnableTLSCertPending]
 		if !ok {
 			return "", ""
 		}
 		return ws.Text, ws.Args[health.ArgDomains]
 	}
 
-	if b.health.IsUnhealthy(certPendingWarnable) {
+	if b.HealthTracker().IsUnhealthy(certPendingWarnable) {
 		t.Fatal("warnable unexpectedly unhealthy before any setCertPending")
 	}
 
-	b.setCertPending("a.example.com", true)
+	e.setCertPending(b, "a.example.com", true)
-	if !b.health.IsUnhealthy(certPendingWarnable) {
+	if !b.HealthTracker().IsUnhealthy(certPendingWarnable) {
 		t.Fatal("warnable not unhealthy after first setCertPending")
 	}
 	if text, domains := currentWarning(); domains != "a.example.com" ||
@@ -870,8 +829,8 @@ func TestCertPendingWarnable(t *testing.T) {
 		t.Errorf("after first setCertPending: text=%q domains=%q", text, domains)
 	}
 
-	b.setCertPending("b.example.com", true)
+	e.setCertPending(b, "b.example.com", true)
-	if !b.health.IsUnhealthy(certPendingWarnable) {
+	if !b.HealthTracker().IsUnhealthy(certPendingWarnable) {
 		t.Fatal("warnable not unhealthy after second setCertPending")
 	}
 	if text, domains := currentWarning(); domains != "a.example.com, b.example.com" ||
@@ -879,8 +838,8 @@ func TestCertPendingWarnable(t *testing.T) {
 		t.Errorf("after second setCertPending: text=%q domains=%q", text, domains)
 	}
 
-	b.setCertPending("a.example.com", false)
+	e.setCertPending(b, "a.example.com", false)
-	if !b.health.IsUnhealthy(certPendingWarnable) {
+	if !b.HealthTracker().IsUnhealthy(certPendingWarnable) {
 		t.Fatal("warnable cleared too early; one domain still pending")
 	}
 	if text, domains := currentWarning(); domains != "b.example.com" ||
@@ -888,8 +847,8 @@ func TestCertPendingWarnable(t *testing.T) {
 		t.Errorf("after clearing a.example.com: text=%q domains=%q", text, domains)
 	}
 
-	b.setCertPending("b.example.com", false)
+	e.setCertPending(b, "b.example.com", false)
-	if b.health.IsUnhealthy(certPendingWarnable) {
+	if b.HealthTracker().IsUnhealthy(certPendingWarnable) {
 		t.Fatal("warnable still unhealthy after clearing all domains")
 	}
 	if text, domains := currentWarning(); text != "" || domains != "" {
@@ -962,17 +921,17 @@ func TestRefreshApplicableCerts(t *testing.T) {
 		certDomain = "node1.example.com"
 		byoDomain  = "byo.example.org"
 	)
-	b := newTestLocalBackend(t)
+	b := ipnlocaltest.NewBackend(t)
-	b.varRoot = t.TempDir()
+	b.SetVarRoot(t.TempDir())
+	e := extOf(t, b)
 
-	b.mu.Lock()
+	b.ForTest().SetNetMap(&netmap.NetworkMap{
-	b.currentNode().SetNetMap(&netmap.NetworkMap{
 		SelfNode: (&tailcfg.Node{}).View(),
 		DNS: tailcfg.DNSConfig{
 			CertDomains: []string{certDomain},
 		},
 	})
-	b.serveConfig = (&ipn.ServeConfig{
+	b.ForTest().SetServeConfig((&ipn.ServeConfig{
 		Web: map[ipn.HostPort]*ipn.WebServerConfig{
 			ipn.HostPort(certDomain + ":443"): {},
 			ipn.HostPort(byoDomain + ":443"):  {},
@@ -982,16 +941,15 @@ func TestRefreshApplicableCerts(t *testing.T) {
 		AllowFunnel: map[ipn.HostPort]bool{
 			ipn.HostPort(byoDomain + ":443"): true,
 		},
-	}).View()
+	}).View())
-	b.mu.Unlock()
 
 	gotCh := make(chan string, 4)
-	b.ForTest().ConfigureCerts(func(host string) (*TLSCertKeyPair, error) {
+	b.ForTest().ConfigureCerts(func(host string) (*ipnlocal.TLSCertKeyPair, error) {
 		gotCh <- host
-		return &TLSCertKeyPair{}, nil
+		return &ipnlocal.TLSCertKeyPair{}, nil
 	})
 
-	b.refreshApplicableCerts(context.Background())
+	e.refreshApplicableCerts(context.Background(), b)
 
 	want := set.Of(certDomain, byoDomain)
 	got := set.Set[string]{}

feature/acme/certstore.go

@@ -0,0 +1,444 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+package acme
+
+import (
+	"bytes"
+	"crypto"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"time"
+
+	"tailscale.com/atomicfile"
+	"tailscale.com/envknob"
+	"tailscale.com/feature/buildfeatures"
+	"tailscale.com/hostinfo"
+	"tailscale.com/ipn"
+	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/ipn/store"
+	"tailscale.com/ipn/store/mem"
+	"tailscale.com/net/bakedroots"
+	"tailscale.com/tempfork/acme"
+	"tailscale.com/util/testenv"
+	"tailscale.com/version"
+	"tailscale.com/version/distro"
+)
+
+// certStore provides a way to perist and retrieve TLS certificates.
+// As of 2023-02-01, we store certs in directories on disk everywhere
+// except on Kubernetes, where we use the state store.
+type certStore interface {
+	// Read returns the cert and key for domain, if they exist and are valid
+	// for now. If they're expired, it returns errCertExpired.
+	// If they don't exist, it returns ipn.ErrStateNotExist.
+	Read(domain string, now time.Time) (*ipnlocal.TLSCertKeyPair, error)
+	// ACMEKey returns the value previously stored via WriteACMEKey.
+	// It is a PEM encoded ECDSA key.
+	ACMEKey() ([]byte, error)
+	// WriteACMEKey stores the provided PEM encoded ECDSA key.
+	WriteACMEKey([]byte) error
+	// WriteTLSCertAndKey writes the cert and key for domain.
+	WriteTLSCertAndKey(domain string, cert, key []byte) error
+}
+
+var errCertExpired = errors.New("cert expired")
+
+var testX509Roots *x509.CertPool // set non-nil by tests
+
+// certDir returns (creating if needed) the directory in which cached
+// cert keypairs are stored.
+func certDir(b *ipnlocal.LocalBackend) (string, error) {
+	d := b.TailscaleVarRoot()
+
+	// As a workaround for Synology DSM6 not having a "var" directory, use the
+	// app's "etc" directory (on a small partition) to hold certs at least.
+	// See https://github.com/tailscale/tailscale/issues/4060#issuecomment-1186592251
+	if buildfeatures.HasSynology && d == "" && runtime.GOOS == "linux" && distro.Get() == distro.Synology && distro.DSMVersion() == 6 {
+		d = "/var/packages/Tailscale/etc" // base; we append "certs" below
+	}
+	if d == "" {
+		return "", errors.New("no TailscaleVarRoot")
+	}
+	full := filepath.Join(d, "certs")
+	if err := os.MkdirAll(full, 0700); err != nil {
+		return "", err
+	}
+	return full, nil
+}
+
+func (e *extension) getCertStore(b *ipnlocal.LocalBackend) (certStore, error) {
+	st := b.Sys().StateStore.Get()
+	switch st.(type) {
+	case *store.FileStore:
+	case *mem.Store:
+	default:
+		if hostinfo.GetEnvType() == hostinfo.Kubernetes {
+			// We're running in Kubernetes with a custom StateStore,
+			// use that instead of the cert directory.
+			// TODO(maisem): expand this to other environments?
+			return certStateStore{StateStore: st}, nil
+		}
+	}
+	dir, err := certDir(b)
+	if err != nil {
+		return nil, err
+	}
+	if testX509Roots != nil && !testenv.InTest() {
+		panic("use of test hook outside of tests")
+	}
+	return certFileStore{dir: dir, testRoots: testX509Roots}, nil
+}
+
+// certFileStore implements certStore by storing the cert & key files in
+// the named directory.
+type certFileStore struct {
+	dir string
+
+	// This field allows a test to override the CA root(s) for certificate
+	// verification. If nil the default system pool is used.
+	testRoots *x509.CertPool
+}
+
+const acmePEMName = "acme-account.key.pem"
+
+func (f certFileStore) ACMEKey() ([]byte, error) {
+	pemName := filepath.Join(f.dir, acmePEMName)
+	v, err := os.ReadFile(pemName)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, ipn.ErrStateNotExist
+		}
+		return nil, err
+	}
+	return v, nil
+}
+
+func (f certFileStore) WriteACMEKey(b []byte) error {
+	pemName := filepath.Join(f.dir, acmePEMName)
+	return atomicfile.WriteFile(pemName, b, 0600)
+}
+
+func (f certFileStore) Read(domain string, now time.Time) (*ipnlocal.TLSCertKeyPair, error) {
+	certPEM, err := os.ReadFile(certFile(f.dir, domain))
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, ipn.ErrStateNotExist
+		}
+		return nil, err
+	}
+	keyPEM, err := os.ReadFile(keyFile(f.dir, domain))
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, ipn.ErrStateNotExist
+		}
+		return nil, err
+	}
+	if !validCertPEM(domain, keyPEM, certPEM, f.testRoots, now) {
+		return nil, errCertExpired
+	}
+	return &ipnlocal.TLSCertKeyPair{CertPEM: certPEM, KeyPEM: keyPEM, Cached: true}, nil
+}
+
+func (f certFileStore) WriteCert(domain string, cert []byte) error {
+	return atomicfile.WriteFile(certFile(f.dir, domain), cert, 0644)
+}
+
+func (f certFileStore) WriteKey(domain string, key []byte) error {
+	return atomicfile.WriteFile(keyFile(f.dir, domain), key, 0600)
+}
+
+func (f certFileStore) WriteTLSCertAndKey(domain string, cert, key []byte) error {
+	if err := f.WriteKey(domain, key); err != nil {
+		return err
+	}
+	return f.WriteCert(domain, cert)
+}
+
+// certStateStore implements certStore by storing the cert & key files
+// in an ipn.StateStore.
+type certStateStore struct {
+	ipn.StateStore
+
+	// This field allows a test to override the CA root(s) for certificate
+	// verification. If nil the default system pool is used.
+	testRoots *x509.CertPool
+}
+
+// TLSCertKeyReader is an interface implemented by state stores where it
+// makes sense to read the TLS cert and key in a single operation that
+// can be distinguished from generic state value reads. Currently this
+// is only implemented by the kubestore.Store, which, in some cases,
+// needs to read cert and key from a non-cached TLS Secret.
+type TLSCertKeyReader interface {
+	ReadTLSCertAndKey(domain string) ([]byte, []byte, error)
+}
+
+func (s certStateStore) Read(domain string, now time.Time) (*ipnlocal.TLSCertKeyPair, error) {
+	// If we're using a store that supports atomic reads, use that
+	if kr, ok := s.StateStore.(TLSCertKeyReader); ok {
+		cert, key, err := kr.ReadTLSCertAndKey(domain)
+		if err != nil {
+			return nil, err
+		}
+		if !validCertPEM(domain, key, cert, s.testRoots, now) {
+			return nil, errCertExpired
+		}
+		return &ipnlocal.TLSCertKeyPair{CertPEM: cert, KeyPEM: key, Cached: true}, nil
+	}
+
+	// Otherwise fall back to separate reads
+	certPEM, err := s.ReadState(ipn.StateKey(domain + ".crt"))
+	if err != nil {
+		return nil, err
+	}
+	keyPEM, err := s.ReadState(ipn.StateKey(domain + ".key"))
+	if err != nil {
+		return nil, err
+	}
+	if !validCertPEM(domain, keyPEM, certPEM, s.testRoots, now) {
+		return nil, errCertExpired
+	}
+	return &ipnlocal.TLSCertKeyPair{CertPEM: certPEM, KeyPEM: keyPEM, Cached: true}, nil
+}
+
+func (s certStateStore) WriteCert(domain string, cert []byte) error {
+	return ipn.WriteState(s.StateStore, ipn.StateKey(domain+".crt"), cert)
+}
+
+func (s certStateStore) WriteKey(domain string, key []byte) error {
+	return ipn.WriteState(s.StateStore, ipn.StateKey(domain+".key"), key)
+}
+
+func (s certStateStore) ACMEKey() ([]byte, error) {
+	return s.ReadState(ipn.StateKey(acmePEMName))
+}
+
+func (s certStateStore) WriteACMEKey(key []byte) error {
+	return ipn.WriteState(s.StateStore, ipn.StateKey(acmePEMName), key)
+}
+
+// TLSCertKeyWriter is an interface implemented by state stores that can
+// write the TLS cert and key in a single atomic operation. Currently
+// this is only implemented by the kubestore.StoreKube.
+type TLSCertKeyWriter interface {
+	WriteTLSCertAndKey(domain string, cert, key []byte) error
+}
+
+// WriteTLSCertAndKey writes the TLS cert and key for domain to the
+// current LocalBackend's StateStore.
+func (s certStateStore) WriteTLSCertAndKey(domain string, cert, key []byte) error {
+	// If we're using a store that supports atomic writes, use that.
+	if aw, ok := s.StateStore.(TLSCertKeyWriter); ok {
+		return aw.WriteTLSCertAndKey(domain, cert, key)
+	}
+	// Otherwise fall back to separate writes for cert and key.
+	if err := s.WriteKey(domain, key); err != nil {
+		return err
+	}
+	return s.WriteCert(domain, cert)
+}
+
+func keyFile(dir, domain string) string {
+	return filepath.Join(dir, strings.Replace(domain, "*.", "wildcard_.", 1)+".key")
+}
+func certFile(dir, domain string) string {
+	return filepath.Join(dir, strings.Replace(domain, "*.", "wildcard_.", 1)+".crt")
+}
+
+// getCertPEMCached returns a non-nil keyPair if a cached keypair for
+// domain exists in the certStore that is valid at the provided now time.
+//
+// If the keypair is expired, it returns errCertExpired.
+// If the keypair doesn't exist, it returns ipn.ErrStateNotExist.
+func getCertPEMCached(cs certStore, domain string, now time.Time) (p *ipnlocal.TLSCertKeyPair, err error) {
+	if !validLookingCertDomain(domain) {
+		// Before we read files from disk using it, validate it's halfway
+		// reasonable looking.
+		return nil, fmt.Errorf("invalid domain %q", domain)
+	}
+	return cs.Read(domain, now)
+}
+
+// certRequest generates a CSR for the given domain and optional SANs.
+func certRequest(key crypto.Signer, domain string, ext []pkix.Extension) ([]byte, error) {
+	dnsNames := []string{domain}
+	if base, ok := strings.CutPrefix(domain, "*."); ok {
+		// Wildcard cert must also include the base domain as a SAN.
+		// This is load-bearing: getCertPEMCached validates certs using
+		// the storage key (base domain), which only passes x509 verification
+		// if the base domain is in DNSNames.
+		dnsNames = append(dnsNames, base)
+	}
+	req := &x509.CertificateRequest{
+		Subject:         pkix.Name{CommonName: domain},
+		DNSNames:        dnsNames,
+		ExtraExtensions: ext,
+	}
+	return x509.CreateCertificateRequest(rand.Reader, req, key)
+}
+
+func encodeECDSAKey(w io.Writer, key *ecdsa.PrivateKey) error {
+	b, err := x509.MarshalECPrivateKey(key)
+	if err != nil {
+		return err
+	}
+	pb := &pem.Block{Type: "EC PRIVATE KEY", Bytes: b}
+	return pem.Encode(w, pb)
+}
+
+// parsePrivateKey is a copy of x/crypto/acme's parsePrivateKey.
+//
+// Attempt to parse the given private key DER block. OpenSSL 0.9.8
+// generates PKCS#1 private keys by default, while OpenSSL 1.0.0
+// generates PKCS#8 keys. OpenSSL ecparam generates SEC1 EC private keys
+// for ECDSA. We try all three.
+//
+// Inspired by parsePrivateKey in crypto/tls/tls.go.
+func parsePrivateKey(der []byte) (crypto.Signer, error) {
+	if key, err := x509.ParsePKCS1PrivateKey(der); err == nil {
+		return key, nil
+	}
+	if key, err := x509.ParsePKCS8PrivateKey(der); err == nil {
+		switch key := key.(type) {
+		case *rsa.PrivateKey:
+			return key, nil
+		case *ecdsa.PrivateKey:
+			return key, nil
+		default:
+			return nil, errors.New("acme/autocert: unknown private key type in PKCS#8 wrapping")
+		}
+	}
+	if key, err := x509.ParseECPrivateKey(der); err == nil {
+		return key, nil
+	}
+
+	return nil, errors.New("acme/autocert: failed to parse private key")
+}
+
+func acmeKey(cs certStore) (crypto.Signer, error) {
+	if v, err := cs.ACMEKey(); err == nil {
+		priv, _ := pem.Decode(v)
+		if priv == nil || !strings.Contains(priv.Type, "PRIVATE") {
+			return nil, errors.New("acme/autocert: invalid account key found in cache")
+		}
+		return parsePrivateKey(priv.Bytes)
+	} else if !errors.Is(err, ipn.ErrStateNotExist) {
+		return nil, err
+	}
+
+	privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		return nil, err
+	}
+	var pemBuf bytes.Buffer
+	if err := encodeECDSAKey(&pemBuf, privKey); err != nil {
+		return nil, err
+	}
+	if err := cs.WriteACMEKey(pemBuf.Bytes()); err != nil {
+		return nil, err
+	}
+	return privKey, nil
+}
+
+func acmeClient(cs certStore) (*acme.Client, error) {
+	key, err := acmeKey(cs)
+	if err != nil {
+		return nil, fmt.Errorf("acmeKey: %w", err)
+	}
+	// Note: if we add support for additional ACME providers (other than
+	// LetsEncrypt), we should make sure that they support ARI extension (see
+	// shouldStartDomainRenewalARI).
+	return &acme.Client{
+		Key:          key,
+		UserAgent:    "tailscaled/" + version.Long(),
+		DirectoryURL: envknob.String("TS_DEBUG_ACME_DIRECTORY_URL"),
+	}, nil
+}
+
+// validCertPEM reports whether the given certificate is valid for
+// domain at now.
+//
+// If roots != nil, it is used instead of the system root pool. This is
+// meant to support testing; production code should pass roots == nil.
+func validCertPEM(domain string, keyPEM, certPEM []byte, roots *x509.CertPool, now time.Time) bool {
+	if len(keyPEM) == 0 || len(certPEM) == 0 {
+		return false
+	}
+	tlsCert, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		return false
+	}
+
+	var leaf *x509.Certificate
+	intermediates := x509.NewCertPool()
+	for i, certDER := range tlsCert.Certificate {
+		cert, err := x509.ParseCertificate(certDER)
+		if err != nil {
+			return false
+		}
+		if i == 0 {
+			leaf = cert
+		} else {
+			intermediates.AddCert(cert)
+		}
+	}
+	return validateLeaf(leaf, intermediates, domain, now, roots)
+}
+
+// validateLeaf is a helper for [validCertPEM].
+//
+// If called with roots == nil, it will use the system root pool as well
+// as the baked-in roots. If non-nil, only those roots are used.
+func validateLeaf(leaf *x509.Certificate, intermediates *x509.CertPool, domain string, now time.Time, roots *x509.CertPool) bool {
+	if leaf == nil {
+		return false
+	}
+	_, err := leaf.Verify(x509.VerifyOptions{
+		DNSName:       domain,
+		CurrentTime:   now,
+		Roots:         roots,
+		Intermediates: intermediates,
+	})
+	if err != nil && roots == nil {
+		// If validation failed and they specified nil for roots (meaning to use
+		// the system roots), then give it another chance to validate using the
+		// binary's baked-in roots (LetsEncrypt). See tailscale/tailscale#14690.
+		return validateLeaf(leaf, intermediates, domain, now, bakedroots.Get())
+	}
+
+	if err == nil {
+		return true
+	}
+
+	// When pointed at a non-prod ACME server, we don't expect to have the CA
+	// in our system or baked-in roots. Verify only throws UnknownAuthorityError
+	// after first checking the leaf cert's expiry, hostnames etc, so we know
+	// that the only reason for an error is to do with constructing a full chain.
+	// Allow this error so that cert caching still works in testing environments.
+	if errors.As(err, &x509.UnknownAuthorityError{}) {
+		acmeURL := envknob.String("TS_DEBUG_ACME_DIRECTORY_URL")
+		if !isDefaultDirectoryURL(acmeURL) {
+			return true
+		}
+	}
+
+	return false
+}
+
+func isDefaultDirectoryURL(u string) bool {
+	return u == "" || u == acme.LetsEncryptURL
+}

feature/acme/refresh.go

@@ -0,0 +1,154 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+package acme
+
+import (
+	"context"
+	"net"
+	"time"
+
+	"tailscale.com/envknob"
+	"tailscale.com/ipn"
+	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/util/set"
+)
+
+// certRefreshInterval is how often the background loop iterates the set
+// of applicable cert domains and pokes the renewal machinery. The loop
+// is only started while there's at least one HTTPS Web entry in the
+// ServeConfig, so this cadence doesn't tick on idle/mobile nodes.
+const certRefreshInterval = time.Hour
+
+// updateCertRefreshLoop starts or stops the background TLS cert refresh
+// loop based on whether the backend currently has any HTTPS-serving
+// hostname whose cert should be kept fresh. The loop runs only while:
+//
+//   - the node is in [ipn.Running], and
+//   - the current [ipn.ServeConfig] has at least one HTTPS-serving entry.
+//
+// We deliberately don't keep an idle timer around on hosts that have no
+// certs to maintain (e.g. mobile devices that never run Serve), so this
+// is called whenever any of those inputs change: state transitions and
+// ServeConfig reloads. The caller (in [ipn/ipnlocal]) holds b.mu when
+// invoking this; we use our own e.mu for the refresh-loop bookkeeping.
+func (e *extension) updateCertRefreshLoop(b *ipnlocal.LocalBackend, state ipn.State, sc ipn.ServeConfigView) {
+	shouldRun := state == ipn.Running && serveConfigUsesACMECerts(sc)
+
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	switch {
+	case shouldRun && e.certRefreshCancel == nil:
+		ctx, cancel := context.WithCancel(context.Background())
+		e.certRefreshCancel = cancel
+		e.Go(func() { e.certRefreshLoop(ctx, b) })
+	case !shouldRun && e.certRefreshCancel != nil:
+		e.certRefreshCancel()
+		e.certRefreshCancel = nil
+	}
+}
+
+// certRefreshLoop periodically iterates the domains configured for
+// Serve or Funnel HTTPS and calls GetCertPEM on each. The existing
+// renewal machinery in getCertPEM decides whether anything needs to
+// happen (ARI check or expiry-based fallback); the loop just ensures
+// it runs even on nodes that see no inbound TLS traffic.
+//
+// The first iteration runs immediately so that a node coming back
+// online with stale or absent certs starts ACME within seconds rather
+// than waiting a full interval.
+func (e *extension) certRefreshLoop(ctx context.Context, b *ipnlocal.LocalBackend) {
+	if envknob.IsCertShareReadOnlyMode() {
+		b.Logger()("cert refresh loop: cert-share read-only mode; loop is a no-op")
+		return
+	}
+
+	ticker, tickerCh := b.Clock().NewTicker(certRefreshInterval)
+	defer ticker.Stop()
+	for {
+		e.refreshApplicableCerts(ctx, b)
+		select {
+		case <-tickerCh:
+		case <-ctx.Done():
+			return
+		}
+	}
+}
+
+// refreshApplicableCerts is one iteration of the cert refresh loop.
+//
+// It enumerates the Serve/Funnel-configured HTTPS hostnames, keeps
+// those that resolveCertDomain accepts (CertDomain, wildcard, or BYO
+// Funnel domain), and calls [LocalBackend.GetCertPEM] for each. The
+// renewal decision is delegated to the existing logic in getCertPEM.
+func (e *extension) refreshApplicableCerts(ctx context.Context, b *ipnlocal.LocalBackend) {
+	sc := b.ServeConfig()
+	if !sc.Valid() {
+		return
+	}
+
+	want := set.Set[string]{}
+	consider := func(host string) {
+		if host == "" {
+			return
+		}
+		if _, err := e.resolveCertDomain(b, host); err != nil {
+			return
+		}
+		want.Add(host)
+	}
+	for hp := range sc.Webs() {
+		host, _, err := net.SplitHostPort(string(hp))
+		if err != nil {
+			continue
+		}
+		consider(host)
+	}
+	for _, tcp := range sc.TCPs() {
+		consider(tcp.TerminateTLS())
+	}
+	for _, svc := range sc.Services().All() {
+		for _, tcp := range svc.TCP().All() {
+			consider(tcp.TerminateTLS())
+		}
+	}
+	if want.Len() == 0 {
+		return
+	}
+
+	for d := range want {
+		e.Go(func() {
+			ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
+			defer cancel()
+			if _, err := e.getCertPEMWithValidity(ctx, b, d, 0); err != nil {
+				b.Logger()("cert refresh: %s: %v", d, err)
+			}
+		})
+	}
+}
+
+// serveConfigUsesACMECerts reports whether sc has any entry that
+// causes tailscaled to obtain ACME-managed TLS certs: an HTTPS Web
+// entry (background, foreground, or service) or a TCP handler with
+// TerminateTLS set (`tailscale serve --tls-terminated-tcp`).
+func serveConfigUsesACMECerts(sc ipn.ServeConfigView) bool {
+	if !sc.Valid() {
+		return false
+	}
+	for range sc.Webs() {
+		return true
+	}
+	for _, tcp := range sc.TCPs() {
+		if tcp.TerminateTLS() != "" {
+			return true
+		}
+	}
+	for _, svc := range sc.Services().All() {
+		for _, tcp := range svc.TCP().All() {
+			if tcp.TerminateTLS() != "" {
+				return true
+			}
+		}
+	}
+	return false
+}

feature/acme/testdata/example.com-key.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCejQaJrntrJSgE
+QtScyTU6TXOU+v1FdFjrsyHFK5mjV1C5pVQxnLn93GRshtIrGOLLrd3Wv2TVYZOX
+xH7f1ZLFbneDURCXbS+7nmsg+TLHRSRKfODbE3oYZj7NSJ163CCvwSJKTdmLpXbn
+ui9F04tyk0zxO4Wre4ukwf6xtse8G5zl2RJrueiVAiouTG/pJdIS08dGQa0GM1n9
+Aesa+TerlZcpRZR6X402yQqa8q/QqbIuzrlfDmgOb8sm6T8+JMtj3hEvnYdpMVOg
+w/XiTlX0v/YrB9sVQ9XnqGsqwTL0OMG0choMNKipwLi2n+XPSCIiRhi666zNNivE
+K1qaPS5RAgMBAAECggEAV9dAGQWPISR70CiKjLa5A60nbRHFQjackTE0c32daC6W
+7dOYGsh/DxOMm8fyJqhp9nhEYJa3MbUWxU27ER3NbA6wrhM6gvqeKG8zYRhPNrGq
+0o3vMdDPozb6cldZ0Fimz1jMO6h373NjtiyjxibWqkrLpRbaDtCq5EQKbMEcVa2D
+Xt5hxCOaCA3OZ/mAcGUNFmDNgNsGP/r6eXdI5pbqnUNMPkv/JsHl8h2HuyKUm4hf
+TRnXPAak6DkUod9QXYFKVBVPa5pjiO09e0aiMUvJ8vYd/6bNIsAKWLPa1PYuUE2l
+kg8Nik+P/XLzffKsLxiFKY0nCqrorM9K5q7baofGdQKBgQDPujjebFg6OKw6MS3S
+PESopvL//C/XgtgifcSSZCWzIZRVBVTbbJCGRtqFzF0XO4YRX3EOAyD/L7wYUPzO
++W3AU2W3/DVJYdcm2CASABbHNy0kk52LI0HHAssbFDgyB9XuuWP+vVZk7B5OmCAD
+Bppuj6Mnu03i282nKNJzvRiVnwKBgQDDZUXv22K8y7GkKw/ZW/wQP2zBNtFc15he
+1EOyUGHlXuQixnDSaqonkwec6IOlo7Sx/vwO/7+v4Jzc24Wq3DFAmMu/EYJgvI+m
+m3kpB4H7Xus4JqnhxqN7GB7zOdguCWZF1HLemZNZlVrUjG5mQ9cizzvvYptnQDLq
+FEJ1hddWDwKBgB+vy276Xfb7oCH8UH4KXXrQhK7RvEaGmgug3bRq/Gk3zRWvC4Ox
+KtagxkK0qtqZZNkPkwJNLeJfWLTo3beAyuIUlqabHVHFT/mH7FRymQbofsVekyCf
+TzBZV7wYuH3BPjv9IajBHwWkEvdwMyni/vmwhXXRF49schF2o6uuA6sHAoGBAL1J
+Xnb+EKjUq0JedPwcIBOdXb3PXQKT2QgEmZAkTrHlOxx1INa2fh/YT4ext9a+wE2u
+tn/RQeEfttY90z+yEASEAN0YGTWddYvxEW6t1z2stjGvQuN1ium0dEcrwkDW2jzL
+knwSSqx+A3/kiw6GqeMO3wEIhYOArdIVzkwLXJABAoGAOXLGhz5u5FWjF3zAeYme
+uHTU/3Z3jeI80PvShGrgAakPOBt3cIFpUaiOEslcqqgDUSGE3EnmkRqaEch+UapF
+ty6Zz7cKjXhQSWOjew1uUW2ANNEpsnYbmZOOnfvosd7jfHSVbL6KIhWmIdC6h0NP
+c/bJnTXEEVsWjLZTwYaq0Us=
+-----END PRIVATE KEY-----

feature/acme/testdata/example.com.pem

@@ -0,0 +1,26 @@
+-----BEGIN CERTIFICATE-----
+MIIEcDCCAtigAwIBAgIRAPmUKRkyFAkVVxFblB/233cwDQYJKoZIhvcNAQELBQAw
+gZ8xHjAcBgNVBAoTFW1rY2VydCBkZXZlbG9wbWVudCBDQTE6MDgGA1UECwwxZnJv
+bWJlcmdlckBzdGFyZHVzdC5sb2NhbCAoTWljaGFlbCBKLiBGcm9tYmVyZ2VyKTFB
+MD8GA1UEAww4bWtjZXJ0IGZyb21iZXJnZXJAc3RhcmR1c3QubG9jYWwgKE1pY2hh
+ZWwgSi4gRnJvbWJlcmdlcikwHhcNMjMwMjA3MjAzNDE4WhcNMjUwNTA3MTkzNDE4
+WjBlMScwJQYDVQQKEx5ta2NlcnQgZGV2ZWxvcG1lbnQgY2VydGlmaWNhdGUxOjA4
+BgNVBAsMMWZyb21iZXJnZXJAc3RhcmR1c3QubG9jYWwgKE1pY2hhZWwgSi4gRnJv
+bWJlcmdlcikwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCejQaJrntr
+JSgEQtScyTU6TXOU+v1FdFjrsyHFK5mjV1C5pVQxnLn93GRshtIrGOLLrd3Wv2TV
+YZOXxH7f1ZLFbneDURCXbS+7nmsg+TLHRSRKfODbE3oYZj7NSJ163CCvwSJKTdmL
+pXbnui9F04tyk0zxO4Wre4ukwf6xtse8G5zl2RJrueiVAiouTG/pJdIS08dGQa0G
+M1n9Aesa+TerlZcpRZR6X402yQqa8q/QqbIuzrlfDmgOb8sm6T8+JMtj3hEvnYdp
+MVOgw/XiTlX0v/YrB9sVQ9XnqGsqwTL0OMG0choMNKipwLi2n+XPSCIiRhi666zN
+NivEK1qaPS5RAgMBAAGjYDBeMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr
+BgEFBQcDATAfBgNVHSMEGDAWgBTXyq2jQVrnqQKL8fB9C4L0QJftwDAWBgNVHREE
+DzANggtleGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAYEAQWzpOaBkRR4M+WqB
+CsT4ARyM6WpZ+jpeSblCzPdlDRW+50G1HV7K930zayq4DwncPY/SqSn0Q31WuzZv
+bTWHkWa+MLPGYANHsusOmMR8Eh16G4+5+GGf8psWa0npAYO35cuNkyyCCc1LEB4M
+NrzCB2+KZ+SyOdfCCA5VzEKN3I8wvVLaYovi24Zjwv+0uETG92TlZmLQRhj8uPxN
+deeLM45aBkQZSYCbGMDVDK/XYKBkNLn3kxD/eZeXxxr41v4pH44+46FkYcYJzdn8
+ccAg5LRGieqTozhLiXARNK1vTy6kR1l/Az8DIx6GN4sP2/LMFYFijiiOCDKS1wWA
+xQgZeHt4GIuBym+Kd+Z5KXcP0AT+47Cby3+B10Kq8vHwjTELiF0UFeEYYMdynPAW
+pbEwVLhsfMsBqFtj3dsxHr8Kz3rnarOYzkaw7EMZnLAthb2CN7y5uGV9imQC5RMI
+/qZdRSuCYZ3A1E/WJkGbPY/YdPql/IE+LIAgKGFHZZNftBCo
+-----END CERTIFICATE-----

feature/acme/testdata/rootCA.pem

@@ -0,0 +1,30 @@
+-----BEGIN CERTIFICATE-----
+MIIFEDCCA3igAwIBAgIRANf5NdPojIfj70wMfJVYUg8wDQYJKoZIhvcNAQELBQAw
+gZ8xHjAcBgNVBAoTFW1rY2VydCBkZXZlbG9wbWVudCBDQTE6MDgGA1UECwwxZnJv
+bWJlcmdlckBzdGFyZHVzdC5sb2NhbCAoTWljaGFlbCBKLiBGcm9tYmVyZ2VyKTFB
+MD8GA1UEAww4bWtjZXJ0IGZyb21iZXJnZXJAc3RhcmR1c3QubG9jYWwgKE1pY2hh
+ZWwgSi4gRnJvbWJlcmdlcikwHhcNMjMwMjA3MjAzNDE4WhcNMzMwMjA3MjAzNDE4
+WjCBnzEeMBwGA1UEChMVbWtjZXJ0IGRldmVsb3BtZW50IENBMTowOAYDVQQLDDFm
+cm9tYmVyZ2VyQHN0YXJkdXN0LmxvY2FsIChNaWNoYWVsIEouIEZyb21iZXJnZXIp
+MUEwPwYDVQQDDDhta2NlcnQgZnJvbWJlcmdlckBzdGFyZHVzdC5sb2NhbCAoTWlj
+aGFlbCBKLiBGcm9tYmVyZ2VyKTCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoC
+ggGBAL5uXNnrZ6dgjcvK0Hc7ZNUIRYEWst9qbO0P9H7le08pJ6d9T2BUWruZtVjk
+Q12msv5/bVWHhVk8dZclI9FLXuMsIrocH8bsoP4wruPMyRyp6EedSKODN51fFSRv
+/jHbS5vzUVAWTYy9qYmd6qL0uhsHCZCCT6gfigamHPUFKM3sHDn5ZHWvySMwcyGl
+AicmPAIkBWqiCZAkB5+WM7+oyRLjmrIalfWIZYxW/rojGLwTfneHv6J5WjVQnpJB
+ayWCzCzaiXukK9MeBWeTOe8UfVN0Engd74/rjLWvjbfC+uZSr6RVkZvs2jANLwPF
+zgzBPHgRPfAhszU1NNAMjnNQ47+OMOTKRt7e6jYzhO5fyO1qVAAvGBqcfpj+JfDk
+cccaUMhUvdiGrhGf1V1tN/PislxvALirzcFipjD01isBKwn0fxRugzvJNrjEo8RA
+RvbcdeKcwex7M0o/Cd0+G2B13gZNOFvR33PmG7iTpp7IUrUKfQg28I83Sp8tMY3s
+ljJSawIDAQABo0UwQzAOBgNVHQ8BAf8EBAMCAgQwEgYDVR0TAQH/BAgwBgEB/wIB
+ADAdBgNVHQ4EFgQU18qto0Fa56kCi/HwfQuC9ECX7cAwDQYJKoZIhvcNAQELBQAD
+ggGBAAzs96LwZVOsRSlBdQqMo8oMAvs7HgnYbXt8SqaACLX3+kJ3cV/vrCE3iJrW
+ma4CiQbxS/HqsiZjota5m4lYeEevRnUDpXhp+7ugZTiz33Flm1RU99c9UYfQ+919
+ANPAKeqNpoPco/HF5Bz0ocepjcfKQrVZZNTj6noLs8o12FHBLO5976AcF9mqlNfh
+8/F0gDJXq6+x7VT5y8u0rY004XKPRe3CklRt8kpeMiP6mhRyyUehOaHeIbNx8ubi
+Pi44ByN/ueAnuRhF9zYtyZVZZOaSLysJge01tuPXF8rBXGruoJIv35xTTBa9BzaP
+YDOGbGn1ZnajdNagHqCba8vjTLDSpqMvgRj3TFrGHdETA2LDQat38uVxX8gxm68K
+va5Tyv7n+6BQ5YTpJjTPnmSJKaXZrrhdLPvG0OU2TxeEsvbcm5LFQofirOOw86Se
+vzF2cQ94mmHRZiEk0Av3NO0jF93ELDrBCuiccVyEKq6TknuvPQlutCXKDOYSEb8I
+MHctBg==
+-----END CERTIFICATE-----

ipn/ipnlocal/c2n_test.go

@@ -5,144 +5,19 @@ package ipnlocal
 
 import (
 	"bytes"
-	"cmp"
-	"crypto/x509"
 	"encoding/json"
 	"net/http/httptest"
-	"net/url"
-	"os"
-	"path/filepath"
-	"reflect"
 	"testing"
-	"time"
 
-	"tailscale.com/health"
-	"tailscale.com/ipn/store/mem"
 	"tailscale.com/tailcfg"
-	"tailscale.com/tstest"
 	"tailscale.com/types/key"
-	"tailscale.com/types/logger"
 	"tailscale.com/types/netmap"
 	"tailscale.com/types/views"
-	"tailscale.com/util/eventbus/eventbustest"
-	"tailscale.com/util/must"
 
 	gcmp "github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
 )
 
-func TestHandleC2NTLSCertStatus(t *testing.T) {
-	b := &LocalBackend{
-		store:   &mem.Store{},
-		varRoot: t.TempDir(),
-		health:  health.NewTracker(eventbustest.NewBus(t)),
-	}
-	certDir, err := b.certDir()
-	if err != nil {
-		t.Fatalf("certDir error: %v", err)
-	}
-	if _, err := b.getCertStore(); err != nil {
-		t.Fatalf("getCertStore error: %v", err)
-	}
-
-	testRoot, err := certTestFS.ReadFile("testdata/rootCA.pem")
-	if err != nil {
-		t.Fatal(err)
-	}
-	roots := x509.NewCertPool()
-	if !roots.AppendCertsFromPEM(testRoot) {
-		t.Fatal("Unable to add test CA to the cert pool")
-	}
-	testX509Roots = roots
-	defer func() { testX509Roots = nil }()
-
-	tests := []struct {
-		name       string
-		domain     string
-		copyFile   bool   // copy testdata/example.com.pem to the certDir
-		wantStatus int    // 0 means 200
-		wantError  string // wanted non-JSON non-200 error
-		now        time.Time
-		want       *tailcfg.C2NTLSCertInfo
-	}{
-		{
-			name:       "no-domain",
-			wantStatus: 400,
-			wantError:  "no 'domain'\n",
-		},
-		{
-			name:   "missing",
-			domain: "example.com",
-			want: &tailcfg.C2NTLSCertInfo{
-				Error:   "no certificate",
-				Missing: true,
-			},
-		},
-		{
-			name:     "valid",
-			domain:   "example.com",
-			now:      time.Date(2023, time.February, 20, 0, 0, 0, 0, time.UTC),
-			copyFile: true,
-			want: &tailcfg.C2NTLSCertInfo{
-				Valid:     true,
-				NotBefore: "2023-02-07T20:34:18Z",
-				NotAfter:  "2025-05-07T19:34:18Z",
-			},
-		},
-		{
-			name:     "expired",
-			domain:   "example.com",
-			now:      time.Date(2030, time.February, 20, 0, 0, 0, 0, time.UTC),
-			copyFile: true,
-			want: &tailcfg.C2NTLSCertInfo{
-				Error:   "cert expired",
-				Expired: true,
-			},
-		},
-	}
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			os.RemoveAll(certDir) // reset per test
-			if tt.copyFile {
-				os.MkdirAll(certDir, 0755)
-				if err := os.WriteFile(filepath.Join(certDir, "example.com.crt"),
-					must.Get(os.ReadFile("testdata/example.com.pem")), 0644); err != nil {
-					t.Fatal(err)
-				}
-				if err := os.WriteFile(filepath.Join(certDir, "example.com.key"),
-					must.Get(os.ReadFile("testdata/example.com-key.pem")), 0644); err != nil {
-					t.Fatal(err)
-				}
-			}
-			b.clock = tstest.NewClock(tstest.ClockOpts{
-				Start: tt.now,
-			})
-
-			rec := httptest.NewRecorder()
-			handleC2NTLSCertStatus(b, rec, httptest.NewRequest("GET", "/tls-cert-status?domain="+url.QueryEscape(tt.domain), nil))
-			res := rec.Result()
-			wantStatus := cmp.Or(tt.wantStatus, 200)
-			if res.StatusCode != wantStatus {
-				t.Fatalf("status code = %v; want %v. Body: %s", res.Status, wantStatus, rec.Body.Bytes())
-			}
-			if wantStatus == 200 {
-				var got tailcfg.C2NTLSCertInfo
-				if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
-					t.Fatalf("bad JSON: %v", err)
-				}
-				if !reflect.DeepEqual(&got, tt.want) {
-					t.Errorf("got %v; want %v", logger.AsJSON(got), logger.AsJSON(tt.want))
-				}
-			} else if tt.wantError != "" {
-				if got := rec.Body.String(); got != tt.wantError {
-					t.Errorf("body = %q; want %q", got, tt.wantError)
-				}
-			}
-		})
-	}
-
-}
-
 func TestHandleC2NDebugNetmap(t *testing.T) {
 	nm := &netmap.NetworkMap{
 		SelfNode: (&tailcfg.Node{

ipn/ipnlocal/cert_disabled.go

@@ -1,58 +0,0 @@
-// Copyright (c) Tailscale Inc & contributors
-// SPDX-License-Identifier: BSD-3-Clause
-
-//go:build js || ts_omit_acme
-
-package ipnlocal
-
-import (
-	"context"
-	"crypto/tls"
-	"errors"
-	"io"
-	"net/http"
-	"time"
-)
-
-func init() {
-	RegisterC2N("GET /tls-cert-status", handleC2NTLSCertStatusDisabled)
-}
-
-var errNoCerts = errors.New("cert support not compiled in this build")
-
-type TLSCertKeyPair struct {
-	CertPEM, KeyPEM []byte
-}
-
-func (b *LocalBackend) GetCertPEM(ctx context.Context, domain string) (*TLSCertKeyPair, error) {
-	return nil, errNoCerts
-}
-
-func serveTLSNextProtos() []string {
-	return []string{"h2", "http/1.1"}
-}
-
-func (b *LocalBackend) getACMETLSALPNCert(hi *tls.ClientHelloInfo) (*tls.Certificate, bool) {
-	return nil, false
-}
-
-func (b *LocalBackend) getACMETLSALPNProto(hi *tls.ClientHelloInfo) (string, bool) {
-	return "", false
-}
-
-var errCertExpired = errors.New("cert expired")
-
-type certStore interface{}
-
-func getCertPEMCached(cs certStore, domain string, now time.Time) (p *TLSCertKeyPair, err error) {
-	return nil, errNoCerts
-}
-
-func (b *LocalBackend) getCertStore() (certStore, error) {
-	return nil, errNoCerts
-}
-
-func handleC2NTLSCertStatusDisabled(b *LocalBackend, w http.ResponseWriter, r *http.Request) {
-	w.Header().Set("Content-Type", "application/json")
-	io.WriteString(w, `{"Missing":true}`) // a minimal tailcfg.C2NTLSCertInfo
-}

ipn/ipnlocal/cert_state.go

@@ -1,84 +0,0 @@
-// Copyright (c) Tailscale Inc & contributors
-// SPDX-License-Identifier: BSD-3-Clause
-
-package ipnlocal
-
-import (
-	"context"
-	"crypto/tls"
-	"sync"
-	"time"
-
-	"tailscale.com/feature"
-	"tailscale.com/syncs"
-	"tailscale.com/util/set"
-)
-
-// CertState holds the per-[LocalBackend] state owned by the
-// feature/acme extension. The struct lives in this package so that
-// cert.go can access its fields directly without method indirection,
-// while the extension in feature/acme remains the canonical owner.
-//
-// In builds without ACME support (js or ts_omit_acme), no extension
-// constructs a CertState, and [LocalBackend.certState] returns nil.
-//
-// CertState is safe for concurrent use; the individual fields document
-// their own synchronization.
-//
-// TODO(bradfitz): continue moving all this cert code into feature/acme's package.
-// This type being here was a compromise to keep the PR small during the move.
-type CertState struct {
-	// acmeMu serializes ACME operations so concurrent requests for
-	// certs don't slam ACME. The first goroutine through populates the
-	// on-disk cache and the rest reuse it.
-	acmeMu syncs.Mutex
-
-	// renewMu guards renewCertAt.
-	// Lock order: acmeMu before renewMu.
-	renewMu     syncs.Mutex
-	renewCertAt map[string]time.Time // lazily initialized under renewMu
-
-	// pendingACMETLSALPNCerts maps SNI names to short-lived ACME
-	// tls-alpn-01 challenge certificates while an ACME order is
-	// waiting for validation. Entries are deleted by the cleanup
-	// function returned from storeACMETLSALPNCert after the challenge
-	// validation path finishes, whether it succeeds or fails.
-	pendingACMETLSALPNCerts syncs.Map[string, *tls.Certificate] // "foo.bar.com" => challenge cert
-
-	// pendingCertDomains tracks the set of domains for which an ACME
-	// issuance is currently in flight with no usable cached cert. It
-	// backs the tls-cert-pending health Warnable.
-	// Guarded by pendingCertDomainsMu.
-	pendingCertDomainsMu sync.Mutex
-	pendingCertDomains   set.Set[string]
-
-	// getCertForTest is used to retrieve TLS certificates in tests.
-	// See [forTest.ConfigureCerts]. Guarded by the containing
-	// [LocalBackend]'s mutex (b.mu).
-	getCertForTest func(hostname string) (*TLSCertKeyPair, error)
-
-	// certRefreshCancel cancels the background TLS cert refresh loop
-	// that periodically pokes [LocalBackend.GetCertPEM] so renewals
-	// happen on idle nodes. Guarded by the containing [LocalBackend]'s
-	// mutex (b.mu). Non-nil while the loop is running.
-	certRefreshCancel context.CancelFunc
-}
-
-// hookCertState is set by the feature/acme extension at init time
-// to a function that returns the [CertState] for backend b, or nil
-// if the cert extension is not registered (e.g. in builds with
-// ts_omit_acme or js).
-var hookCertState feature.Hook[func(*LocalBackend) *CertState]
-
-// HookCertState exposes [hookCertState] to the feature/acme package
-// for installation. It must be set exactly once at init time.
-var HookCertState = &hookCertState
-
-// certState returns the cert state for b, or nil if the cert
-// extension is not registered.
-func (b *LocalBackend) certState() *CertState {
-	if f, ok := hookCertState.GetOk(); ok {
-		return f(b)
-	}
-	return nil
-}

ipn/ipnlocal/cert_state_test.go

@@ -1,36 +0,0 @@
-// Copyright (c) Tailscale Inc & contributors
-// SPDX-License-Identifier: BSD-3-Clause
-
-//go:build !js && !ts_omit_acme
-
-package ipnlocal
-
-import "sync"
-
-// In tests we can't import feature/acme (it would import this package
-// and form a cycle), so the real cert extension is never registered.
-// Install a default [hookCertState] provider here that lazily creates
-// a [CertState] per [LocalBackend].
-//
-// Tests that want different behavior can use
-// [feature.Hook.SetForTest] to override this hook for the duration
-// of the test.
-func init() {
-	if hookCertState.IsSet() {
-		return
-	}
-	var (
-		mu     sync.Mutex
-		states = map[*LocalBackend]*CertState{}
-	)
-	hookCertState.Set(func(b *LocalBackend) *CertState {
-		mu.Lock()
-		defer mu.Unlock()
-		if s, ok := states[b]; ok {
-			return s
-		}
-		s := new(CertState)
-		states[b] = s
-		return s
-	})
-}

ipn/ipnlocal/fortest.go

@@ -4,12 +4,14 @@
 package ipnlocal
 
 import (
+	"crypto/tls"
 	"net/http"
 
 	"tailscale.com/control/controlclient"
 	"tailscale.com/ipn"
 	"tailscale.com/ipn/ipnauth"
 	"tailscale.com/tailcfg"
+	"tailscale.com/tstime"
 	"tailscale.com/types/key"
 	"tailscale.com/types/netmap"
 	"tailscale.com/util/testenv"
@@ -99,16 +101,45 @@ func (f forTest) CurrentUser() (ipn.WindowsUserID, ipnauth.Actor) {
 // ConfigureCerts sets a certificate retrieval function to be used by this
 // local backend, skipping the usual ACME certificate registration.
 func (f forTest) ConfigureCerts(getCert func(hostname string) (*TLSCertKeyPair, error)) {
-	b := f.b
+	hook, ok := HookConfigureCertsForTest.GetOk()
-	cs := b.certState()
+	if !ok {
-	if cs == nil {
 		panic("forTest.ConfigureCerts called without cert extension registered")
 	}
-	b.mu.Lock()
+	hook(f.b, getCert)
-	cs.getCertForTest = getCert
-	b.mu.Unlock()
 }
 
+// GetACMETLSALPNCert returns the short-lived ACME tls-alpn-01 challenge
+// certificate for hi, if any.
+func (f forTest) GetACMETLSALPNCert(hi *tls.ClientHelloInfo) (*tls.Certificate, bool) {
+	return f.b.getACMETLSALPNCert(hi)
+}
+
+// SetServeConfig installs sc as the backend's current
+// [ipn.ServeConfig] without going through the validation in
+// [LocalBackend.SetServeConfig]. It is intended for tests that need a
+// specific serve config without first standing up the prerequisites
+// (netmap, prefs, etc.).
+func (f forTest) SetServeConfig(sc ipn.ServeConfigView) {
+	b := f.b
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	b.serveConfig = sc
+}
+
+// SetNetMap installs nm as the backend's current netmap without going
+// through control-plane plumbing. It is intended for tests that need a
+// specific netmap (e.g. CertDomains, capabilities).
+func (f forTest) SetNetMap(nm *netmap.NetworkMap) {
+	b := f.b
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	b.currentNode().SetNetMap(nm)
+}
+
+// SetClock replaces b's clock with c, for tests that need
+// time-dependent behavior to be deterministic.
+func (f forTest) SetClock(c tstime.Clock) { f.b.clock = c }
+
 // SetPrefs replaces the current prefs with newp.
 func (f forTest) SetPrefs(newp *ipn.Prefs) {
 	if newp == nil {

ipn/ipnlocal/ipnlocaltest/ipnlocaltest.go

@@ -0,0 +1,64 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+// Package ipnlocaltest provides test helpers for constructing a
+// [*ipnlocal.LocalBackend] from external test packages that cannot
+// access ipnlocal's internal test helpers.
+package ipnlocaltest
+
+import (
+	"testing"
+
+	"tailscale.com/ipn/ipnlocal"
+	"tailscale.com/ipn/store/mem"
+	"tailscale.com/net/netmon"
+	"tailscale.com/net/tsdial"
+	"tailscale.com/tsd"
+	"tailscale.com/types/logger"
+	"tailscale.com/types/logid"
+	"tailscale.com/util/eventbus/eventbustest"
+	"tailscale.com/util/testenv"
+	"tailscale.com/wgengine"
+)
+
+// NewBackend creates a new [*ipnlocal.LocalBackend] suitable for tests,
+// using an in-memory state store, a fake userspace engine, and a static
+// network monitor. Shutdown is registered as a t.Cleanup.
+func NewBackend(t testing.TB) *ipnlocal.LocalBackend {
+	testenv.AssertInTest()
+	bus := eventbustest.NewBus(t)
+	return NewBackendWithSys(t, tsd.NewSystemWithBus(bus))
+}
+
+// NewBackendWithSys creates a new [*ipnlocal.LocalBackend] with the
+// given [*tsd.System]. Missing components in sys (state store, engine,
+// dialer) are filled in with test fakes.
+func NewBackendWithSys(t testing.TB, sys *tsd.System) *ipnlocal.LocalBackend {
+	testenv.AssertInTest()
+	var logf logger.Logf = logger.Discard
+	if _, ok := sys.StateStore.GetOK(); !ok {
+		sys.Set(new(mem.Store))
+		t.Log("Added memory store for testing")
+	}
+	if _, ok := sys.Engine.GetOK(); !ok {
+		eng, err := wgengine.NewFakeUserspaceEngine(logf, sys.Set, sys.HealthTracker.Get(), sys.UserMetricsRegistry(), sys.Bus.Get())
+		if err != nil {
+			t.Fatalf("NewFakeUserspaceEngine: %v", err)
+		}
+		t.Cleanup(eng.Close)
+		sys.Set(eng)
+		t.Log("Added fake userspace engine for testing")
+	}
+	if _, ok := sys.Dialer.GetOK(); !ok {
+		dialer := tsdial.NewDialer(netmon.NewStatic())
+		dialer.SetBus(sys.Bus.Get())
+		sys.Set(dialer)
+		t.Log("Added static dialer for testing")
+	}
+	lb, err := ipnlocal.NewLocalBackend(logf, logid.PublicID{}, sys, 0)
+	if err != nil {
+		t.Fatalf("NewLocalBackend: %v", err)
+	}
+	t.Cleanup(lb.Shutdown)
+	return lb
+}

ipn/ipnlocal/local.go

@@ -1221,11 +1221,6 @@ var (
 	hookCheckCaptivePortalLoop    feature.Hook[func(*LocalBackend, context.Context)]
 )
 
-// hookCertRefreshLoop is set by the ACME-enabled cert code to a function
-// that periodically refreshes TLS certs for Serve/Funnel-configured
-// domains so renewals proceed even on otherwise-idle nodes.
-var hookCertRefreshLoop feature.Hook[func(*LocalBackend, context.Context)]
-
 func (b *LocalBackend) onHealthChange(change health.Change) {
 	if !buildfeatures.HasHealth {
 		return
@@ -1327,12 +1322,7 @@ func (b *LocalBackend) Shutdown() {
 		b.captiveCancel()
 	}
 
-	if buildfeatures.HasACME {
+	b.shutdownCertRefreshLoopLocked()
-		if state := b.certState(); state != nil && state.certRefreshCancel != nil {
-			state.certRefreshCancel()
-			state.certRefreshCancel = nil
-		}
-	}
 
 	b.stopReconnectTimerLocked()
 
@@ -7478,69 +7468,6 @@ func (b *LocalBackend) setTCPPortsInterceptedFromNetmapAndPrefsLocked(prefs ipn.
 // The LocalBackend's mutex is held while calling.
 var hookMaybeMutateHostinfoLocked feature.Hooks[func(*LocalBackend, *tailcfg.Hostinfo, ipn.PrefsView) bool]
 
-// updateCertRefreshLoopLocked starts or stops the background TLS cert
-// refresh loop based on whether we currently have any HTTPS-serving
-// hostname whose cert we should keep fresh. The loop runs only while:
-//
-//   - ACME support is compiled in,
-//   - the node is in [ipn.Running], and
-//   - the current [ipn.ServeConfig] has at least one HTTPS Web entry.
-//
-// We deliberately don't keep an idle timer around on hosts that have no
-// certs to maintain (e.g. mobile devices that never run Serve), so this
-// must be called whenever any of those inputs change: state transitions
-// and ServeConfig reloads.
-//
-// b.mu must be held.
-func (b *LocalBackend) updateCertRefreshLoopLocked() {
-	if !buildfeatures.HasACME {
-		return
-	}
-	state := b.certState()
-	if state == nil {
-		return
-	}
-	shouldRun := hookCertRefreshLoop.IsSet() &&
-		b.state == ipn.Running &&
-		serveConfigUsesACMECerts(b.serveConfig)
-
-	switch {
-	case shouldRun && state.certRefreshCancel == nil:
-		ctx, cancel := context.WithCancel(b.ctx)
-		state.certRefreshCancel = cancel
-		b.goTracker.Go(func() { hookCertRefreshLoop.Get()(b, ctx) })
-	case !shouldRun && state.certRefreshCancel != nil:
-		state.certRefreshCancel()
-		state.certRefreshCancel = nil
-	}
-}
-
-// serveConfigUsesACMECerts reports whether sc has any entry that
-// causes tailscaled to obtain ACME-managed TLS certs: an HTTPS Web
-// entry (background, foreground, or service) or a TCP handler with
-// TerminateTLS set (`tailscale serve --tls-terminated-tcp`).
-func serveConfigUsesACMECerts(sc ipn.ServeConfigView) bool {
-	if !sc.Valid() {
-		return false
-	}
-	for range sc.Webs() {
-		return true
-	}
-	for _, tcp := range sc.TCPs() {
-		if tcp.TerminateTLS() != "" {
-			return true
-		}
-	}
-	for _, svc := range sc.Services().All() {
-		for _, tcp := range svc.TCP().All() {
-			if tcp.TerminateTLS() != "" {
-				return true
-			}
-		}
-	}
-	return false
-}
-
 // maybeSentHostinfoIfChangedLocked updates the hostinfo.ServicesHash, hostinfo.WireIngress and
 // hostinfo.IngressEnabled fields and kicks off a Hostinfo update if the values have changed.
 //

ipn/ipnlocal/serve.go

@@ -1367,7 +1367,9 @@ func (b *LocalBackend) serveTLSConfig(getCert func(*tls.ClientHelloInfo) (*tls.C
 	return base
 }
 
-func (b *LocalBackend) hasFunnelForHostPort(host string, port uint16) bool {
+// HasFunnelForHostPort reports whether the LocalBackend's serve config
+// has Funnel enabled for host:port.
+func (b *LocalBackend) HasFunnelForHostPort(host string, port uint16) bool {
 	b.mu.Lock()
 	defer b.mu.Unlock()
 	if !b.serveConfig.Valid() {

ipn/ipnlocal/serve_disabled.go

@@ -28,7 +28,7 @@ type funnelFlow = struct{}
 
 func (*LocalBackend) hasIngressEnabledLocked() bool         { return false }
 func (*LocalBackend) shouldWireInactiveIngressLocked() bool { return false }
-func (*LocalBackend) hasFunnelForHostPort(host string, port uint16) bool {
+func (*LocalBackend) HasFunnelForHostPort(host string, port uint16) bool {
 	return false
 }
 

tsnet/depaware.txt

@@ -165,7 +165,7 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
         tailscale.com/ipn/ipnlocal/netmapcache                       from tailscale.com/ipn/ipnlocal
         tailscale.com/ipn/ipnstate                                   from tailscale.com/client/local+
         tailscale.com/ipn/localapi                                   from tailscale.com/tsnet
-        tailscale.com/ipn/store                                      from tailscale.com/ipn/ipnlocal+
+        tailscale.com/ipn/store                                      from tailscale.com/tsnet+
         tailscale.com/ipn/store/mem                                  from tailscale.com/ipn/ipnlocal+
         tailscale.com/kube/kubetypes                                 from tailscale.com/envknob
  LDWI    tailscale.com/licenses                                       from tailscale.com/client/web
@@ -175,7 +175,7 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
         tailscale.com/logtail                                        from tailscale.com/control/controlclient+
         tailscale.com/logtail/filch                                  from tailscale.com/log/sockstatlog+
         tailscale.com/metrics                                        from tailscale.com/tsweb+
-        tailscale.com/net/bakedroots                                 from tailscale.com/ipn/ipnlocal+
+        tailscale.com/net/bakedroots                                 from tailscale.com/net/tlsdial+
      💣 tailscale.com/net/batching                                   from tailscale.com/wgengine/magicsock
         tailscale.com/net/captivedetection                           from tailscale.com/ipn/ipnlocal+
         tailscale.com/net/dns                                        from tailscale.com/ipn/ipnlocal+
@@ -222,7 +222,7 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
      💣 tailscale.com/safesocket                                     from tailscale.com/client/local+
         tailscale.com/syncs                                          from tailscale.com/control/controlhttp+
         tailscale.com/tailcfg                                        from tailscale.com/client/local+
-        tailscale.com/tempfork/acme                                  from tailscale.com/ipn/ipnlocal
+        tailscale.com/tempfork/acme                                  from tailscale.com/feature/acme
         tailscale.com/tempfork/heap                                  from tailscale.com/wgengine/magicsock
         tailscale.com/tempfork/httprec                               from tailscale.com/feature/c2n
         tailscale.com/tka                                            from tailscale.com/client/local+
@@ -407,22 +407,22 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
         crypto/internal/boring/bbig                                  from crypto/ecdsa+
         crypto/internal/boring/sig                                   from crypto/internal/boring
         crypto/internal/constanttime                                 from crypto/internal/fips140/edwards25519+
-        crypto/internal/fips140                                      from crypto/internal/fips140/aes+
+        crypto/internal/fips140                                      from crypto/fips140+
         crypto/internal/fips140/aes                                  from crypto/aes+
         crypto/internal/fips140/aes/gcm                              from crypto/cipher+
         crypto/internal/fips140/alias                                from crypto/cipher+
         crypto/internal/fips140/bigmod                               from crypto/internal/fips140/ecdsa+
-        crypto/internal/fips140/check                                from crypto/internal/fips140/aes+
+        crypto/internal/fips140/check                                from crypto/fips140+
-        crypto/internal/fips140/drbg                                 from crypto/internal/fips140/aes/gcm+
+        crypto/internal/fips140/drbg                                 from crypto/hpke+
         crypto/internal/fips140/ecdh                                 from crypto/ecdh
         crypto/internal/fips140/ecdsa                                from crypto/ecdsa
         crypto/internal/fips140/ed25519                              from crypto/ed25519
         crypto/internal/fips140/edwards25519                         from crypto/internal/fips140/ed25519
         crypto/internal/fips140/edwards25519/field                   from crypto/ecdh+
-        crypto/internal/fips140/hkdf                                 from crypto/internal/fips140/tls13+
+        crypto/internal/fips140/hkdf                                 from crypto/hkdf+
         crypto/internal/fips140/hmac                                 from crypto/hmac+
         crypto/internal/fips140/mlkem                                from crypto/mlkem
-        crypto/internal/fips140/nistec                               from crypto/elliptic+
+        crypto/internal/fips140/nistec                               from crypto/ecdsa+
         crypto/internal/fips140/nistec/fiat                          from crypto/internal/fips140/nistec
         crypto/internal/fips140/rsa                                  from crypto/rsa
         crypto/internal/fips140/sha256                               from crypto/internal/fips140/check+
@@ -491,13 +491,13 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
         internal/filepathlite                                        from os+
         internal/fmtsort                                             from fmt+
         internal/goarch                                              from crypto/internal/fips140deps/cpu+
-        internal/godebug                                             from crypto/internal/fips140deps/godebug+
+        internal/godebug                                             from crypto/ed25519+
         internal/godebugs                                            from internal/godebug+
         internal/goexperiment                                        from net/http/pprof+
         internal/goos                                                from crypto/x509+
         internal/msan                                                from internal/runtime/maps+
         internal/nettrace                                            from net+
-        internal/oserror                                             from io/fs+
+        internal/oserror                                             from internal/syscall/windows+
         internal/poll                                                from net+
  LDWI    internal/profile                                             from net/http/pprof
         internal/profilerecord                                       from runtime+
@@ -507,9 +507,9 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
         internal/runtime/atomic                                      from internal/runtime/exithook+
   LA    internal/runtime/cgroup                                      from runtime
         internal/runtime/exithook                                    from runtime
-        internal/runtime/gc                                          from runtime+
+        internal/runtime/gc                                          from internal/runtime/gc/scan+
         internal/runtime/gc/scan                                     from runtime
-        internal/runtime/maps                                        from reflect+
+        internal/runtime/maps                                        from hash/maphash+
         internal/runtime/math                                        from internal/runtime/maps+
         internal/runtime/pprof/label                                 from runtime+
         internal/runtime/sys                                         from crypto/subtle+
@@ -523,7 +523,7 @@ tailscale.com/tsnet dependencies: (generated by github.com/tailscale/depaware)
         internal/synctest                                            from sync
         internal/syscall/execenv                                     from os+
  LDAI    internal/syscall/unix                                        from crypto/internal/sysrand+
-   W    internal/syscall/windows                                     from crypto/internal/sysrand+
+   W    internal/syscall/windows                                     from crypto/internal/fips140deps/time+
    W    internal/syscall/windows/registry                            from mime+
    W    internal/syscall/windows/sysdll                              from internal/syscall/windows+
         internal/testlog                                             from os