wgengine, cmd/tailscaled, control/controlclient: remove Engine watchdog

The Engine watchdog wrapped every wgengine.Engine method call in a
goroutine with a 45s timeout and crashed the process on timeout. It
was added years ago to surface deadlocks during development, but the
underlying deadlocks have long since been fixed, and even when it did
fire it produced obscure stack traces (from inside the watchdog
goroutine, not the original caller) without buying much.

Audit of userspaceEngine's methods shows none have cyclic locking or
unbounded blocking now that ResetAndStop no longer loops waiting for
DERPs to drain (fa49009ee). The watchdog is dead weight; remove it
along with the TS_DEBUG_DISABLE_WATCHDOG escape hatch.

Updates #19759

Change-Id: Iba9d718fe1f8718a6631296e336b138c31b99ff1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

.github/workflows/installer.yml

@@ -37,8 +37,6 @@ jobs:
           - "elementary/docker:stable"
           - "elementary/docker:unstable"
           - "parrotsec/core:latest"
-          - "kalilinux/kali-rolling"
-          - "kalilinux/kali-dev"
           - "oraclelinux:9"
           - "oraclelinux:8"
           - "fedora:latest"
@@ -61,6 +59,9 @@ jobs:
           - { image: "debian:stable-slim", deps: "curl" }
           - { image: "ubuntu:24.04", deps: "curl" }
           - { image: "fedora:latest", deps: "curl" }
+          # Kali doesn't have ca-certificates installed by default anymore
+          - { image: "kalilinux/kali-dev", "deps": "curl ca-certificates"}
+          - { image: "kalilinux/kali-rolling", "deps": "curl ca-certificates"}
           # Test TAILSCALE_VERSION pinning on a subset of distros.
           # Skip Alpine as community repos don't reliably keep old versions.
           - { image: "debian:stable-slim", deps: "curl", version: "1.80.0" }

.github/workflows/natlab-basic.yml

@@ -1,6 +1,7 @@
-# Run some natlab integration tests.
+# Run a single natlab smoke test on every PR. The full natlab suite
+# is opt-in and lives in .github/workflows/natlab-test.yml.
 # See https://github.com/tailscale/tailscale/issues/13038
-name: "natlab-integrationtest"
+name: "natlab-basic"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -17,17 +18,28 @@ on:
     branches:
       - "main"
 jobs:
-  natlab-integrationtest:
+  EasyEasy:
     runs-on: ubuntu-latest
     steps:
       - name: Check out code
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - name: Enable KVM
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
       - name: Install qemu
         run: |
           sudo rm -f /var/lib/man-db/auto-update
           sudo apt-get -y update
           sudo apt-get -y remove man-db
           sudo apt-get install -y qemu-system-x86 qemu-utils
+      - name: Build VM image
+        # The test will build this if missing, but we do it explicitly
+        # to avoid cutting into the go test -timeout budget, and to
+        # fail earlier with a clearer error if the image build breaks.
+        run: |
+          make -C gokrazy natlab
       - name: Run natlab integration tests
         run: |
-          ./tool/go test -v -run=^TestEasyEasy$ -timeout=3m -count=1 ./tstest/integration/nat  --run-vm-tests
+          ./tool/go test -v -run=^TestEasyEasy$ -timeout=3m -count=1 ./tstest/natlab/vmtest --run-vm-tests

.github/workflows/natlab-test.yml

@@ -0,0 +1,182 @@
+# Run the full natlab/vmtest opt-in test suite. These tests boot QEMU VMs
+# (gokrazy, Ubuntu, FreeBSD) and exercise vnet-driven networking scenarios.
+# They are gated behind --run-vm-tests because they need KVM and are slow.
+#
+# This workflow runs:
+#   - on demand (workflow_dispatch)
+#   - on PRs that carry the "run-natlab-tests" label
+#   - on main, every 12 hours, via cron
+#
+# Layout:
+#   - "prepare" builds the gokrazy VM image, downloads the cloud images
+#     (Ubuntu, FreeBSD), and discovers every Test* function in the two
+#     opt-in packages.
+#   - "test" is a per-TestFoo matrix that depends on prepare. Each matrix
+#     job restores the shared caches and runs a single test. Adding a new
+#     TestFoo automatically gets its own job — no workflow edits needed.
+#
+# A separate workflow (.github/workflows/natlab-basic.yml) runs a single
+# canary natlab test on every PR; this one runs the full suite.
+name: "natlab-test"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+on:
+  workflow_dispatch:
+  pull_request:
+    types: [labeled, synchronize, reopened]
+  schedule:
+    # Every 12 hours, off-the-hour to avoid GitHub's :00 cron-stampede window.
+    - cron: "23 3,15 * * *"
+
+jobs:
+  # prepare warms the per-workflow-run caches (gokrazy image, cloud VM
+  # images) and emits the dynamic matrix of test names. By doing the work
+  # once here, the matrix test jobs never race to rebuild or re-download
+  # the same artifacts on a cold cache.
+  prepare:
+    if: |
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'schedule' ||
+      (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-natlab-tests'))
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    outputs:
+      matrix: ${{ steps.list.outputs.matrix }}
+    steps:
+      - name: Check out code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      # The cloud VM image cache is keyed only on images.go (image URLs and
+      # SHAs), so it survives across workflow runs and is invalidated only
+      # when a new image source is added.
+      - name: Cache cloud VM images
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
+        with:
+          path: ~/.cache/tailscale/vmtest/images
+          key: natlab-vmimages-${{ hashFiles('tstest/natlab/vmtest/images.go') }}
+
+      # The gokrazy VM image is keyed by github.sha. That means we rebuild
+      # it once per commit but matrix test jobs in the same run all share
+      # the result. Per-PR re-runs of the same sha (e.g. a rerun-failed)
+      # also get the cache.
+      - name: Cache gokrazy VM image
+        id: gokrazy-cache
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
+        with:
+          path: gokrazy/natlabapp.qcow2
+          key: natlab-gokrazy-${{ github.sha }}
+
+      # qemu-utils provides qemu-img, which the gokrazy Makefile uses to
+      # convert natlabapp.img to qcow2. Only install if we need it (cache
+      # miss); the test matrix jobs install qemu separately for the runtime.
+      - name: Install qemu-utils
+        if: steps.gokrazy-cache.outputs.cache-hit != 'true'
+        run: |
+          sudo rm -f /var/lib/man-db/auto-update
+          sudo apt-get -y update
+          sudo apt-get -y remove man-db
+          sudo apt-get install -y qemu-utils
+
+      - name: Download cloud VM images
+        # natlabprep is idempotent: it checks the cache before downloading.
+        run: |
+          ./tool/go run ./tstest/natlab/vmtest/cmd/natlabprep
+
+      - name: Build gokrazy VM image
+        if: steps.gokrazy-cache.outputs.cache-hit != 'true'
+        run: |
+          make -C gokrazy natlab
+
+      - name: Discover tests
+        id: list
+        # Grep the test files directly rather than invoking `go test -list`
+        # so we don't pay the cost of compiling the test binaries here. The
+        # only test functions in these packages use the canonical
+        # `func TestFoo(t *testing.T)` signature.
+        #
+        # exclude is the set of tests that need special invocation
+        # (extra flags, a specific environment) and don't fit the
+        # single-test-per-matrix-job model. They stay runnable locally.
+        run: |
+          set -euo pipefail
+          exclude='^(TestGrid)$'
+          tmp=$(mktemp)
+          for pkg_dir in tstest/natlab/vmtest tstest/integration/nat; do
+            pkg="./${pkg_dir}/"
+            for f in "${pkg_dir}"/*_test.go; do
+              [ -e "$f" ] || continue
+              grep -hE '^func Test[A-Z][A-Za-z0-9_]*\(t \*testing\.T\)' "$f" \
+                | sed -E 's/^func (Test[A-Za-z0-9_]+).*/\1/' \
+                | grep -vE "$exclude" \
+                | while read -r t; do
+                    jq -nc --arg pkg "$pkg" --arg test "$t" \
+                      '{pkg: $pkg, test: $test}' >> "$tmp"
+                  done
+            done
+          done
+          matrix=$(jq -s -c . "$tmp")
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+          echo "Discovered tests:"
+          jq . "$tmp"
+
+  test:
+    needs: prepare
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    name: "${{ matrix.test }}"
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJson(needs.prepare.outputs.matrix) }}
+    steps:
+      - name: Check out code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Enable KVM
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
+
+      - name: Install qemu
+        run: |
+          sudo rm -f /var/lib/man-db/auto-update
+          sudo apt-get -y update
+          sudo apt-get -y remove man-db
+          sudo apt-get install -y qemu-system-x86 qemu-utils
+
+      # restore-only: prepare is the single writer of these caches, so
+      # matrix jobs don't write back. fail-on-cache-miss would be too
+      # strict for the gokrazy cache (e.g. a non-fatal cache eviction
+      # between prepare and us); we just rebuild on miss instead.
+      - name: Restore cloud VM images
+        uses: actions/cache/restore@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
+        with:
+          path: ~/.cache/tailscale/vmtest/images
+          key: natlab-vmimages-${{ hashFiles('tstest/natlab/vmtest/images.go') }}
+
+      - name: Restore gokrazy VM image
+        uses: actions/cache/restore@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
+        with:
+          path: gokrazy/natlabapp.qcow2
+          key: natlab-gokrazy-${{ github.sha }}
+
+      # The gokrazy-based tests boot the kernel directly from
+      # vmlinuz that ships in the tailscale/gokrazy-kernel module.
+      # Tests look it up under GOMODCACHE via findKernelPath, so the
+      # module has to be present even though no Go source imports it
+      # in the test package itself.
+      - name: Download gokrazy-kernel module
+        run: |
+          ./tool/go mod download github.com/tailscale/gokrazy-kernel
+
+      - name: Run ${{ matrix.test }}
+        # Per-test timeout is well above the few-minute typical runtime
+        # but small enough that a stuck test fails fast instead of holding
+        # the runner for the job's 20-minute budget.
+        run: |
+          ./tool/go test -v -timeout=15m -count=1 ${{ matrix.pkg }} \
+            -run='^${{ matrix.test }}$' --run-vm-tests

.github/workflows/ssh-integrationtest.yml

@@ -1,5 +1,5 @@
-# Run the ssh integration tests with `make sshintegrationtest`.
+# Run the ssh integration tests in various Docker containers.
-# These tests can also be running locally.
+# These tests can also be run locally via `make sshintegrationtest`.
 name: "ssh-integrationtest"
 
 concurrency:
@@ -15,9 +15,25 @@ on:
 jobs:
   ssh-integrationtest:
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - base: "ubuntu:focal"
+            tag: "ssh-ubuntu-focal"
+          - base: "ubuntu:jammy"
+            tag: "ssh-ubuntu-jammy"
+          - base: "ubuntu:noble"
+            tag: "ssh-ubuntu-noble"
+          - base: "alpine:latest"
+            tag: "ssh-alpine-latest"
     steps:
       - name: Check out code
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-      - name: Run SSH integration tests
+      - name: Build test binaries
         run: |
-          make sshintegrationtest
+          GOOS=linux GOARCH=amd64 CGO_ENABLED=0 ./tool/go test -tags integrationtest -c ./ssh/tailssh -o ssh/tailssh/testcontainers/tailssh.test
+          GOOS=linux GOARCH=amd64 CGO_ENABLED=0 ./tool/go build -o ssh/tailssh/testcontainers/tailscaled ./cmd/tailscaled
+      - name: Run SSH integration tests (${{ matrix.base }})
+        run: |
+          docker build --build-arg="BASE=${{ matrix.base }}" -t "${{ matrix.tag }}" ssh/tailssh/testcontainers

.github/workflows/test.yml

@@ -361,7 +361,7 @@ jobs:
       run: chown -R $(id -u):$(id -g) $PWD
     - name: privileged tests
       working-directory: src
-      run: ./tool/go test ./util/linuxfw ./derp/xdp
+      run: ./tool/go test $(./tool/go run ./tool/listpkgs --has-root-tests)
 
   vm:
     needs: gomod-cache
@@ -787,6 +787,14 @@ jobs:
         echo
         echo
         git diff --name-only --exit-code || (echo "The files above need updating. Please run 'go generate'."; exit 1)
+    - name: check that 'genreadme' is clean
+      working-directory: src
+      run: |
+        ./tool/go run ./misc/genreadme
+        git add -N . # ensure untracked files are noticed
+        echo
+        echo
+        git diff --name-only --exit-code || (echo "The files above need updating. Please run './tool/go run ./misc/genreadme'."; exit 1)
 
   make_tidy:
     runs-on: ubuntu-24.04

.github/workflows/update-flake.yml

@@ -23,8 +23,8 @@ jobs:
       - name: Check out code
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
 
-      - name: Run update-flakes
+      - name: Run updateflakes
-        run: ./update-flake.sh
+        run: ./tool/go run ./tool/updateflakes
 
       - name: Get access token
         uses: actions/create-github-app-token@f8d387b68d61c58ab83c6c016672934102569859 # v3.0.0
@@ -41,8 +41,8 @@ jobs:
           author: Flakes Updater <noreply+flakes-updater@tailscale.com>
           committer: Flakes Updater <noreply+flakes-updater@tailscale.com>
           branch: flakes
-          commit-message: "go.mod.sri: update SRI hash for go.mod changes"
+          commit-message: "flakehashes.json: update SRI hash for go.mod changes"
-          title: "go.mod.sri: update SRI hash for go.mod changes"
+          title: "flakehashes.json: update SRI hash for go.mod changes"
           body: Triggered by ${{ github.repository }}@${{ github.sha }}
           signoff: true
           delete-branch: true

.gitignore

@@ -1,12 +1,15 @@
 # Binaries for programs and plugins
 *~
 *.tmp
-*.exe
 *.dll
 *.so
 *.dylib
 *.spk
 
+*.exe
+# tool/go.exe is built specially and committed.
+!/tool/go.exe
+
 cmd/tailscale/tailscale
 cmd/tailscaled/tailscaled
 ssh/tailssh/testcontainers/tailscaled

Makefile

@@ -10,7 +10,7 @@ vet: ## Run go vet
 
 tidy: ## Run go mod tidy and update nix flake hashes
 	./tool/go mod tidy
-	./update-flake.sh
+	./tool/go run ./tool/updateflakes
 
 lint: ## Run golangci-lint
 	./tool/go run github.com/golangci/golangci-lint/cmd/golangci-lint run
@@ -137,10 +137,12 @@ publishdevproxy: check-image-repo ## Build and publish k8s-proxy image to locati
 sshintegrationtest: ## Run the SSH integration tests in various Docker containers
 	@GOOS=linux GOARCH=amd64 CGO_ENABLED=0 ./tool/go test -tags integrationtest -c ./ssh/tailssh -o ssh/tailssh/testcontainers/tailssh.test && \
 	GOOS=linux GOARCH=amd64 CGO_ENABLED=0 ./tool/go build -o ssh/tailssh/testcontainers/tailscaled ./cmd/tailscaled && \
-	echo "Testing on ubuntu:focal" && docker build --build-arg="BASE=ubuntu:focal" -t ssh-ubuntu-focal ssh/tailssh/testcontainers && \
+	echo "Testing on ubuntu:focal, ubuntu:jammy, ubuntu:noble, alpine:latest (in parallel)" && \
-	echo "Testing on ubuntu:jammy" && docker build --build-arg="BASE=ubuntu:jammy" -t ssh-ubuntu-jammy ssh/tailssh/testcontainers && \
+	docker build --build-arg="BASE=ubuntu:focal" -t ssh-ubuntu-focal ssh/tailssh/testcontainers & \
-	echo "Testing on ubuntu:noble" && docker build --build-arg="BASE=ubuntu:noble" -t ssh-ubuntu-noble ssh/tailssh/testcontainers && \
+	docker build --build-arg="BASE=ubuntu:jammy" -t ssh-ubuntu-jammy ssh/tailssh/testcontainers & \
-	echo "Testing on alpine:latest" && docker build --build-arg="BASE=alpine:latest" -t ssh-alpine-latest ssh/tailssh/testcontainers
+	docker build --build-arg="BASE=ubuntu:noble" -t ssh-ubuntu-noble ssh/tailssh/testcontainers & \
+	docker build --build-arg="BASE=alpine:latest" -t ssh-alpine-latest ssh/tailssh/testcontainers & \
+	wait
 
 .PHONY: generate
 generate: ## Generate code

VERSION.txt

@@ -1 +1 @@
-1.97.0
+1.99.0

appc/appconnector_test.go

@@ -736,6 +736,7 @@ func TestRateLogger(t *testing.T) {
 }
 
 func TestRouteStoreMetrics(t *testing.T) {
+	clientmetric.ResetForTest(t)
 	metricStoreRoutes(1, 1)
 	metricStoreRoutes(1, 1)         // the 1 buckets value should be 2
 	metricStoreRoutes(5, 5)         // the 5 buckets value should be 1

appc/conn25.go

@@ -6,6 +6,7 @@ package appc
 import (
 	"cmp"
 	"slices"
+	"strings"
 
 	"tailscale.com/ipn/ipnext"
 	"tailscale.com/tailcfg"
@@ -16,7 +17,7 @@ import (
 
 const AppConnectorsExperimentalAttrName = "tailscale.com/app-connectors-experimental"
 
-func isEligibleConnector(peer tailcfg.NodeView) bool {
+func isPeerEligibleConnector(peer tailcfg.NodeView) bool {
 	if !peer.Valid() || !peer.Hostinfo().Valid() {
 		return false
 	}
@@ -39,7 +40,7 @@ func sortByPreference(ns []tailcfg.NodeView) {
 func PickConnector(nb ipnext.NodeBackend, app appctype.Conn25Attr) []tailcfg.NodeView {
 	appTagsSet := set.SetOf(app.Connectors)
 	matches := nb.AppendMatchingPeers(nil, func(n tailcfg.NodeView) bool {
-		if !isEligibleConnector(n) {
+		if !isPeerEligibleConnector(n) {
 			return false
 		}
 		for _, t := range n.Tags().All() {
@@ -55,7 +56,7 @@ func PickConnector(nb ipnext.NodeBackend, app appctype.Conn25Attr) []tailcfg.Nod
 
 // PickSplitDNSPeers looks at the netmap peers capabilities and finds which peers
 // want to be connectors for which domains.
-func PickSplitDNSPeers(hasCap func(c tailcfg.NodeCapability) bool, self tailcfg.NodeView, peers map[tailcfg.NodeID]tailcfg.NodeView) map[string][]tailcfg.NodeView {
+func PickSplitDNSPeers(hasCap func(c tailcfg.NodeCapability) bool, self tailcfg.NodeView, peers map[tailcfg.NodeID]tailcfg.NodeView, isSelfEligibleConnector bool) map[string][]tailcfg.NodeView {
 	var m map[string][]tailcfg.NodeView
 	if !hasCap(AppConnectorsExperimentalAttrName) {
 		return m
@@ -64,22 +65,43 @@ func PickSplitDNSPeers(hasCap func(c tailcfg.NodeCapability) bool, self tailcfg.
 	if err != nil {
 		return m
 	}
-	tagToDomain := make(map[string][]string)
+
+	// We strip the leading *. from any domains because the OS treats all domains
+	// that we pass to it as wildcard domains, and the OS would treat the * character
+	// as a literal domain component instead of treating it as a wildcard.
+	// We also use a Set to deduplicate the domains we pass to the OS in case removing
+	// the *. prefix resulted in duplicate entries.
+	tagToDomain := make(map[string]set.Set[string])
+	selfTags := set.SetOf(self.Tags().AsSlice())
+	selfRoutedDomains := set.Set[string]{}
 	for _, app := range apps {
+		domains := make(set.Set[string])
+		for _, domain := range app.Domains {
+			domains.Add(strings.ToLower(strings.TrimPrefix(domain, "*.")))
+		}
 		for _, tag := range app.Connectors {
-			tagToDomain[tag] = append(tagToDomain[tag], app.Domains...)
+			if tagToDomain[tag] == nil {
+				tagToDomain[tag] = set.Set[string]{}
+			}
+			tagToDomain[tag].AddSet(domains)
+			if isSelfEligibleConnector && selfTags.Contains(tag) {
+				selfRoutedDomains.AddSet(domains)
+			}
 		}
 	}
 	// NodeIDs are Comparable, and we have a map of NodeID to NodeView anyway, so
 	// use a Set of NodeIDs to deduplicate, and populate into a []NodeView later.
 	var work map[string]set.Set[tailcfg.NodeID]
 	for _, peer := range peers {
-		if !isEligibleConnector(peer) {
+		if !isPeerEligibleConnector(peer) {
 			continue
 		}
 		for _, t := range peer.Tags().All() {
 			domains := tagToDomain[t]
-			for _, domain := range domains {
+			for domain := range domains {
+				if selfRoutedDomains.Contains(domain) {
+					continue
+				}
 				if work[domain] == nil {
 					mak.Set(&work, domain, set.Set[tailcfg.NodeID]{})
 				}

appc/conn25_test.go

@@ -32,6 +32,8 @@ func TestPickSplitDNSPeers(t *testing.T) {
 	appTwoBytes := getBytesForAttr("app2", []string{"a.example.com"}, []string{"tag:two"})
 	appThreeBytes := getBytesForAttr("app3", []string{"woo.b.example.com", "hoo.b.example.com"}, []string{"tag:three1", "tag:three2"})
 	appFourBytes := getBytesForAttr("app4", []string{"woo.b.example.com", "c.example.com"}, []string{"tag:four1", "tag:four2"})
+	appFiveBytes := getBytesForAttr("app5", []string{"*.example.com", "example.com"}, []string{"tag:one"})
+	appSixBytes := getBytesForAttr("app6", []string{"*.Example.com", "EXAMPLE.com", "EXAMPLE.COM"}, []string{"tag:one"})
 
 	makeNodeView := func(id tailcfg.NodeID, name string, tags []string) tailcfg.NodeView {
 		return (&tailcfg.Node{
@@ -48,9 +50,11 @@ func TestPickSplitDNSPeers(t *testing.T) {
 
 	for _, tt := range []struct {
 		name                string
-		want   map[string][]tailcfg.NodeView
 		peers               []tailcfg.NodeView
 		config              []tailcfg.RawMessage
+		isEligibleConnector bool
+		selfTags            []string
+		want                map[string][]tailcfg.NodeView
 	}{
 		{
 			name: "empty",
@@ -111,6 +115,128 @@ func TestPickSplitDNSPeers(t *testing.T) {
 				"c.example.com":     {nvp2, nvp4},
 			},
 		},
+		{
+			name: "self-connector-exclude-self-domains",
+			config: []tailcfg.RawMessage{
+				tailcfg.RawMessage(appOneBytes),
+				tailcfg.RawMessage(appTwoBytes),
+				tailcfg.RawMessage(appThreeBytes),
+				tailcfg.RawMessage(appFourBytes),
+			},
+			peers: []tailcfg.NodeView{
+				nvp1,
+				nvp2,
+				nvp3,
+				nvp4,
+			},
+			isEligibleConnector: true,
+			selfTags:            []string{"tag:three1"},
+			want: map[string][]tailcfg.NodeView{
+				// woo.b.example.com and hoo.b.example.com are covered
+				// by tag:three1, and so is this self-node.
+				// So those domains should not be routed to peers.
+				// woo.b.example.com is also covered by another tag,
+				// but still not included since this connector can route to it.
+				"example.com":   {nvp1},
+				"a.example.com": {nvp3, nvp4},
+				"c.example.com": {nvp2, nvp4},
+			},
+		},
+		{
+			name: "self-eligible-connector-no-matching-tag-include-all-domains",
+			config: []tailcfg.RawMessage{
+				tailcfg.RawMessage(appOneBytes),
+				tailcfg.RawMessage(appTwoBytes),
+				tailcfg.RawMessage(appThreeBytes),
+				tailcfg.RawMessage(appFourBytes),
+			},
+			peers: []tailcfg.NodeView{
+				nvp1,
+				nvp2,
+				nvp3,
+				nvp4,
+			},
+			isEligibleConnector: true,
+			selfTags:            []string{"tag:unrelated"},
+			want: map[string][]tailcfg.NodeView{
+				// Self has prefs set but no tags matching any app,
+				// so no domains are self-routed and all appear.
+				"example.com":       {nvp1},
+				"a.example.com":     {nvp3, nvp4},
+				"woo.b.example.com": {nvp2, nvp3, nvp4},
+				"hoo.b.example.com": {nvp3, nvp4},
+				"c.example.com":     {nvp2, nvp4},
+			},
+		},
+		{
+			name: "self-not-eligible-connector-but-tagged-include-all-domains",
+			config: []tailcfg.RawMessage{
+				tailcfg.RawMessage(appOneBytes),
+				tailcfg.RawMessage(appTwoBytes),
+				tailcfg.RawMessage(appThreeBytes),
+				tailcfg.RawMessage(appFourBytes),
+			},
+			peers: []tailcfg.NodeView{
+				nvp1,
+				nvp2,
+				nvp3,
+				nvp4,
+			},
+			selfTags: []string{"tag:three1"},
+			want: map[string][]tailcfg.NodeView{
+				// Even though this self node has a tag for an app
+				// the prefs don't advertise as connector, so
+				// should still route through other connectors.
+				"example.com":       {nvp1},
+				"a.example.com":     {nvp3, nvp4},
+				"woo.b.example.com": {nvp2, nvp3, nvp4},
+				"hoo.b.example.com": {nvp3, nvp4},
+				"c.example.com":     {nvp2, nvp4},
+			},
+		},
+		{
+			name: "wildcards-are-stripped-and-deduped",
+			config: []tailcfg.RawMessage{
+				tailcfg.RawMessage(appOneBytes),
+				tailcfg.RawMessage(appFiveBytes),
+			},
+			peers: []tailcfg.NodeView{
+				nvp1,
+			},
+			want: map[string][]tailcfg.NodeView{
+				// All the domains should be normalized to example.com
+				"example.com": {nvp1},
+			},
+		},
+		{
+			name: "domains-are-normalized-and-deduped",
+			config: []tailcfg.RawMessage{
+				tailcfg.RawMessage(appSixBytes),
+			},
+			peers: []tailcfg.NodeView{
+				nvp1,
+			},
+			want: map[string][]tailcfg.NodeView{
+				// All the domains should be normalized to example.com
+				"example.com": {nvp1},
+			},
+		},
+		{
+			name: "sub-domains-and-top-domains-do-not-collide",
+			config: []tailcfg.RawMessage{
+				tailcfg.RawMessage(appTwoBytes),
+				tailcfg.RawMessage(appFiveBytes),
+			},
+			peers: []tailcfg.NodeView{
+				nvp1,
+				nvp3,
+			},
+			want: map[string][]tailcfg.NodeView{
+				// The sub.example.com should remain distinct from example.com
+				"example.com":   {nvp1},
+				"a.example.com": {nvp3},
+			},
+		},
 	} {
 		t.Run(tt.name, func(t *testing.T) {
 			selfNode := &tailcfg.Node{}
@@ -119,6 +245,7 @@ func TestPickSplitDNSPeers(t *testing.T) {
 					tailcfg.NodeCapability(AppConnectorsExperimentalAttrName): tt.config,
 				}
 			}
+			selfNode.Tags = append(selfNode.Tags, tt.selfTags...)
 			selfView := selfNode.View()
 			peers := map[tailcfg.NodeID]tailcfg.NodeView{}
 			for _, p := range tt.peers {
@@ -126,7 +253,8 @@ func TestPickSplitDNSPeers(t *testing.T) {
 			}
 			got := PickSplitDNSPeers(func(_ tailcfg.NodeCapability) bool {
 				return true
-			}, selfView, peers)
+			}, selfView, peers, tt.isEligibleConnector)
+
 			if !reflect.DeepEqual(got, tt.want) {
 				t.Fatalf("got %v, want %v", got, tt.want)
 			}

cache_key_test.go

@@ -0,0 +1,57 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+package tailscaleroot
+
+import (
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+
+	"tailscale.com/util/cibuild"
+)
+
+// TestTsgoRevInCacheKey verifies that the Tailscale Go toolchain's git
+// revision (from go.toolchain.rev) is blended into Go build cache keys.
+// Without this, bumping the toolchain to a new commit that doesn't change
+// the Go version number would silently reuse stale cached build artifacts.
+//
+// See https://github.com/tailscale/tailscale/issues/36589.
+func TestTsgoRevInCacheKey(t *testing.T) {
+	goRoot := goEnv(t, "GOROOT")
+	isTsgo := strings.Contains(goRoot, "/.cache/tsgo/")
+	if !cibuild.OnTailscaleCI() && !isTsgo {
+		t.Skip("skipping; not in Tailscale CI and not using the Tailscale Go toolchain")
+	}
+
+	rev := strings.TrimSpace(GoToolchainRev)
+	if rev == "" {
+		t.Fatal("go.toolchain.rev is empty")
+	}
+
+	// Build the small stdlib "errors" package with GODEBUG=gocachehash=1,
+	// which causes cmd/go to log its cache key computations to stderr.
+	cmd := exec.Command("go", "build", "errors")
+	cmd.Env = append(os.Environ(), "GODEBUG=gocachehash=1")
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Fatalf("go build errors failed: %v\n%s", err, out)
+	}
+
+	// The cache key output should contain the toolchain rev alongside the
+	// Go version, e.g.:
+	//   HASH[moduleIndex]: "go1.26.2 dfe2a5fd8ee2e68b08ce5ff259269f50ecadf2f4"
+	if !strings.Contains(string(out), rev) {
+		t.Errorf("go.toolchain.rev %q not found in GODEBUG=gocachehash=1 output:\n%s", rev, out)
+	}
+}
+
+func goEnv(t *testing.T, key string) string {
+	t.Helper()
+	out, err := exec.Command("go", "env", key).Output()
+	if err != nil {
+		t.Fatalf("go env %s: %v", key, err)
+	}
+	return strings.TrimSpace(string(out))
+}

client/local/local.go

@@ -327,6 +327,35 @@ func (lc *Client) WhoIs(ctx context.Context, remoteAddr string) (*apitype.WhoIsR
 	return decodeJSON[*apitype.WhoIsResponse](body)
 }
 
+// WhoIsForService is like [Client.WhoIs] but scopes the returned CapMap to
+// capabilities that apply to the named VIP service. This enables per-service
+// capability resolution on hosts that advertise multiple VIP services.
+func (lc *Client) WhoIsForService(ctx context.Context, remoteAddr string, svcName tailcfg.ServiceName) (*apitype.WhoIsResponse, error) {
+	body, err := lc.get200(ctx, "/localapi/v0/whois?addr="+url.QueryEscape(remoteAddr)+"&svc_name="+url.QueryEscape(string(svcName)))
+	if err != nil {
+		if hs, ok := err.(httpStatusError); ok && hs.HTTPStatus == http.StatusNotFound {
+			return nil, ErrPeerNotFound
+		}
+		return nil, err
+	}
+	return decodeJSON[*apitype.WhoIsResponse](body)
+}
+
+// WhoIsForIP is like [Client.WhoIs] but scopes the returned CapMap to
+// capabilities that apply to the given destination IP. The IP may be a
+// VIP service address, the node's own tailnet address, or any other
+// routable IP the node handles.
+func (lc *Client) WhoIsForIP(ctx context.Context, remoteAddr string, dst netip.Addr) (*apitype.WhoIsResponse, error) {
+	body, err := lc.get200(ctx, "/localapi/v0/whois?addr="+url.QueryEscape(remoteAddr)+"&dst_ip="+url.QueryEscape(dst.String()))
+	if err != nil {
+		if hs, ok := err.(httpStatusError); ok && hs.HTTPStatus == http.StatusNotFound {
+			return nil, ErrPeerNotFound
+		}
+		return nil, err
+	}
+	return decodeJSON[*apitype.WhoIsResponse](body)
+}
+
 // ErrPeerNotFound is returned by [Client.WhoIs], [Client.WhoIsNodeKey] and
 // [Client.WhoIsProto] when a peer is not found.
 var ErrPeerNotFound = errors.New("peer not found")
@@ -607,6 +636,24 @@ func (lc *Client) DebugResultJSON(ctx context.Context, action string) (any, erro
 	return x, nil
 }
 
+// GetDebugResultJSON invokes a debug action and decodes the JSON response
+// into a value of type T. It avoids the marshal/unmarshal roundtrip that
+// callers of [Client.DebugResultJSON] otherwise need to do to get a typed
+// value.
+//
+// These are development tools and subject to change or removal over time.
+func GetDebugResultJSON[T any](ctx context.Context, lc *Client, action string) (T, error) {
+	var v T
+	body, err := lc.send(ctx, "POST", "/localapi/v0/debug?action="+url.QueryEscape(action), 200, nil)
+	if err != nil {
+		return v, fmt.Errorf("error %w: %s", err, body)
+	}
+	if err := json.Unmarshal(body, &v); err != nil {
+		return v, err
+	}
+	return v, nil
+}
+
 // QueryOptionalFeatures queries the optional features supported by the Tailscale daemon.
 func (lc *Client) QueryOptionalFeatures(ctx context.Context) (*apitype.OptionalFeatures, error) {
 	body, err := lc.send(ctx, "POST", "/localapi/v0/debug-optional-features", 200, nil)
@@ -972,6 +1019,19 @@ func (lc *Client) UserDial(ctx context.Context, network, host string, port uint1
 	if res.StatusCode != http.StatusSwitchingProtocols {
 		body, _ := io.ReadAll(res.Body)
 		res.Body.Close()
+		if res.StatusCode == http.StatusOK && res.Header.Get("Dial-Self") == "true" {
+			// Server told us to dial the address ourselves rather than
+			// proxying through the daemon. This happens for non-Tailscale
+			// addresses where the daemon shouldn't dial as root on the
+			// client's behalf. The server provides the resolved address
+			// to avoid a TOCTOU race with DNS re-resolution.
+			addr := res.Header.Get("Dial-Addr")
+			if addr == "" {
+				return nil, errors.New("server returned Dial-Self without Dial-Addr")
+			}
+			var d net.Dialer
+			return d.DialContext(ctx, network, addr)
+		}
 		return nil, fmt.Errorf("unexpected HTTP response: %s, %s", res.Status, body)
 	}
 	// From here on, the underlying net.Conn is ours to use, but there
@@ -1009,6 +1069,44 @@ func (lc *Client) CurrentDERPMap(ctx context.Context) (*tailcfg.DERPMap, error)
 	return &derpMap, nil
 }
 
+// CertDomains returns the list of domains for which the local tailscaled can
+// fetch TLS certificates, equivalent to the DNS.CertDomains field of the
+// current netmap. The returned list is sorted in ascending order, and is
+// empty if no netmap has been received yet.
+func (lc *Client) CertDomains(ctx context.Context) ([]string, error) {
+	body, err := lc.get200(ctx, "/localapi/v0/cert-domains")
+	if err != nil {
+		return nil, err
+	}
+	return decodeJSON[[]string](body)
+}
+
+// DNSConfig returns the [tailcfg.DNSConfig] from the current netmap.
+// It returns an error if no netmap has been received yet.
+// It is intended for callers that need fields like ExtraRecords or CertDomains
+// without pulling the rest of the netmap.
+func (lc *Client) DNSConfig(ctx context.Context) (*tailcfg.DNSConfig, error) {
+	body, err := lc.get200(ctx, "/localapi/v0/dns-config")
+	if err != nil {
+		return nil, err
+	}
+	return decodeJSON[*tailcfg.DNSConfig](body)
+}
+
+// PeerByID returns a peer's current full [tailcfg.Node] looked up by its
+// [tailcfg.NodeID], in O(1) time on the daemon side. It returns an error
+// if no peer with that NodeID is in the current netmap.
+//
+// It is intended for callers that need the latest state of a single peer
+// without fetching the entire netmap.
+func (lc *Client) PeerByID(ctx context.Context, id tailcfg.NodeID) (*tailcfg.Node, error) {
+	body, err := lc.get200(ctx, "/localapi/v0/peer-by-id?id="+strconv.FormatInt(int64(id), 10))
+	if err != nil {
+		return nil, err
+	}
+	return decodeJSON[*tailcfg.Node](body)
+}
+
 // PingOpts contains options for the ping request.
 //
 // The zero value is valid, which means to use defaults.
@@ -1422,3 +1520,13 @@ func (lc *Client) GetAppConnectorRouteInfo(ctx context.Context) (appctype.RouteI
 	}
 	return decodeJSON[appctype.RouteInfo](body)
 }
+
+// GetServices returns the Services visible to this node,
+// including their names, IP addresses, and ports, keyed by service name.
+func (lc *Client) GetServices(ctx context.Context) (map[tailcfg.ServiceName]tailcfg.ServiceDetails, error) {
+	body, err := lc.get200(ctx, "/localapi/v0/services")
+	if err != nil {
+		return nil, err
+	}
+	return decodeJSON[map[tailcfg.ServiceName]tailcfg.ServiceDetails](body)
+}

client/local/local_test.go

@@ -61,6 +61,57 @@ func TestWhoIsPeerNotFound(t *testing.T) {
 	}
 }
 
+func TestUserDialSelf(t *testing.T) {
+	// Start a real TCP listener that the client should dial directly
+	// when the server tells it to dial-self.
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer ln.Close()
+	go func() {
+		for {
+			c, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			c.Write([]byte("hello"))
+			c.Close()
+		}
+	}()
+	targetAddr := ln.Addr().(*net.TCPAddr)
+
+	// Mock LocalAPI server that returns Dial-Self response.
+	nw := nettest.GetNetwork(t)
+	ts := nettest.NewHTTPServer(nw, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Dial-Self", "true")
+		w.Header().Set("Dial-Addr", targetAddr.String())
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer ts.Close()
+
+	lc := &Client{
+		Dial: func(ctx context.Context, network, addr string) (net.Conn, error) {
+			return nw.Dial(ctx, network, ts.Listener.Addr().String())
+		},
+	}
+
+	conn, err := lc.UserDial(context.Background(), "tcp", targetAddr.IP.String(), uint16(targetAddr.Port))
+	if err != nil {
+		t.Fatalf("UserDial: %v", err)
+	}
+	defer conn.Close()
+
+	buf := make([]byte, 5)
+	n, err := conn.Read(buf)
+	if err != nil {
+		t.Fatalf("Read: %v", err)
+	}
+	if got := string(buf[:n]); got != "hello" {
+		t.Errorf("got %q, want %q", got, "hello")
+	}
+}
+
 func TestDeps(t *testing.T) {
 	deptest.DepChecker{
 		BadDeps: map[string]string{

client/local/tailnetlock.go

@@ -117,7 +117,7 @@ func (lc *Client) NetworkLockAffectedSigs(ctx context.Context, keyID tkatype.Key
 	return decodeJSON[[]tkatype.MarshaledSignature](body)
 }
 
-// NetworkLockLog returns up to maxEntries number of changes to network-lock state.
+// NetworkLockLog returns up to maxEntries number of changes to tailnet-lock state.
 func (lc *Client) NetworkLockLog(ctx context.Context, maxEntries int) ([]ipnstate.NetworkLockUpdate, error) {
 	v := url.Values{}
 	v.Set("limit", fmt.Sprint(maxEntries))
@@ -128,7 +128,7 @@ func (lc *Client) NetworkLockLog(ctx context.Context, maxEntries int) ([]ipnstat
 	return decodeJSON[[]ipnstate.NetworkLockUpdate](body)
 }
 
-// NetworkLockForceLocalDisable forcibly shuts down network lock on this node.
+// NetworkLockForceLocalDisable forcibly shuts down tailnet lock on this node.
 func (lc *Client) NetworkLockForceLocalDisable(ctx context.Context) error {
 	// This endpoint expects an empty JSON stanza as the payload.
 	var b bytes.Buffer
@@ -142,7 +142,7 @@ func (lc *Client) NetworkLockForceLocalDisable(ctx context.Context) error {
 	return nil
 }
 
-// NetworkLockVerifySigningDeeplink verifies the network lock deeplink contained
+// NetworkLockVerifySigningDeeplink verifies the tailnet lock deeplink contained
 // in url and returns information extracted from it.
 func (lc *Client) NetworkLockVerifySigningDeeplink(ctx context.Context, url string) (*tka.DeeplinkValidationResult, error) {
 	vr := struct {
@@ -193,7 +193,7 @@ func (lc *Client) NetworkLockSubmitRecoveryAUM(ctx context.Context, aum tka.AUM)
 	return nil
 }
 
-// NetworkLockDisable shuts down network-lock across the tailnet.
+// NetworkLockDisable shuts down tailnet-lock across the tailnet.
 func (lc *Client) NetworkLockDisable(ctx context.Context, secret []byte) error {
 	if _, err := lc.send(ctx, "POST", "/localapi/v0/tka/disable", 200, bytes.NewReader(secret)); err != nil {
 		return fmt.Errorf("error: %w", err)

client/systray/logo.go

@@ -11,6 +11,7 @@ import (
 	"image"
 	"image/color"
 	"image/png"
+	"log"
 	"runtime"
 	"sync"
 	"time"
@@ -204,12 +205,49 @@ var (
 )
 
 var (
-	bg   = color.NRGBA{0, 0, 0, 255}
+	black       = color.NRGBA{0, 0, 0, 255}
-	fg   = color.NRGBA{255, 255, 255, 255}
+	white       = color.NRGBA{255, 255, 255, 255}
-	gray = color.NRGBA{255, 255, 255, 102}
+	darkGray    = color.NRGBA{102, 102, 102, 255}
+	lightGray   = color.NRGBA{153, 153, 153, 255}
 	red         = color.NRGBA{229, 111, 74, 255}
+	transparent = color.NRGBA{}
+
+	// default values to dark theme
+	bg   = black
+	fg   = white
+	gray = darkGray
 )
 
+// SetTheme sets the color theme of the systray icon.
+//
+// Supported themes are:
+//   - dark       - white and gray dots over black background
+//   - dark:nobg  - white and grey dots over transparent background
+//   - light      - black and gray dots over white background
+//   - light:nobg - black and grey dots over transparent background
+func SetTheme(theme string) {
+	switch theme {
+	case "dark":
+		bg = black
+		fg = white
+		gray = darkGray
+	case "dark:nobg":
+		bg = transparent
+		fg = white
+		gray = darkGray
+	case "light":
+		bg = white
+		fg = black
+		gray = lightGray
+	case "light:nobg":
+		bg = transparent
+		fg = black
+		gray = lightGray
+	default:
+		log.Printf("unknown theme: %q", theme)
+	}
+}
+
 // render returns a PNG image of the logo.
 func (logo tsLogo) render() *bytes.Buffer {
 	const borderUnits = 1

client/systray/startup-creator.go

@@ -3,7 +3,6 @@
 
 //go:build cgo || !darwin
 
-// Package systray provides a minimal Tailscale systray application.
 package systray
 
 import (

client/systray/systray.go

@@ -621,11 +621,9 @@ func (menu *Menu) rebuildExitNodeMenu(ctx context.Context) {
 				title += strings.Split(sugg.Name, ".")[0]
 			}
 			menu.exitNodes.AddSeparator()
-			rm := menu.exitNodes.AddSubMenuItemCheckbox(title, "", false)
+			active := recommendedIsActive(status, sugg.ID, sugg.Location.CountryCode(), sugg.Location.City())
+			rm := menu.exitNodes.AddSubMenuItemCheckbox(title, "", active)
 			setExitNodeOnClick(rm, sugg.ID)
-			if status.ExitNodeStatus != nil && sugg.ID == status.ExitNodeStatus.ID {
-				rm.Check()
-			}
 		}
 	}
 
@@ -647,13 +645,11 @@ func (menu *Menu) rebuildExitNodeMenu(ctx context.Context) {
 			if !ps.Online {
 				name += " (offline)"
 			}
-			sm := menu.exitNodes.AddSubMenuItemCheckbox(name, "", false)
+			active := status.ExitNodeStatus != nil && ps.ID == status.ExitNodeStatus.ID
+			sm := menu.exitNodes.AddSubMenuItemCheckbox(name, "", active)
 			if !ps.Online {
 				sm.Disable()
 			}
-			if status.ExitNodeStatus != nil && ps.ID == status.ExitNodeStatus.ID {
-				sm.Check()
-			}
 			setExitNodeOnClick(sm, ps.ID)
 		}
 	}
@@ -743,6 +739,30 @@ func (mc *mvCountry) sortedCities() []*mvCity {
 	return cities
 }
 
+// recommendedIsActive reports whether the suggested exit node corresponds to
+// the currently active exit node in status.
+func recommendedIsActive(status *ipnstate.Status, suggID tailcfg.StableNodeID, suggCountry, suggCity string) bool {
+	if status == nil || status.ExitNodeStatus == nil || status.ExitNodeStatus.ID.IsZero() {
+		return false
+	}
+	if suggID == status.ExitNodeStatus.ID {
+		return true
+	}
+	if suggCountry == "" || suggCity == "" {
+		return false
+	}
+	for _, p := range status.Peer {
+		if p.ID != status.ExitNodeStatus.ID {
+			continue
+		}
+		if loc := p.Location; loc != nil && loc.CountryCode == suggCountry && loc.City == suggCity {
+			return true
+		}
+		return false
+	}
+	return false
+}
+
 // countryFlag takes a 2-character ASCII string and returns the corresponding emoji flag.
 // It returns the empty string on error.
 func countryFlag(code string) string {

client/systray/systray_test.go

@@ -0,0 +1,120 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build cgo || !darwin
+
+package systray
+
+import (
+	"testing"
+
+	"tailscale.com/ipn/ipnstate"
+	"tailscale.com/tailcfg"
+	"tailscale.com/types/key"
+)
+
+func TestRecommendedIsActive(t *testing.T) {
+	t.Parallel()
+
+	const (
+		activeID = tailcfg.StableNodeID("active")
+		suggID   = tailcfg.StableNodeID("suggestion")
+	)
+	usNYC := &tailcfg.Location{CountryCode: "US", City: "New York"}
+	usCHI := &tailcfg.Location{CountryCode: "US", City: "Chicago"}
+	seSTO := &tailcfg.Location{CountryCode: "SE", City: "Stockholm"}
+
+	statusWith := func(activePeer *ipnstate.PeerStatus) *ipnstate.Status {
+		s := &ipnstate.Status{
+			ExitNodeStatus: &ipnstate.ExitNodeStatus{ID: activeID},
+		}
+		if activePeer != nil {
+			s.Peer = map[key.NodePublic]*ipnstate.PeerStatus{{}: activePeer}
+		}
+		return s
+	}
+
+	tests := []struct {
+		name        string
+		status      *ipnstate.Status
+		suggID      tailcfg.StableNodeID
+		suggCountry string
+		suggCity    string
+		isActive    bool
+	}{
+		{
+			name:   "nil_status",
+			status: nil,
+			suggID: suggID,
+		},
+		{
+			name:   "no_exit_node",
+			status: &ipnstate.Status{},
+			suggID: suggID,
+		},
+		{
+			name:   "exit_node_id_is_zero",
+			status: &ipnstate.Status{ExitNodeStatus: &ipnstate.ExitNodeStatus{}},
+			suggID: suggID,
+		},
+		{
+			name:        "exact_id_match_short-circuits",
+			status:      statusWith(&ipnstate.PeerStatus{ID: activeID, Location: usCHI}),
+			suggID:      activeID,
+			suggCountry: "US",
+			suggCity:    "New York",
+			isActive:    true,
+		},
+		{
+			name:        "id_mismatch_but_same_city",
+			status:      statusWith(&ipnstate.PeerStatus{ID: activeID, Location: usNYC}),
+			suggID:      suggID,
+			suggCountry: "US",
+			suggCity:    "New York",
+			isActive:    true,
+		},
+		{
+			name:        "different_city",
+			status:      statusWith(&ipnstate.PeerStatus{ID: activeID, Location: usCHI}),
+			suggID:      suggID,
+			suggCountry: "US",
+			suggCity:    "New York",
+		},
+		{
+			name:        "different_country",
+			status:      statusWith(&ipnstate.PeerStatus{ID: activeID, Location: seSTO}),
+			suggID:      suggID,
+			suggCountry: "US",
+			suggCity:    "New York",
+		},
+		{
+			name:   "id_mismatch_suggestion_has_no_location",
+			status: statusWith(&ipnstate.PeerStatus{ID: activeID, Location: usNYC}),
+			suggID: suggID,
+		},
+		{
+			name:        "id_mismatch_active_peer_has_no_location",
+			status:      statusWith(&ipnstate.PeerStatus{ID: activeID}),
+			suggID:      suggID,
+			suggCountry: "US",
+			suggCity:    "New York",
+		},
+		{
+			name:        "active_peer_not_in_status",
+			status:      statusWith(nil),
+			suggID:      suggID,
+			suggCountry: "US",
+			suggCity:    "New York",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			isExitNodeActive := recommendedIsActive(tt.status, tt.suggID, tt.suggCountry, tt.suggCity)
+			if isExitNodeActive != tt.isActive {
+				t.Errorf("recommendedIsActive; got %v, want %v", isExitNodeActive, tt.isActive)
+			}
+		})
+	}
+}

client/web/web.go

@@ -35,8 +35,10 @@ import (
 	"tailscale.com/net/netutil"
 	"tailscale.com/net/tsaddr"
 	"tailscale.com/tailcfg"
+	"tailscale.com/tsweb"
 	"tailscale.com/types/logger"
 	"tailscale.com/types/views"
+	"tailscale.com/util/ctxkey"
 	"tailscale.com/util/httpm"
 	"tailscale.com/util/syspolicy/policyclient"
 	"tailscale.com/version"
@@ -527,45 +529,40 @@ func (s *Server) serveLoginAPI(w http.ResponseWriter, r *http.Request) {
 	}
 }
 
-type apiHandler[data any] struct {
+// handleJSON manages decoding the request's body JSON as data and passing it
-	s *Server
+// on to the provided handler function.
-	w http.ResponseWriter
+func handleJSON[data any](h func(ctx context.Context, data data) error) http.HandlerFunc {
-	r *http.Request
+	return func(w http.ResponseWriter, r *http.Request) {
-
+		defer r.Body.Close()
-	// permissionCheck allows for defining whether a requesting peer's
+		var body data
-	// capabilities grant them access to make the given data update.
+		if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
-	// If permissionCheck reports false, the request fails as unauthorized.
+			http.Error(w, err.Error(), http.StatusInternalServerError)
-	permissionCheck func(data data, peer peerCapabilities) bool
+			return
-}
+		}
-
+		if err := h(r.Context(), body); err != nil {
-// newHandler constructs a new api handler which restricts the given request
+			if httpErr, ok := errors.AsType[tsweb.HTTPError](err); ok {
-// to the specified permission check. If the permission check fails for
+				tsweb.WriteHTTPError(w, r, httpErr)
-// the peer associated with the request, an unauthorized error is returned
+			} else {
-// to the client.
+				http.Error(w, err.Error(), http.StatusInternalServerError)
-func newHandler[data any](s *Server, w http.ResponseWriter, r *http.Request, permissionCheck func(data data, peer peerCapabilities) bool) *apiHandler[data] {
+			}
-	return &apiHandler[data]{
+			return
-		s:               s,
+		}
-		w:               w,
+		w.WriteHeader(http.StatusOK)
-		r:               r,
-		permissionCheck: permissionCheck,
 	}
 }
 
-// alwaysAllowed can be passed as the permissionCheck argument to newHandler
+var contextKeyPeer = ctxkey.New("peer-capabilities", peerCapabilities{})
-// for requests that are always allowed to complete regardless of a peer's
-// capabilities.
-func alwaysAllowed[data any](_ data, _ peerCapabilities) bool { return true }
 
-func (a *apiHandler[data]) getPeer() (peerCapabilities, error) {
+func (s *Server) setPeer(r *http.Request) (*http.Request, error) {
 	// TODO(tailscale/corp#16695,sonia): We also call StatusWithoutPeers and
 	// WhoIs when originally checking for a session from authorizeRequest.
 	// Would be nice if we could pipe those through to here so we don't end
 	// up having to re-call them to grab the peer capabilities.
-	status, err := a.s.lc.StatusWithoutPeers(a.r.Context())
+	status, err := s.lc.StatusWithoutPeers(r.Context())
 	if err != nil {
 		return nil, err
 	}
-	whois, err := a.s.lc.WhoIs(a.r.Context(), a.r.RemoteAddr)
+	whois, err := s.lc.WhoIs(r.Context(), r.RemoteAddr)
 	if err != nil {
 		return nil, err
 	}
@@ -573,56 +570,11 @@ func (a *apiHandler[data]) getPeer() (peerCapabilities, error) {
 	if err != nil {
 		return nil, err
 	}
-	return peer, nil
+	return r.WithContext(contextKeyPeer.WithValue(r.Context(), peer)), nil
 }
 
-type noBodyData any // empty type, for use from serveAPI for endpoints with empty body
+func (s *Server) getPeer(ctx context.Context) peerCapabilities {
-
+	return contextKeyPeer.Value(ctx)
-// handle runs the given handler if the source peer satisfies the
-// constraints for running this request.
-//
-// handle is expected for use when `data` type is empty, or set to
-// `noBodyData` in practice. For requests that expect JSON body data
-// to be attached, use handleJSON instead.
-func (a *apiHandler[data]) handle(h http.HandlerFunc) {
-	peer, err := a.getPeer()
-	if err != nil {
-		http.Error(a.w, err.Error(), http.StatusInternalServerError)
-		return
-	}
-	var body data // not used
-	if !a.permissionCheck(body, peer) {
-		http.Error(a.w, "not allowed", http.StatusUnauthorized)
-		return
-	}
-	h(a.w, a.r)
-}
-
-// handleJSON manages decoding the request's body JSON and passing
-// it on to the provided function if the source peer satisfies the
-// constraints for running this request.
-func (a *apiHandler[data]) handleJSON(h func(ctx context.Context, data data) error) {
-	defer a.r.Body.Close()
-	var body data
-	if err := json.NewDecoder(a.r.Body).Decode(&body); err != nil {
-		http.Error(a.w, err.Error(), http.StatusInternalServerError)
-		return
-	}
-	peer, err := a.getPeer()
-	if err != nil {
-		http.Error(a.w, err.Error(), http.StatusInternalServerError)
-		return
-	}
-	if !a.permissionCheck(body, peer) {
-		http.Error(a.w, "not allowed", http.StatusUnauthorized)
-		return
-	}
-
-	if err := h(a.r.Context(), body); err != nil {
-		http.Error(a.w, err.Error(), http.StatusInternalServerError)
-		return
-	}
-	a.w.WriteHeader(http.StatusOK)
 }
 
 // serveAPI serves requests for the web client api.
@@ -637,67 +589,44 @@ func (s *Server) serveAPI(w http.ResponseWriter, r *http.Request) {
 		}
 	}
 
+	var err error
+	r, err = s.setPeer(r)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusUnauthorized)
+		return
+	}
+
 	path := strings.TrimPrefix(r.URL.Path, "/api")
 	switch {
 	case path == "/data" && r.Method == httpm.GET:
-		newHandler[noBodyData](s, w, r, alwaysAllowed).
+		s.serveGetNodeData(w, r)
-			handle(s.serveGetNodeData)
 		return
 	case path == "/exit-nodes" && r.Method == httpm.GET:
-		newHandler[noBodyData](s, w, r, alwaysAllowed).
+		s.serveGetExitNodes(w, r)
-			handle(s.serveGetExitNodes)
 		return
 	case path == "/routes" && r.Method == httpm.POST:
-		peerAllowed := func(d postRoutesRequest, p peerCapabilities) bool {
+		handleJSON[postRoutesRequest](s.servePostRoutes)(w, r)
-			if d.SetExitNode && !p.canEdit(capFeatureExitNodes) {
-				return false
-			} else if d.SetRoutes && !p.canEdit(capFeatureSubnets) {
-				return false
-			}
-			return true
-		}
-		newHandler[postRoutesRequest](s, w, r, peerAllowed).
-			handleJSON(s.servePostRoutes)
 		return
 	case path == "/device-details-click" && r.Method == httpm.POST:
-		newHandler[noBodyData](s, w, r, alwaysAllowed).
+		s.serveDeviceDetailsClick(w, r)
-			handle(s.serveDeviceDetailsClick)
 		return
 	case path == "/local/v0/logout" && r.Method == httpm.POST:
-		peerAllowed := func(_ noBodyData, peer peerCapabilities) bool {
+		s.proxyRequestToLocalAPI(w, r)
-			return peer.canEdit(capFeatureAccount)
-		}
-		newHandler[noBodyData](s, w, r, peerAllowed).
-			handle(s.proxyRequestToLocalAPI)
 		return
 	case path == "/local/v0/prefs" && r.Method == httpm.PATCH:
-		peerAllowed := func(data maskedPrefs, peer peerCapabilities) bool {
+		handleJSON[maskedPrefs](s.serveUpdatePrefs)(w, r)
-			if data.RunSSHSet && !peer.canEdit(capFeatureSSH) {
-				return false
-			}
-			return true
-		}
-		newHandler[maskedPrefs](s, w, r, peerAllowed).
-			handleJSON(s.serveUpdatePrefs)
 		return
 	case path == "/local/v0/update/check" && r.Method == httpm.GET:
-		newHandler[noBodyData](s, w, r, alwaysAllowed).
+		s.proxyRequestToLocalAPI(w, r)
-			handle(s.proxyRequestToLocalAPI)
 		return
 	case path == "/local/v0/update/check" && r.Method == httpm.POST:
-		peerAllowed := func(_ noBodyData, peer peerCapabilities) bool {
+		s.proxyRequestToLocalAPI(w, r)
-			return peer.canEdit(capFeatureAccount)
-		}
-		newHandler[noBodyData](s, w, r, peerAllowed).
-			handle(s.proxyRequestToLocalAPI)
 		return
 	case path == "/local/v0/update/progress" && r.Method == httpm.POST:
-		newHandler[noBodyData](s, w, r, alwaysAllowed).
+		s.proxyRequestToLocalAPI(w, r)
-			handle(s.proxyRequestToLocalAPI)
 		return
 	case path == "/local/v0/upload-client-metrics" && r.Method == httpm.POST:
-		newHandler[noBodyData](s, w, r, alwaysAllowed).
+		s.proxyRequestToLocalAPI(w, r)
-			handle(s.proxyRequestToLocalAPI)
 		return
 	}
 	http.Error(w, "invalid endpoint", http.StatusNotFound)
@@ -1122,6 +1051,11 @@ type maskedPrefs struct {
 }
 
 func (s *Server) serveUpdatePrefs(ctx context.Context, prefs maskedPrefs) error {
+	peer := s.getPeer(ctx)
+	if prefs.RunSSHSet && !peer.canEdit(capFeatureSSH) {
+		return tsweb.Error(http.StatusUnauthorized, "RunSSHSet not allowed", nil)
+	}
+
 	_, err := s.lc.EditPrefs(ctx, &ipn.MaskedPrefs{
 		RunSSHSet: prefs.RunSSHSet,
 		Prefs: ipn.Prefs{
@@ -1140,6 +1074,17 @@ type postRoutesRequest struct {
 }
 
 func (s *Server) servePostRoutes(ctx context.Context, data postRoutesRequest) error {
+	if !data.SetExitNode && !data.SetRoutes {
+		return tsweb.Error(http.StatusBadRequest, "must specify SetExitNode or SetRoutes", nil)
+	}
+	peer := s.getPeer(ctx)
+	if data.SetExitNode && !peer.canEdit(capFeatureExitNodes) {
+		return tsweb.Error(http.StatusUnauthorized, "SetExitNode not allowed", nil)
+	}
+	if data.SetRoutes && !peer.canEdit(capFeatureSubnets) {
+		return tsweb.Error(http.StatusUnauthorized, "SetRoutes not allowed", nil)
+	}
+
 	prefs, err := s.lc.GetPrefs(ctx)
 	if err != nil {
 		return err
@@ -1153,13 +1098,14 @@ func (s *Server) servePostRoutes(ctx context.Context, data postRoutesRequest) er
 		}
 		currNonExitRoutes = append(currNonExitRoutes, r.String())
 	}
-	// Set non-edited fields to their current values.
+	// For each group of fields not being set, preserve the current prefs.
-	if data.SetExitNode {
+	if !data.SetExitNode {
-		data.AdvertiseRoutes = currNonExitRoutes
-	} else if data.SetRoutes {
 		data.AdvertiseExitNode = currAdvertisingExitNode
 		data.UseExitNode = prefs.ExitNodeID
 	}
+	if !data.SetRoutes {
+		data.AdvertiseRoutes = currNonExitRoutes
+	}
 
 	// Calculate routes.
 	routesStr := strings.Join(data.AdvertiseRoutes, ",")
@@ -1336,6 +1282,19 @@ func (s *Server) proxyRequestToLocalAPI(w http.ResponseWriter, r *http.Request)
 		return
 	}
 
+	switch path {
+	case "/v0/logout":
+		if !s.getPeer(r.Context()).canEdit(capFeatureAccount) {
+			http.Error(w, "not allowed", http.StatusUnauthorized)
+			return
+		}
+	case "/v0/update/check":
+		if r.Method == httpm.POST && !s.getPeer(r.Context()).canEdit(capFeatureAccount) {
+			http.Error(w, "not allowed", http.StatusUnauthorized)
+			return
+		}
+	}
+
 	localAPIURL := "http://" + apitype.LocalAPIHost + "/localapi" + path
 	req, err := http.NewRequestWithContext(r.Context(), r.Method, localAPIURL, r.Body)
 	if err != nil {

client/web/web_test.go

@@ -191,7 +191,7 @@ func TestServeAPI(t *testing.T) {
 		reqBody:   "{\"setExitNode\":true}",
 		tests: []requestTest{{
 			remoteIP:     remoteIPWithNoCapabilities,
-			wantResponse: "not allowed",
+			wantResponse: "SetExitNode not allowed",
 			wantStatus:   http.StatusUnauthorized,
 		}, {
 			remoteIP:   remoteIPWithAllCapabilities,
@@ -204,7 +204,7 @@ func TestServeAPI(t *testing.T) {
 		reqContentType: "application/json",
 		tests: []requestTest{{
 			remoteIP:     remoteIPWithNoCapabilities,
-			wantResponse: "not allowed",
+			wantResponse: "RunSSHSet not allowed",
 			wantStatus:   http.StatusUnauthorized,
 		}, {
 			remoteIP:   remoteIPWithAllCapabilities,
@@ -1604,3 +1604,149 @@ func TestCSRFProtect(t *testing.T) {
 		})
 	}
 }
+
+func TestServePostRoutes(t *testing.T) {
+	existingExitNodeID := tailcfg.StableNodeID("existing-exit-node")
+	existingRoute := netip.MustParsePrefix("192.168.1.0/24")
+
+	existingPrefs := &ipn.Prefs{
+		ExitNodeID:      existingExitNodeID,
+		AdvertiseRoutes: []netip.Prefix{existingRoute},
+	}
+
+	tests := []struct {
+		name           string
+		data           postRoutesRequest
+		peerCaps       peerCapabilities
+		wantErr        bool
+		wantEditPrefs  bool // whether EditPrefs (PATCH /prefs) should be called
+		wantExitNodeID tailcfg.StableNodeID
+		wantRoutes     []netip.Prefix
+	}{
+		{
+			name:          "empty-request",
+			data:          postRoutesRequest{},
+			peerCaps:      peerCapabilities{capFeatureExitNodes: true, capFeatureSubnets: true},
+			wantErr:       true,
+			wantEditPrefs: false,
+		},
+		{
+			name: "SetExitNode-only",
+			data: postRoutesRequest{
+				SetExitNode: true,
+				UseExitNode: "new-exit-node",
+			},
+			peerCaps:       peerCapabilities{capFeatureExitNodes: true, capFeatureSubnets: true},
+			wantEditPrefs:  true,
+			wantExitNodeID: "new-exit-node",
+			wantRoutes:     []netip.Prefix{existingRoute},
+		},
+		{
+			name: "SetExitNode-not-allowed",
+			data: postRoutesRequest{
+				SetExitNode: true,
+				UseExitNode: "new-exit-node",
+			},
+			peerCaps: peerCapabilities{capFeatureSubnets: true},
+			wantErr:  true,
+		},
+		{
+			name: "SetRoutes-only",
+			data: postRoutesRequest{
+				SetRoutes:       true,
+				AdvertiseRoutes: []string{"10.0.0.0/8"},
+			},
+			peerCaps:       peerCapabilities{capFeatureExitNodes: true, capFeatureSubnets: true},
+			wantEditPrefs:  true,
+			wantExitNodeID: existingExitNodeID,
+			wantRoutes:     []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")},
+		},
+		{
+			name: "SetRoutes-not-allowed",
+			data: postRoutesRequest{
+				SetRoutes:       true,
+				AdvertiseRoutes: []string{"10.0.0.0/8"},
+			},
+			peerCaps: peerCapabilities{capFeatureExitNodes: true},
+			wantErr:  true,
+		},
+		{
+			name: "SetExitNode-and-SetRoutes",
+			data: postRoutesRequest{
+				SetExitNode:     true,
+				SetRoutes:       true,
+				UseExitNode:     "new-exit-node",
+				AdvertiseRoutes: []string{"10.0.0.0/8"},
+			},
+			peerCaps:       peerCapabilities{capFeatureExitNodes: true, capFeatureSubnets: true},
+			wantEditPrefs:  true,
+			wantExitNodeID: "new-exit-node",
+			wantRoutes:     []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var gotPrefs *ipn.MaskedPrefs
+
+			lal := memnet.Listen("local-tailscaled.sock:80")
+			defer lal.Close()
+
+			localapi := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				if r.URL.Path != "/localapi/v0/prefs" {
+					t.Errorf("unexpected localapi call to %q", r.URL.Path)
+					http.Error(w, "unexpected localapi call", http.StatusInternalServerError)
+					return
+				}
+				switch r.Method {
+				case httpm.GET:
+					writeJSON(w, existingPrefs)
+				case httpm.PATCH:
+					var mp ipn.MaskedPrefs
+					if err := json.NewDecoder(r.Body).Decode(&mp); err != nil {
+						http.Error(w, err.Error(), http.StatusBadRequest)
+						return
+					}
+					gotPrefs = &mp
+					writeJSON(w, gotPrefs.Prefs)
+				default:
+					t.Errorf("unexpected method %q on /prefs", r.Method)
+					http.Error(w, "unexpected method", http.StatusMethodNotAllowed)
+				}
+			})}
+			defer localapi.Close()
+			go localapi.Serve(lal)
+
+			s := &Server{
+				mode: ManageServerMode,
+				lc:   &local.Client{Dial: lal.Dial},
+			}
+
+			ctx := contextKeyPeer.WithValue(t.Context(), tt.peerCaps)
+			err := s.servePostRoutes(ctx, tt.data)
+
+			if tt.wantErr {
+				if err == nil {
+					t.Error("wanted error, got nil")
+				}
+				if gotPrefs != nil {
+					t.Error("EditPrefs should not have been called on error")
+				}
+				return
+			}
+			if err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+
+			if gotPrefs == nil {
+				t.Fatal("expected EditPrefs to be called")
+			}
+			if diff := cmp.Diff(tt.wantExitNodeID, gotPrefs.ExitNodeID); diff != "" {
+				t.Errorf("ExitNodeID mismatch (-want +got):\n%s", diff)
+			}
+			if diff := cmp.Diff(tt.wantRoutes, gotPrefs.AdvertiseRoutes, cmp.Comparer(func(a, b netip.Prefix) bool { return a.Compare(b) == 0 })); diff != "" {
+				t.Errorf("AdvertiseRoutes mismatch (-want +got):\n%s", diff)
+			}
+		})
+	}
+}

clientupdate/clientupdate_windows.go

@@ -38,12 +38,12 @@ const (
 	updaterPrefix = "tailscale-updater"
 )
 
-func makeSelfCopy() (origPathExe, tmpPathExe string, err error) {
+func makeCmdTailscaleCopy() (origPathExe, tmpPathExe string, err error) {
-	selfExe, err := os.Executable()
+	srcExe, err := findCmdTailscale()
 	if err != nil {
 		return "", "", err
 	}
-	f, err := os.Open(selfExe)
+	f, err := os.Open(srcExe)
 	if err != nil {
 		return "", "", err
 	}
@@ -59,7 +59,25 @@ func makeSelfCopy() (origPathExe, tmpPathExe string, err error) {
 		f2.Close()
 		return "", "", err
 	}
-	return selfExe, f2.Name(), f2.Close()
+	return srcExe, f2.Name(), f2.Close()
+}
+
+// findCmdTailscale returns the path to the binary that should be copied for the update
+// re-execution. The copy is re-executed with "update" as a subcommand, so it must be
+// a binary that handles "update" (ie tailscale.exe, not tailscaled.exe)
+func findCmdTailscale() (string, error) {
+	selfExe, err := os.Executable()
+	if err != nil {
+		return "", err
+	}
+	if strings.EqualFold(filepath.Base(selfExe), "tailscale.exe") {
+		return selfExe, nil
+	}
+	ts := filepath.Join(filepath.Dir(selfExe), "tailscale.exe")
+	if _, err := os.Stat(ts); err != nil {
+		return "", fmt.Errorf("cannot find tailscale.exe alongside %s: %w", selfExe, err)
+	}
+	return ts, nil
 }
 
 func markTempFileWindows(name string) error {
@@ -159,14 +177,14 @@ you can run the command prompt as Administrator one of these ways:
 
 	up.Logf("making tailscale.exe copy to switch to...")
 	up.cleanupOldDownloads(filepath.Join(os.TempDir(), updaterPrefix+"-*.exe"))
-	_, selfCopy, err := makeSelfCopy()
+	_, cmdTailscaleCopy, err := makeCmdTailscaleCopy()
 	if err != nil {
 		return err
 	}
-	defer os.Remove(selfCopy)
+	defer os.Remove(cmdTailscaleCopy)
 	up.Logf("running tailscale.exe copy for final install...")
 
-	cmd := exec.Command(selfCopy, "update")
+	cmd := exec.Command(cmdTailscaleCopy, "update")
 	cmd.Env = append(os.Environ(), winMSIEnv+"="+msiTarget, winVersionEnv+"="+ver)
 	cmd.Stdout = up.Stderr
 	cmd.Stderr = up.Stderr

cmd/cloner/cloner.go

@@ -143,25 +143,9 @@ func gen(buf *bytes.Buffer, it *codegen.ImportTracker, typ *types.Named) {
 				writef("if src.%s != nil {", fname)
 				writef("dst.%s = make([]%s, len(src.%s))", fname, n, fname)
 				writef("for i := range dst.%s {", fname)
-				if ptr, isPtr := ft.Elem().(*types.Pointer); isPtr {
+				writeSliceElemClone(writef, ft.Elem(),
-					writef("if src.%s[i] == nil { dst.%s[i] = nil } else {", fname, fname)
+					fmt.Sprintf("src.%s[i]", fname),
-					if codegen.ContainsPointers(ptr.Elem()) {
+					fmt.Sprintf("dst.%s[i]", fname))
-						if _, isIface := ptr.Elem().Underlying().(*types.Interface); isIface {
-							writef("\tdst.%s[i] = new((*src.%s[i]).Clone())", fname, fname)
-						} else {
-							writef("\tdst.%s[i] = src.%s[i].Clone()", fname, fname)
-						}
-					} else {
-						writef("\tdst.%s[i] = new(*src.%s[i])", fname, fname)
-					}
-					writef("}")
-				} else if ft.Elem().String() == "encoding/json.RawMessage" {
-					writef("\tdst.%s[i] = append(src.%s[i][:0:0], src.%s[i]...)", fname, fname, fname)
-				} else if _, isIface := ft.Elem().Underlying().(*types.Interface); isIface {
-					writef("\tdst.%s[i] = src.%s[i].Clone()", fname, fname)
-				} else {
-					writef("\tdst.%s[i] = *src.%s[i].Clone()", fname, fname)
-				}
 				writef("}")
 				writef("}")
 			} else {
@@ -189,11 +173,28 @@ func gen(buf *bytes.Buffer, it *codegen.ImportTracker, typ *types.Named) {
 				n := it.QualifiedName(sliceType.Elem())
 				writef("if dst.%s != nil {", fname)
 				writef("\tdst.%s = map[%s]%s{}", fname, it.QualifiedName(ft.Key()), it.QualifiedName(elem))
+				if codegen.ContainsPointers(sliceType.Elem()) {
+					writef("\tfor k, sv := range src.%s {", fname)
+					writef("\t\tif sv == nil {")
+					writef("\t\t\tdst.%s[k] = nil", fname)
+					writef("\t\t\tcontinue")
+					writef("\t\t}")
+					writef("\t\tdst.%s[k] = make([]%s, len(sv))", fname, n)
+					writef("\t\tfor i := range sv {")
+					innerWritef := func(format string, args ...any) {
+						writef("\t\t"+format, args...)
+					}
+					writeSliceElemClone(innerWritef, sliceType.Elem(),
+						"sv[i]", fmt.Sprintf("dst.%s[k][i]", fname))
+					writef("\t\t}")
+					writef("\t}")
+				} else {
 					writef("\tfor k := range src.%s {", fname)
 					// use zero-length slice instead of nil to ensure
 					// the key is always copied.
 					writef("\t\tdst.%s[k] = append([]%s{}, src.%s[k]...)", fname, n, fname)
 					writef("\t}")
+				}
 				writef("}")
 			} else if codegen.IsViewType(elem) || !codegen.ContainsPointers(elem) {
 				// If the map values are view types (which are
@@ -242,6 +243,31 @@ func gen(buf *bytes.Buffer, it *codegen.ImportTracker, typ *types.Named) {
 	buf.Write(codegen.AssertStructUnchanged(t, name, typeParams, "Clone", it))
 }
 
+// writeSliceElemClone generates code to deep-clone a single slice element
+// from srcExpr to dstExpr. It handles pointer, json.RawMessage, interface,
+// and named struct element types.
+func writeSliceElemClone(writef func(string, ...any), elemType types.Type, srcExpr, dstExpr string) {
+	if ptr, isPtr := elemType.(*types.Pointer); isPtr {
+		writef("if %s == nil { %s = nil } else {", srcExpr, dstExpr)
+		if codegen.ContainsPointers(ptr.Elem()) {
+			if _, isIface := ptr.Elem().Underlying().(*types.Interface); isIface {
+				writef("\t%s = new((*%s).Clone())", dstExpr, srcExpr)
+			} else {
+				writef("\t%s = %s.Clone()", dstExpr, srcExpr)
+			}
+		} else {
+			writef("\t%s = new(*%s)", dstExpr, srcExpr)
+		}
+		writef("}")
+	} else if elemType.String() == "encoding/json.RawMessage" {
+		writef("%s = append(%s[:0:0], %s...)", dstExpr, srcExpr, srcExpr)
+	} else if _, isIface := elemType.Underlying().(*types.Interface); isIface {
+		writef("%s = %s.Clone()", dstExpr, srcExpr)
+	} else {
+		writef("%s = *%s.Clone()", dstExpr, srcExpr)
+	}
+}
+
 // hasBasicUnderlying reports true when typ.Underlying() is a slice or a map.
 func hasBasicUnderlying(typ types.Type) bool {
 	switch typ.Underlying().(type) {

cmd/cloner/cloner_test.go

@@ -7,6 +7,7 @@ import (
 	"reflect"
 	"testing"
 
+	"github.com/google/go-cmp/cmp"
 	"tailscale.com/cmd/cloner/clonerex"
 )
 
@@ -182,6 +183,46 @@ func TestNamedMapContainer(t *testing.T) {
 	}
 }
 
+func TestMapSlicePointerContainer(t *testing.T) {
+	num := 42
+	orig := &clonerex.MapSlicePointerContainer{
+		Routes: map[string][]*clonerex.SliceContainer{
+			"route1": {
+				{Slice: []*int{&num}},
+				{Slice: []*int{&num, &num}},
+			},
+			"route2": {
+				{Slice: []*int{&num}},
+			},
+		},
+	}
+
+	cloned := orig.Clone()
+	if !reflect.DeepEqual(orig, cloned) {
+		t.Errorf("Clone() = %v, want %v", cloned, orig)
+	}
+
+	// Mutate cloned.Routes pointer values
+	*cloned.Routes["route1"][0].Slice[0] = 999
+	if *orig.Routes["route1"][0].Slice[0] == 999 {
+		t.Errorf("Clone() aliased memory in Routes: original was modified")
+	}
+}
+
+func TestMapSlicePointerContainerNilValue(t *testing.T) {
+	num := 7
+	orig := &clonerex.MapSlicePointerContainer{
+		Routes: map[string][]*clonerex.SliceContainer{
+			"nil-value": nil,
+			"non-nil":   {{Slice: []*int{&num}}},
+		},
+	}
+	cloned := orig.Clone()
+	if diff := cmp.Diff(orig.Routes, cloned.Routes); diff != "" {
+		t.Errorf("Clone() Routes mismatch (-orig +cloned):\n%s", diff)
+	}
+}
+
 func TestDeeplyNestedMap(t *testing.T) {
 	num := 123
 	orig := &clonerex.DeeplyNestedMap{

cmd/cloner/clonerex/clonerex.go

@@ -1,7 +1,7 @@
 // Copyright (c) Tailscale Inc & contributors
 // SPDX-License-Identifier: BSD-3-Clause
 
-//go:generate go run tailscale.com/cmd/cloner  -clonefunc=true -type SliceContainer,InterfaceContainer,MapWithPointers,DeeplyNestedMap,NamedMapContainer
+//go:generate go run tailscale.com/cmd/cloner  -clonefunc=true -type SliceContainer,InterfaceContainer,MapWithPointers,DeeplyNestedMap,NamedMapContainer,MapSlicePointerContainer
 
 // Package clonerex is an example package for the cloner tool.
 package clonerex
@@ -60,6 +60,13 @@ type NamedMapContainer struct {
 	Attrs NamedMap
 }
 
+// MapSlicePointerContainer has a map whose values are slices of pointers.
+// This tests that the cloner deep-clones the pointer elements in the slice,
+// not just the slice itself (which would leave aliased pointers).
+type MapSlicePointerContainer struct {
+	Routes map[string][]*SliceContainer
+}
+
 // DeeplyNestedMap tests arbitrary depth of map nesting (3+ levels)
 type DeeplyNestedMap struct {
 	ThreeLevels map[string]map[string]map[string]int

cmd/cloner/clonerex/clonerex_clone.go

@@ -176,9 +176,42 @@ var _NamedMapContainerCloneNeedsRegeneration = NamedMapContainer(struct {
 	Attrs NamedMap
 }{})
 
+// Clone makes a deep copy of MapSlicePointerContainer.
+// The result aliases no memory with the original.
+func (src *MapSlicePointerContainer) Clone() *MapSlicePointerContainer {
+	if src == nil {
+		return nil
+	}
+	dst := new(MapSlicePointerContainer)
+	*dst = *src
+	if dst.Routes != nil {
+		dst.Routes = map[string][]*SliceContainer{}
+		for k, sv := range src.Routes {
+			if sv == nil {
+				dst.Routes[k] = nil
+				continue
+			}
+			dst.Routes[k] = make([]*SliceContainer, len(sv))
+			for i := range sv {
+				if sv[i] == nil {
+					dst.Routes[k][i] = nil
+				} else {
+					dst.Routes[k][i] = sv[i].Clone()
+				}
+			}
+		}
+	}
+	return dst
+}
+
+// A compilation failure here means this code must be regenerated, with the command at the top of this file.
+var _MapSlicePointerContainerCloneNeedsRegeneration = MapSlicePointerContainer(struct {
+	Routes map[string][]*SliceContainer
+}{})
+
 // Clone duplicates src into dst and reports whether it succeeded.
 // To succeed, <src, dst> must be of types <*T, *T> or <*T, **T>,
-// where T is one of SliceContainer,InterfaceContainer,MapWithPointers,DeeplyNestedMap,NamedMapContainer.
+// where T is one of SliceContainer,InterfaceContainer,MapWithPointers,DeeplyNestedMap,NamedMapContainer,MapSlicePointerContainer.
 func Clone(dst, src any) bool {
 	switch src := src.(type) {
 	case *SliceContainer:
@@ -226,6 +259,15 @@ func Clone(dst, src any) bool {
 			*dst = src.Clone()
 			return true
 		}
+	case *MapSlicePointerContainer:
+		switch dst := dst.(type) {
+		case *MapSlicePointerContainer:
+			*dst = *src.Clone()
+			return true
+		case **MapSlicePointerContainer:
+			*dst = src.Clone()
+			return true
+		}
 	}
 	return false
 }

cmd/containerboot/egressservices.go

@@ -22,11 +22,12 @@ import (
 	"time"
 
 	"github.com/fsnotify/fsnotify"
+
 	"tailscale.com/client/local"
-	"tailscale.com/ipn"
 	"tailscale.com/kube/egressservices"
 	"tailscale.com/kube/kubeclient"
 	"tailscale.com/kube/kubetypes"
+	"tailscale.com/types/netmap"
 	"tailscale.com/util/httpm"
 	"tailscale.com/util/linuxfw"
 	"tailscale.com/util/mak"
@@ -54,7 +55,7 @@ type egressProxy struct {
 
 	tsClient *local.Client // never nil
 
-	netmapChan chan ipn.Notify // chan to receive netmap updates on
+	netmapChan chan *netmap.NetworkMap // chan to receive netmap updates on
 
 	podIPv4 string // never empty string, currently only IPv4 is supported
 
@@ -86,7 +87,7 @@ type httpClient interface {
 // - the mounted egress config has changed
 // - the proxy's tailnet IP addresses have changed
 // - tailnet IPs have changed for any backend targets specified by tailnet FQDN
-func (ep *egressProxy) run(ctx context.Context, n ipn.Notify, opts egressProxyRunOpts) error {
+func (ep *egressProxy) run(ctx context.Context, nm *netmap.NetworkMap, opts egressProxyRunOpts) error {
 	ep.configure(opts)
 	var tickChan <-chan time.Time
 	var eventChan <-chan fsnotify.Event
@@ -105,7 +106,7 @@ func (ep *egressProxy) run(ctx context.Context, n ipn.Notify, opts egressProxyRu
 		eventChan = w.Events
 	}
 
-	if err := ep.sync(ctx, n); err != nil {
+	if err := ep.sync(ctx, nm); err != nil {
 		return err
 	}
 	for {
@@ -116,14 +117,14 @@ func (ep *egressProxy) run(ctx context.Context, n ipn.Notify, opts egressProxyRu
 			log.Printf("periodic sync, ensuring firewall config is up to date...")
 		case <-eventChan:
 			log.Printf("config file change detected, ensuring firewall config is up to date...")
-		case n = <-ep.netmapChan:
+		case nm = <-ep.netmapChan:
-			shouldResync := ep.shouldResync(n)
+			shouldResync := ep.shouldResync(nm)
 			if !shouldResync {
 				continue
 			}
 			log.Printf("netmap change detected, ensuring firewall config is up to date...")
 		}
-		if err := ep.sync(ctx, n); err != nil {
+		if err := ep.sync(ctx, nm); err != nil {
 			return fmt.Errorf("error syncing egress service config: %w", err)
 		}
 	}
@@ -135,7 +136,7 @@ type egressProxyRunOpts struct {
 	kc           kubeclient.Client
 	tsClient     *local.Client
 	stateSecret  string
-	netmapChan   chan ipn.Notify
+	netmapChan   chan *netmap.NetworkMap
 	podIPv4      string
 	tailnetAddrs []netip.Prefix
 }
@@ -164,7 +165,7 @@ func (ep *egressProxy) configure(opts egressProxyRunOpts) {
 // any firewall rules need to be updated. Currently using status in state Secret as a reference for what is the current
 // firewall configuration is good enough because - the status is keyed by the Pod IP - we crash the Pod on errors such
 // as failed firewall update
-func (ep *egressProxy) sync(ctx context.Context, n ipn.Notify) error {
+func (ep *egressProxy) sync(ctx context.Context, nm *netmap.NetworkMap) error {
 	cfgs, err := ep.getConfigs()
 	if err != nil {
 		return fmt.Errorf("error retrieving egress service configs: %w", err)
@@ -173,12 +174,12 @@ func (ep *egressProxy) sync(ctx context.Context, n ipn.Notify) error {
 	if err != nil {
 		return fmt.Errorf("error retrieving current egress proxy status: %w", err)
 	}
-	newStatus, err := ep.syncEgressConfigs(cfgs, status, n)
+	newStatus, err := ep.syncEgressConfigs(cfgs, status, nm)
 	if err != nil {
 		return fmt.Errorf("error syncing egress service configs: %w", err)
 	}
 	if !servicesStatusIsEqual(newStatus, status) {
-		if err := ep.setStatus(ctx, newStatus, n); err != nil {
+		if err := ep.setStatus(ctx, newStatus, nm); err != nil {
 			return fmt.Errorf("error setting egress proxy status: %w", err)
 		}
 	}
@@ -187,14 +188,14 @@ func (ep *egressProxy) sync(ctx context.Context, n ipn.Notify) error {
 
 // addrsHaveChanged returns true if the provided netmap update contains tailnet address change for this proxy node.
 // Netmap must not be nil.
-func (ep *egressProxy) addrsHaveChanged(n ipn.Notify) bool {
+func (ep *egressProxy) addrsHaveChanged(nm *netmap.NetworkMap) bool {
-	return !reflect.DeepEqual(ep.tailnetAddrs, n.NetMap.SelfNode.Addresses())
+	return !reflect.DeepEqual(ep.tailnetAddrs, nm.SelfNode.Addresses())
 }
 
 // syncEgressConfigs adds and deletes firewall rules to match the desired
 // configuration. It uses the provided status to determine what is currently
 // applied and updates the status after a successful sync.
-func (ep *egressProxy) syncEgressConfigs(cfgs *egressservices.Configs, status *egressservices.Status, n ipn.Notify) (*egressservices.Status, error) {
+func (ep *egressProxy) syncEgressConfigs(cfgs egressservices.Configs, status *egressservices.Status, nm *netmap.NetworkMap) (*egressservices.Status, error) {
 	if !(wantsServicesConfigured(cfgs) || hasServicesConfigured(status)) {
 		return nil, nil
 	}
@@ -212,8 +213,8 @@ func (ep *egressProxy) syncEgressConfigs(cfgs *egressservices.Configs, status *e
 	// Add new services, update rules for any that have changed.
 	rulesPerSvcToAdd := make(map[string][]rule, 0)
 	rulesPerSvcToDelete := make(map[string][]rule, 0)
-	for svcName, cfg := range *cfgs {
+	for svcName, cfg := range cfgs {
-		tailnetTargetIPs, err := ep.tailnetTargetIPsForSvc(cfg, n)
+		tailnetTargetIPs, err := ep.tailnetTargetIPsForSvc(cfg, nm)
 		if err != nil {
 			return nil, fmt.Errorf("error determining tailnet target IPs: %w", err)
 		}
@@ -228,12 +229,12 @@ func (ep *egressProxy) syncEgressConfigs(cfgs *egressservices.Configs, status *e
 		if len(rulesToDelete) != 0 {
 			mak.Set(&rulesPerSvcToDelete, svcName, rulesToDelete)
 		}
-		if len(rulesToAdd) != 0 || ep.addrsHaveChanged(n) {
+		if len(rulesToAdd) != 0 || ep.addrsHaveChanged(nm) {
 			// For each tailnet target, set up SNAT from the local tailnet device address of the matching
 			// family.
 			for _, t := range tailnetTargetIPs {
 				var local netip.Addr
-				for _, pfx := range n.NetMap.SelfNode.Addresses().All() {
+				for _, pfx := range nm.SelfNode.Addresses().All() {
 					if !pfx.IsSingleIP() {
 						continue
 					}
@@ -352,7 +353,7 @@ func updatesForCfg(svcName string, cfg egressservices.Config, status *egressserv
 
 // deleteUnneccessaryServices ensure that any services found on status, but not
 // present in config are deleted.
-func (ep *egressProxy) deleteUnnecessaryServices(cfgs *egressservices.Configs, status *egressservices.Status) error {
+func (ep *egressProxy) deleteUnnecessaryServices(cfgs egressservices.Configs, status *egressservices.Status) error {
 	if !hasServicesConfigured(status) {
 		return nil
 	}
@@ -367,7 +368,7 @@ func (ep *egressProxy) deleteUnnecessaryServices(cfgs *egressservices.Configs, s
 	}
 
 	for svcName, svc := range status.Services {
-		if _, ok := (*cfgs)[svcName]; !ok {
+		if _, ok := cfgs[svcName]; !ok {
 			log.Printf("service %s is no longer required, deleting", svcName)
 			if err := ensureServiceDeleted(svcName, svc, ep.nfr); err != nil {
 				return fmt.Errorf("error deleting service %s: %w", svcName, err)
@@ -379,7 +380,7 @@ func (ep *egressProxy) deleteUnnecessaryServices(cfgs *egressservices.Configs, s
 }
 
 // getConfigs gets the mounted egress service configuration.
-func (ep *egressProxy) getConfigs() (*egressservices.Configs, error) {
+func (ep *egressProxy) getConfigs() (egressservices.Configs, error) {
 	svcsCfg := filepath.Join(ep.cfgPath, egressservices.KeyEgressServices)
 	j, err := os.ReadFile(svcsCfg)
 	if os.IsNotExist(err) {
@@ -391,7 +392,7 @@ func (ep *egressProxy) getConfigs() (*egressservices.Configs, error) {
 	if len(j) == 0 || string(j) == "" {
 		return nil, nil
 	}
-	cfg := &egressservices.Configs{}
+	cfg := egressservices.Configs{}
 	if err := json.Unmarshal(j, &cfg); err != nil {
 		return nil, err
 	}
@@ -423,7 +424,7 @@ func (ep *egressProxy) getStatus(ctx context.Context) (*egressservices.Status, e
 
 // setStatus writes egress proxy's currently configured firewall to the state
 // Secret and updates proxy's tailnet addresses.
-func (ep *egressProxy) setStatus(ctx context.Context, status *egressservices.Status, n ipn.Notify) error {
+func (ep *egressProxy) setStatus(ctx context.Context, status *egressservices.Status, nm *netmap.NetworkMap) error {
 	// Pod IP is used to determine if a stored status applies to THIS proxy Pod.
 	if status == nil {
 		status = &egressservices.Status{}
@@ -446,7 +447,7 @@ func (ep *egressProxy) setStatus(ctx context.Context, status *egressservices.Sta
 	if err := ep.kc.JSONPatchResource(ctx, ep.stateSecret, kubeclient.TypeSecrets, []kubeclient.JSONPatch{patch}); err != nil {
 		return fmt.Errorf("error patching state Secret: %w", err)
 	}
-	ep.tailnetAddrs = n.NetMap.SelfNode.Addresses().AsSlice()
+	ep.tailnetAddrs = nm.SelfNode.Addresses().AsSlice()
 	return nil
 }
 
@@ -456,7 +457,7 @@ func (ep *egressProxy) setStatus(ctx context.Context, status *egressservices.Sta
 // FQDN, resolve the FQDN and return the resolved IPs. It checks if the
 // netfilter runner supports IPv6 NAT and skips any IPv6 addresses if it
 // doesn't.
-func (ep *egressProxy) tailnetTargetIPsForSvc(svc egressservices.Config, n ipn.Notify) (addrs []netip.Addr, err error) {
+func (ep *egressProxy) tailnetTargetIPsForSvc(svc egressservices.Config, nm *netmap.NetworkMap) (addrs []netip.Addr, err error) {
 	if svc.TailnetTarget.IP != "" {
 		addr, err := netip.ParseAddr(svc.TailnetTarget.IP)
 		if err != nil {
@@ -472,11 +473,11 @@ func (ep *egressProxy) tailnetTargetIPsForSvc(svc egressservices.Config, n ipn.N
 	if svc.TailnetTarget.FQDN == "" {
 		return nil, errors.New("unexpected egress service config- neither tailnet target IP nor FQDN is set")
 	}
-	if n.NetMap == nil {
+	if nm == nil {
 		log.Printf("netmap is not available, unable to determine backend addresses for %s", svc.TailnetTarget.FQDN)
 		return addrs, nil
 	}
-	egressAddrs, err := resolveTailnetFQDN(n.NetMap, svc.TailnetTarget.FQDN)
+	egressAddrs, err := resolveTailnetFQDN(nm, svc.TailnetTarget.FQDN)
 	if err != nil {
 		log.Printf("error fetching backend addresses for %q: %v", svc.TailnetTarget.FQDN, err)
 		return addrs, nil
@@ -502,22 +503,22 @@ func (ep *egressProxy) tailnetTargetIPsForSvc(svc egressservices.Config, n ipn.N
 
 // shouldResync parses netmap update and returns true if the update contains
 // changes for which the egress proxy's firewall should be reconfigured.
-func (ep *egressProxy) shouldResync(n ipn.Notify) bool {
+func (ep *egressProxy) shouldResync(nm *netmap.NetworkMap) bool {
-	if n.NetMap == nil {
+	if nm == nil {
 		return false
 	}
 
 	// If proxy's tailnet addresses have changed, resync.
-	if !reflect.DeepEqual(n.NetMap.SelfNode.Addresses().AsSlice(), ep.tailnetAddrs) {
+	if !reflect.DeepEqual(nm.SelfNode.Addresses().AsSlice(), ep.tailnetAddrs) {
 		log.Printf("node addresses have changed, trigger egress config resync")
-		ep.tailnetAddrs = n.NetMap.SelfNode.Addresses().AsSlice()
+		ep.tailnetAddrs = nm.SelfNode.Addresses().AsSlice()
 		return true
 	}
 
 	// If the IPs for any of the egress services configured via FQDN have
 	// changed, resync.
 	for fqdn, ips := range ep.targetFQDNs {
-		for _, nn := range n.NetMap.Peers {
+		for _, nn := range nm.Peers {
 			if equalFQDNs(nn.Name(), fqdn) {
 				if !reflect.DeepEqual(ips, nn.Addresses().AsSlice()) {
 					log.Printf("backend addresses for egress target %q have changed old IPs %v, new IPs %v trigger egress config resync", nn.Name(), ips, nn.Addresses().AsSlice())
@@ -602,8 +603,8 @@ type rule struct {
 	protocol      string
 }
 
-func wantsServicesConfigured(cfgs *egressservices.Configs) bool {
+func wantsServicesConfigured(cfgs egressservices.Configs) bool {
-	return cfgs != nil && len(*cfgs) != 0
+	return cfgs != nil && len(cfgs) != 0
 }
 
 func hasServicesConfigured(status *egressservices.Status) bool {
@@ -657,13 +658,13 @@ func (ep *egressProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 // would normally be this Pod. When this Pod is being deleted, the operator should have removed it from the Service
 // backends and eventually kube proxy routing rules should be updated to no longer route traffic for the Service to this
 // Pod.
-func (ep *egressProxy) waitTillSafeToShutdown(ctx context.Context, cfgs *egressservices.Configs, hp int) {
+func (ep *egressProxy) waitTillSafeToShutdown(ctx context.Context, cfgs egressservices.Configs, hp int) {
-	if cfgs == nil || len(*cfgs) == 0 { // avoid sleeping if no services are configured
+	if cfgs == nil || len(cfgs) == 0 { // avoid sleeping if no services are configured
 		return
 	}
 	log.Printf("Ensuring that cluster traffic for egress targets is no longer routed via this Pod...")
 	var wg sync.WaitGroup
-	for s, cfg := range *cfgs {
+	for s, cfg := range cfgs {
 		hep := cfg.HealthCheckEndpoint
 		if hep == "" {
 			log.Printf("Tailnet target %q does not have a cluster healthcheck specified, unable to verify if cluster traffic for the target is still routed via this Pod", s)

cmd/containerboot/egressservices_test.go

@@ -255,13 +255,13 @@ func TestWaitTillSafeToShutdown(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			cfgs := &egressservices.Configs{}
+			cfgs := egressservices.Configs{}
 			switches := make(map[string]int)
 
 			for svc, callsToSwitch := range tt.services {
 				endpoint := fmt.Sprintf("http://%s.local", svc)
 				if tt.healthCheckSet {
-					(*cfgs)[svc] = egressservices.Config{
+					cfgs[svc] = egressservices.Config{
 						HealthCheckEndpoint: endpoint,
 					}
 				}

cmd/containerboot/kube.go

@@ -21,6 +21,7 @@ import (
 	"github.com/fsnotify/fsnotify"
 	"tailscale.com/client/local"
 	"tailscale.com/ipn"
+	"tailscale.com/kube/authkey"
 	"tailscale.com/kube/egressservices"
 	"tailscale.com/kube/ingressservices"
 	"tailscale.com/kube/kubeapi"
@@ -32,7 +33,6 @@ import (
 )
 
 const fieldManager = "tailscale-container"
-const kubeletMountedConfigLn = "..data"
 
 // kubeClient is a wrapper around Tailscale's internal kube client that knows how to talk to the kube API server. We use
 // this rather than any of the upstream Kubernetes client libaries to avoid extra imports.
@@ -127,6 +127,9 @@ func (kc *kubeClient) deleteAuthKey(ctx context.Context) error {
 
 // resetContainerbootState resets state from previous runs of containerboot to
 // ensure the operator doesn't use stale state when a Pod is first recreated.
+//
+// Device identity keys (device_id, device_fqdn, device_ips) are preserved so
+// the operator can clean up the old device from the control plane.
 func (kc *kubeClient) resetContainerbootState(ctx context.Context, podUID string, tailscaledConfigAuthkey string) error {
 	existingSecret, err := kc.GetSecret(ctx, kc.stateSecret)
 	switch {
@@ -140,11 +143,6 @@ func (kc *kubeClient) resetContainerbootState(ctx context.Context, podUID string
 	s := &kubeapi.Secret{
 		Data: map[string][]byte{
 			kubetypes.KeyCapVer:              fmt.Appendf(nil, "%d", tailcfg.CurrentCapabilityVersion),
-
-			// TODO(tomhjp): Perhaps shouldn't clear device ID and use a different signal, as this could leak tailnet devices.
-			kubetypes.KeyDeviceID:            nil,
-			kubetypes.KeyDeviceFQDN:          nil,
-			kubetypes.KeyDeviceIPs:           nil,
 			kubetypes.KeyHTTPSEndpoint:       nil,
 			egressservices.KeyEgressServices: nil,
 			ingressservices.IngressConfigKey: nil,
@@ -169,47 +167,18 @@ func (kc *kubeClient) setAndWaitForAuthKeyReissue(ctx context.Context, client *l
 		return fmt.Errorf("error disconnecting from control: %w", err)
 	}
 
-	err = kc.setReissueAuthKey(ctx, tailscaledConfigAuthKey)
+	err = authkey.SetReissueAuthKey(ctx, kc.Client, kc.stateSecret, tailscaledConfigAuthKey, authkey.TailscaleContainerFieldManager)
 	if err != nil {
 		return fmt.Errorf("failed to set reissue_authkey in Kubernetes Secret: %w", err)
 	}
 
-	err = kc.waitForAuthKeyReissue(ctx, cfg.TailscaledConfigFilePath, tailscaledConfigAuthKey, 10*time.Minute)
+	clearFn := func(ctx context.Context) error {
-	if err != nil {
+		return authkey.ClearReissueAuthKey(ctx, kc.Client, kc.stateSecret, authkey.TailscaleContainerFieldManager)
-		return fmt.Errorf("failed to receive new auth key: %w", err)
 	}
 
-	return nil
+	getAuthKey := func() string { return authkey.AuthKeyFromConfig(cfg.TailscaledConfigFilePath) }
-}
+	tailscaledCfgDir := filepath.Dir(cfg.TailscaledConfigFilePath)
-
+	var notify <-chan struct{}
-func (kc *kubeClient) setReissueAuthKey(ctx context.Context, authKey string) error {
-	s := &kubeapi.Secret{
-		Data: map[string][]byte{
-			kubetypes.KeyReissueAuthkey: []byte(authKey),
-		},
-	}
-
-	log.Printf("Requesting a new auth key from operator")
-	return kc.StrategicMergePatchSecret(ctx, kc.stateSecret, s, fieldManager)
-}
-
-func (kc *kubeClient) waitForAuthKeyReissue(ctx context.Context, configPath string, oldAuthKey string, maxWait time.Duration) error {
-	log.Printf("Waiting for operator to provide new auth key (max wait: %v)", maxWait)
-
-	ctx, cancel := context.WithTimeout(ctx, maxWait)
-	defer cancel()
-
-	tailscaledCfgDir := filepath.Dir(configPath)
-	toWatch := filepath.Join(tailscaledCfgDir, kubeletMountedConfigLn)
-
-	var (
-		pollTicker <-chan time.Time
-		eventChan  <-chan fsnotify.Event
-	)
-
-	pollInterval := 5 * time.Second
-
-	// Try to use fsnotify for faster notification
 	if w, err := fsnotify.NewWatcher(); err != nil {
 		log.Printf("auth key reissue: fsnotify unavailable, using polling: %v", err)
 	} else if err := w.Add(tailscaledCfgDir); err != nil {
@@ -217,54 +186,28 @@ func (kc *kubeClient) waitForAuthKeyReissue(ctx context.Context, configPath stri
 		log.Printf("auth key reissue: fsnotify watch failed, using polling: %v", err)
 	} else {
 		defer w.Close()
-		log.Printf("auth key reissue: watching for config changes via fsnotify")
+		ch := make(chan struct{}, 1)
-		eventChan = w.Events
+		toWatch := filepath.Join(tailscaledCfgDir, "..data")
-	}
+		go func() {
-
+			for ev := range w.Events {
-	// still keep polling if using fsnotify, for logging and in case fsnotify fails
+				if ev.Name == toWatch {
-	pt := time.NewTicker(pollInterval)
-	defer pt.Stop()
-	pollTicker = pt.C
-
-	start := time.Now()
-
-	for {
 					select {
-		case <-ctx.Done():
+					case ch <- struct{}{}:
-			return fmt.Errorf("timeout waiting for auth key reissue after %v", maxWait)
+					default:
-		case <-pollTicker: // Waits for polling tick, continues when received
-		case event := <-eventChan:
-			if event.Name != toWatch {
-				continue
 					}
 				}
+			}
+		}()
+		notify = ch
+		log.Printf("auth key reissue: watching for config changes via fsnotify")
+	}
 
-		newAuthKey := authkeyFromTailscaledConfig(configPath)
+	err = authkey.WaitForAuthKeyReissue(ctx, tailscaledConfigAuthKey, 10*time.Minute, getAuthKey, clearFn, notify)
-		if newAuthKey != "" && newAuthKey != oldAuthKey {
+	if err != nil {
-			log.Printf("New auth key received from operator after %v", time.Since(start).Round(time.Second))
+		return fmt.Errorf("failed to receive new auth key: %w", err)
-
-			if err := kc.clearReissueAuthKeyRequest(ctx); err != nil {
-				log.Printf("Warning: failed to clear reissue request: %v", err)
 	}
 
 	return nil
-		}
-
-		if eventChan == nil && pollTicker != nil {
-			log.Printf("Waiting for new auth key from operator (%v elapsed)", time.Since(start).Round(time.Second))
-		}
-	}
-}
-
-// clearReissueAuthKeyRequest removes the reissue_authkey marker from the Secret
-// to signal to the operator that we've successfully received the new key.
-func (kc *kubeClient) clearReissueAuthKeyRequest(ctx context.Context) error {
-	s := &kubeapi.Secret{
-		Data: map[string][]byte{
-			kubetypes.KeyReissueAuthkey: nil,
-		},
-	}
-	return kc.StrategicMergePatchSecret(ctx, kc.stateSecret, s, fieldManager)
 }
 
 // waitForConsistentState waits for tailscaled to finish writing state if it

cmd/containerboot/kube_test.go

@@ -259,10 +259,6 @@ func TestResetContainerbootState(t *testing.T) {
 			expected: map[string][]byte{
 				kubetypes.KeyCapVer:              capver,
 				kubetypes.KeyPodUID:              []byte("1234"),
-				// Cleared keys.
-				kubetypes.KeyDeviceID:            nil,
-				kubetypes.KeyDeviceFQDN:          nil,
-				kubetypes.KeyDeviceIPs:           nil,
 				kubetypes.KeyHTTPSEndpoint:       nil,
 				egressservices.KeyEgressServices: nil,
 				ingressservices.IngressConfigKey: nil,
@@ -272,10 +268,6 @@ func TestResetContainerbootState(t *testing.T) {
 			initial: map[string][]byte{},
 			expected: map[string][]byte{
 				kubetypes.KeyCapVer:              capver,
-				// Cleared keys.
-				kubetypes.KeyDeviceID:            nil,
-				kubetypes.KeyDeviceFQDN:          nil,
-				kubetypes.KeyDeviceIPs:           nil,
 				kubetypes.KeyHTTPSEndpoint:       nil,
 				egressservices.KeyEgressServices: nil,
 				ingressservices.IngressConfigKey: nil,
@@ -303,9 +295,6 @@ func TestResetContainerbootState(t *testing.T) {
 				kubetypes.KeyCapVer: capver,
 				kubetypes.KeyPodUID: []byte("1234"),
 				// Cleared keys.
-				kubetypes.KeyDeviceID:            nil,
-				kubetypes.KeyDeviceFQDN:          nil,
-				kubetypes.KeyDeviceIPs:           nil,
 				kubetypes.KeyHTTPSEndpoint:       nil,
 				egressservices.KeyEgressServices: nil,
 				ingressservices.IngressConfigKey: nil,
@@ -321,9 +310,6 @@ func TestResetContainerbootState(t *testing.T) {
 				kubetypes.KeyCapVer:         capver,
 				kubetypes.KeyReissueAuthkey: nil,
 				// Cleared keys.
-				kubetypes.KeyDeviceID:            nil,
-				kubetypes.KeyDeviceFQDN:          nil,
-				kubetypes.KeyDeviceIPs:           nil,
 				kubetypes.KeyHTTPSEndpoint:       nil,
 				egressservices.KeyEgressServices: nil,
 				ingressservices.IngressConfigKey: nil,
@@ -338,9 +324,6 @@ func TestResetContainerbootState(t *testing.T) {
 				kubetypes.KeyCapVer: capver,
 				// reissue_authkey not cleared.
 				// Cleared keys.
-				kubetypes.KeyDeviceID:            nil,
-				kubetypes.KeyDeviceFQDN:          nil,
-				kubetypes.KeyDeviceIPs:           nil,
 				kubetypes.KeyHTTPSEndpoint:       nil,
 				egressservices.KeyEgressServices: nil,
 				ingressservices.IngressConfigKey: nil,
@@ -355,9 +338,6 @@ func TestResetContainerbootState(t *testing.T) {
 				kubetypes.KeyCapVer: capver,
 				// reissue_authkey not cleared.
 				// Cleared keys.
-				kubetypes.KeyDeviceID:            nil,
-				kubetypes.KeyDeviceFQDN:          nil,
-				kubetypes.KeyDeviceIPs:           nil,
 				kubetypes.KeyHTTPSEndpoint:       nil,
 				egressservices.KeyEgressServices: nil,
 				ingressservices.IngressConfigKey: nil,

cmd/containerboot/main.go

@@ -136,11 +136,12 @@ import (
 	"time"
 
 	"golang.org/x/sys/unix"
-	"tailscale.com/client/tailscale"
+
+	"tailscale.com/client/local"
 	"tailscale.com/health"
 	"tailscale.com/ipn"
-	"tailscale.com/ipn/conffile"
 	kubeutils "tailscale.com/k8s-operator"
+	"tailscale.com/kube/authkey"
 	healthz "tailscale.com/kube/health"
 	"tailscale.com/kube/kubetypes"
 	klc "tailscale.com/kube/localclient"
@@ -173,7 +174,6 @@ func main() {
 
 func run() error {
 	log.SetPrefix("boot: ")
-	tailscale.I_Acknowledge_This_API_Is_Unstable = true
 
 	cfg, err := configFromEnv()
 	if err != nil {
@@ -210,7 +210,7 @@ func run() error {
 
 	var tailscaledConfigAuthkey string
 	if isOneStepConfig(cfg) {
-		tailscaledConfigAuthkey = authkeyFromTailscaledConfig(cfg.TailscaledConfigFilePath)
+		tailscaledConfigAuthkey = authkey.AuthKeyFromConfig(cfg.TailscaledConfigFilePath)
 	}
 
 	var kc *kubeClient
@@ -375,7 +375,7 @@ authLoop:
 					if hasKubeStateStore(cfg) {
 						log.Printf("Auth key missing or invalid (NeedsLogin state), disconnecting from control and requesting new key from operator")
 
-						err := kc.setAndWaitForAuthKeyReissue(bootCtx, client, cfg, tailscaledConfigAuthkey)
+						err := kc.setAndWaitForAuthKeyReissue(ctx, client, cfg, tailscaledConfigAuthkey)
 						if err != nil {
 							return fmt.Errorf("failed to get a reissued authkey: %w", err)
 						}
@@ -415,7 +415,7 @@ authLoop:
 				if isOneStepConfig(cfg) && hasKubeStateStore(cfg) {
 					log.Printf("Auth key failed to authenticate (may be expired or single-use), disconnecting from control and requesting new key from operator")
 
-					err := kc.setAndWaitForAuthKeyReissue(bootCtx, client, cfg, tailscaledConfigAuthkey)
+					err := kc.setAndWaitForAuthKeyReissue(ctx, client, cfg, tailscaledConfigAuthkey)
 					if err != nil {
 						return fmt.Errorf("failed to get a reissued authkey: %w", err)
 					}
@@ -537,7 +537,7 @@ authLoop:
 		failedResolveAttempts++
 	}
 
-	var egressSvcsNotify chan ipn.Notify
+	var egressSvcsNotify chan *netmap.NetworkMap
 	notifyChan := make(chan ipn.Notify)
 	errChan := make(chan error)
 	go func() {
@@ -551,10 +551,17 @@ authLoop:
 			}
 		}
 	}()
+	// Peer set changes (Add/Remove) no longer ride on the IPN bus; poll
+	// periodically so egress FQDN resolution and peer-aware work picks
+	// them up. SelfChange covers prompt self changes.
+	const peerPollInterval = 15 * time.Second
+	peerPoll := time.NewTicker(peerPollInterval)
+	defer peerPoll.Stop()
 	var wg sync.WaitGroup
 
 runLoop:
 	for {
+		var processNetmap bool
 		select {
 		case <-ctx.Done():
 			// Although killTailscaled() is deferred earlier, if we
@@ -567,6 +574,8 @@ runLoop:
 			return fmt.Errorf("failed to read from tailscaled: %w", err)
 		case err := <-cfgWatchErrChan:
 			return fmt.Errorf("failed to watch tailscaled config: %w", err)
+		case <-peerPoll.C:
+			processNetmap = true
 		case n := <-notifyChan:
 			// TODO: (ChaosInTheCRD) Add node removed check when supported by ipn
 			if n.State != nil && *n.State != ipn.Running {
@@ -577,8 +586,43 @@ runLoop:
 				// whereupon we'll go through initial auth again.
 				return fmt.Errorf("tailscaled left running state (now in state %q), exiting", *n.State)
 			}
-			if n.NetMap != nil {
+			if n.SelfChange != nil {
-				addrs = n.NetMap.SelfNode.Addresses().AsSlice()
+				processNetmap = true
+			}
+		case <-tc:
+			newBackendAddrs, err := resolveDNS(ctx, cfg.ProxyTargetDNSName)
+			if err != nil {
+				log.Printf("[unexpected] error resolving DNS name %s: %v", cfg.ProxyTargetDNSName, err)
+				resetTimer(true)
+				continue
+			}
+			backendsHaveChanged := !(slices.EqualFunc(backendAddrs, newBackendAddrs, func(ip1 net.IP, ip2 net.IP) bool {
+				return slices.ContainsFunc(newBackendAddrs, func(ip net.IP) bool { return ip.Equal(ip1) })
+			}))
+			if backendsHaveChanged && len(addrs) != 0 {
+				log.Printf("Backend address change detected, installing proxy rules for backends %v", newBackendAddrs)
+				if err := installIngressForwardingRuleForDNSTarget(ctx, newBackendAddrs, addrs, nfr); err != nil {
+					return fmt.Errorf("installing ingress proxy rules for DNS target %s: %v", cfg.ProxyTargetDNSName, err)
+				}
+			}
+			backendAddrs = newBackendAddrs
+			resetTimer(false)
+			continue
+		case e := <-egressSvcsErrorChan:
+			return fmt.Errorf("egress proxy failed: %v", e)
+		case e := <-ingressSvcsErrorChan:
+			return fmt.Errorf("ingress proxy failed: %v", e)
+		}
+		if !processNetmap {
+			continue
+		}
+		nm, err := fetchNetMap(ctx, client)
+		if err != nil {
+			log.Printf("error fetching netmap: %v", err)
+			continue
+		}
+		if nm != nil {
+			addrs = nm.SelfNode.Addresses().AsSlice()
 			newCurrentIPs := deephash.Hash(&addrs)
 			ipsHaveChanged := newCurrentIPs != currentIPs
 
@@ -590,14 +634,14 @@ runLoop:
 			// Kubernetes Secret to clean up tailnet nodes
 			// for proxies whose route setup continuously
 			// fails.
-				deviceID := n.NetMap.SelfNode.StableID()
+			deviceID := nm.SelfNode.StableID()
 			if hasKubeStateStore(cfg) && deephash.Update(&currentDeviceID, &deviceID) {
-					if err := kc.storeDeviceID(ctx, n.NetMap.SelfNode.StableID()); err != nil {
+				if err := kc.storeDeviceID(ctx, nm.SelfNode.StableID()); err != nil {
 					return fmt.Errorf("storing device ID in Kubernetes Secret: %w", err)
 				}
 			}
 			if cfg.TailnetTargetFQDN != "" {
-					egressAddrs, err := resolveTailnetFQDN(n.NetMap, cfg.TailnetTargetFQDN)
+				egressAddrs, err := resolveTailnetFQDN(nm, cfg.TailnetTargetFQDN)
 				if err != nil {
 					log.Print(err.Error())
 					break
@@ -653,7 +697,7 @@ runLoop:
 				backendAddrs = newBackendAddrs
 			}
 			if cfg.ServeConfigPath != "" {
-					cd := certDomainFromNetmap(n.NetMap)
+				cd := certDomainFromNetmap(nm)
 				if cd == "" {
 					cd = kubetypes.ValueNoHTTPS
 				}
@@ -696,9 +740,9 @@ runLoop:
 			// set up ensures that the operator does not
 			// advertize endpoints of broken proxies.
 			// TODO (irbekrm): instead of using the IP and FQDN, have some other mechanism for the proxy signal that it is 'Ready'.
-				deviceEndpoints := []any{n.NetMap.SelfNode.Name(), n.NetMap.SelfNode.Addresses()}
+			deviceEndpoints := []any{nm.SelfNode.Name(), nm.SelfNode.Addresses()}
 			if hasKubeStateStore(cfg) && deephash.Update(&currentDeviceEndpoints, &deviceEndpoints) {
-					if err := kc.storeDeviceEndpoints(ctx, n.NetMap.SelfNode.Name(), n.NetMap.SelfNode.Addresses().AsSlice()); err != nil {
+				if err := kc.storeDeviceEndpoints(ctx, nm.SelfNode.Name(), nm.SelfNode.Addresses().AsSlice()); err != nil {
 					return fmt.Errorf("storing device IPs and FQDN in Kubernetes Secret: %w", err)
 				}
 			}
@@ -727,7 +771,7 @@ runLoop:
 			}
 
 			if egressSvcsNotify != nil {
-					egressSvcsNotify <- n
+				egressSvcsNotify <- nm
 			}
 		}
 		if !startupTasksDone {
@@ -749,7 +793,7 @@ runLoop:
 				// will crash this node.
 				if cfg.EgressProxiesCfgPath != "" {
 					log.Printf("configuring egress proxy using configuration file at %s", cfg.EgressProxiesCfgPath)
-						egressSvcsNotify = make(chan ipn.Notify)
+					egressSvcsNotify = make(chan *netmap.NetworkMap)
 					opts := egressProxyRunOpts{
 						cfgPath:      cfg.EgressProxiesCfgPath,
 						nfr:          nfr,
@@ -761,7 +805,7 @@ runLoop:
 						tailnetAddrs: addrs,
 					}
 					go func() {
-							if err := ep.run(ctx, n, opts); err != nil {
+						if err := ep.run(ctx, nm, opts); err != nil {
 							egressSvcsErrorChan <- err
 						}
 					}()
@@ -807,29 +851,6 @@ runLoop:
 				go reaper()
 			}
 		}
-		case <-tc:
-			newBackendAddrs, err := resolveDNS(ctx, cfg.ProxyTargetDNSName)
-			if err != nil {
-				log.Printf("[unexpected] error resolving DNS name %s: %v", cfg.ProxyTargetDNSName, err)
-				resetTimer(true)
-				continue
-			}
-			backendsHaveChanged := !(slices.EqualFunc(backendAddrs, newBackendAddrs, func(ip1 net.IP, ip2 net.IP) bool {
-				return slices.ContainsFunc(newBackendAddrs, func(ip net.IP) bool { return ip.Equal(ip1) })
-			}))
-			if backendsHaveChanged && len(addrs) != 0 {
-				log.Printf("Backend address change detected, installing proxy rules for backends %v", newBackendAddrs)
-				if err := installIngressForwardingRuleForDNSTarget(ctx, newBackendAddrs, addrs, nfr); err != nil {
-					return fmt.Errorf("installing ingress proxy rules for DNS target %s: %v", cfg.ProxyTargetDNSName, err)
-				}
-			}
-			backendAddrs = newBackendAddrs
-			resetTimer(false)
-		case e := <-egressSvcsErrorChan:
-			return fmt.Errorf("egress proxy failed: %v", e)
-		case e := <-ingressSvcsErrorChan:
-			return fmt.Errorf("ingress proxy failed: %v", e)
-		}
 	}
 	wg.Wait()
 
@@ -964,6 +985,15 @@ func runHTTPServer(mux *http.ServeMux, addr string) (close func() error) {
 	}
 }
 
+// fetchNetMap fetches the current netmap from tailscaled via the
+// "current-netmap" localapi debug action. The debug action's payload
+// shape is intentionally not part of any stable API; containerboot
+// reads its own internal-package types out of it. New external consumers
+// should not rely on this — see [local.Client.Status] and friends.
+func fetchNetMap(ctx context.Context, lc *local.Client) (*netmap.NetworkMap, error) {
+	return local.GetDebugResultJSON[*netmap.NetworkMap](ctx, lc, "current-netmap")
+}
+
 // resolveTailnetFQDN resolves a tailnet FQDN to a list of IP prefixes, which
 // can be either a peer device or a Tailscale Service.
 func resolveTailnetFQDN(nm *netmap.NetworkMap, fqdn string) ([]netip.Prefix, error) {
@@ -1025,11 +1055,3 @@ func serviceIPsFromNetMap(nm *netmap.NetworkMap, fqdn dnsname.FQDN) []netip.Pref
 
 	return prefixes
 }
-
-func authkeyFromTailscaledConfig(path string) string {
-	if cfg, err := conffile.Load(path); err == nil && cfg.Parsed.AuthKey != nil {
-		return *cfg.Parsed.AuthKey
-	}
-
-	return ""
-}

cmd/containerboot/main_test.go

@@ -32,6 +32,7 @@ import (
 
 	"github.com/google/go-cmp/cmp"
 	"golang.org/x/sys/unix"
+	"tailscale.com/cmd/testwrapper/flakytest"
 	"tailscale.com/health"
 	"tailscale.com/ipn"
 	"tailscale.com/kube/egressservices"
@@ -45,6 +46,7 @@ import (
 const configFileAuthKey = "some-auth-key"
 
 func TestContainerBoot(t *testing.T) {
+	flakytest.Mark(t, "https://github.com/tailscale/tailscale/issues/19380")
 	boot := filepath.Join(t.TempDir(), "containerboot")
 	if err := exec.Command("go", "build", "-ldflags", "-X main.testSleepDuration=1ms", "-o", boot, "tailscale.com/cmd/containerboot").Run(); err != nil {
 		t.Fatalf("Building containerboot: %v", err)
@@ -69,6 +71,12 @@ func TestContainerBoot(t *testing.T) {
 		// Waits below to be true before proceeding to the next phase.
 		Notify *ipn.Notify
 
+		// If non-nil, install this NetMap on the fake LocalAPI before
+		// sending Notify. This is the replacement for the old
+		// Notify.NetMap field; reactive consumers fetch the current
+		// netmap via /localapi/v0/netmap on their own.
+		NetMap *netmap.NetworkMap
+
 		// WantCmds is the commands that containerboot should run in this phase.
 		WantCmds []string
 
@@ -103,12 +111,10 @@ func TestContainerBoot(t *testing.T) {
 	}
 	runningNotify := &ipn.Notify{
 		State: new(ipn.Running),
-		NetMap: &netmap.NetworkMap{
+		SelfChange: &tailcfg.Node{
-			SelfNode: (&tailcfg.Node{
 			StableID:  tailcfg.StableNodeID("myID"),
 			Name:      "test-node.test.ts.net.",
 			Addresses: []netip.Prefix{netip.MustParsePrefix("100.64.0.1/32")},
-			}).View(),
 		},
 	}
 	type testCase struct {
@@ -381,6 +387,12 @@ func TestContainerBoot(t *testing.T) {
 					{
 						Notify: &ipn.Notify{
 							State: new(ipn.Running),
+							SelfChange: &tailcfg.Node{
+								StableID:  tailcfg.StableNodeID("myID"),
+								Name:      "test-node.test.ts.net.",
+								Addresses: []netip.Prefix{netip.MustParsePrefix("100.64.0.1/32")},
+							},
+						},
 						NetMap: &netmap.NetworkMap{
 							SelfNode: (&tailcfg.Node{
 								StableID:  tailcfg.StableNodeID("myID"),
@@ -395,7 +407,6 @@ func TestContainerBoot(t *testing.T) {
 								}).View(),
 							},
 						},
-						},
 						WantLog:      "no forwarding rules for egress addresses [::1/128], host supports IPv6: false",
 						WantExitCode: new(1),
 					},
@@ -629,6 +640,12 @@ func TestContainerBoot(t *testing.T) {
 					{
 						Notify: &ipn.Notify{
 							State: new(ipn.Running),
+							SelfChange: &tailcfg.Node{
+								StableID:  tailcfg.StableNodeID("newID"),
+								Name:      "new-name.test.ts.net.",
+								Addresses: []netip.Prefix{netip.MustParsePrefix("100.64.0.1/32")},
+							},
+						},
 						NetMap: &netmap.NetworkMap{
 							SelfNode: (&tailcfg.Node{
 								StableID:  tailcfg.StableNodeID("newID"),
@@ -636,7 +653,6 @@ func TestContainerBoot(t *testing.T) {
 								Addresses: []netip.Prefix{netip.MustParsePrefix("100.64.0.1/32")},
 							}).View(),
 						},
-						},
 						WantKubeSecret: map[string]string{
 							"authkey":           "tskey-key",
 							"device_fqdn":       "new-name.test.ts.net.",
@@ -1093,6 +1109,12 @@ func TestContainerBoot(t *testing.T) {
 					{
 						Notify: &ipn.Notify{
 							State: new(ipn.Running),
+							SelfChange: &tailcfg.Node{
+								StableID:  tailcfg.StableNodeID("myID"),
+								Name:      "test-node.test.ts.net.",
+								Addresses: []netip.Prefix{netip.MustParsePrefix("100.64.0.1/32")},
+							},
+						},
 						NetMap: &netmap.NetworkMap{
 							SelfNode: (&tailcfg.Node{
 								StableID:  tailcfg.StableNodeID("myID"),
@@ -1107,7 +1129,6 @@ func TestContainerBoot(t *testing.T) {
 								}).View(),
 							},
 						},
-						},
 						WantKubeSecret: map[string]string{
 							"egress-services":   string(mustJSON(t, egressStatus)),
 							"authkey":           "tskey-key",
@@ -1274,6 +1295,18 @@ func TestContainerBoot(t *testing.T) {
 						t.Fatalf("phase %d: updating mtime for %q: %v", i, path, err)
 					}
 				}
+				nmForFake := p.NetMap
+				if nmForFake == nil && p.Notify != nil && p.Notify.SelfChange != nil {
+					// Synthesize a minimal netmap from SelfChange so
+					// containerboot's NetMap() fetch returns
+					// something usable when the test only set Notify.
+					nmForFake = &netmap.NetworkMap{
+						SelfNode: p.Notify.SelfChange.View(),
+					}
+				}
+				if nmForFake != nil {
+					env.lapi.SetNetMap(nmForFake)
+				}
 				env.lapi.Notify(p.Notify)
 				if p.Signal != nil {
 					cmd.Process.Signal(*p.Signal)
@@ -1466,6 +1499,7 @@ type localAPI struct {
 	sync.Mutex
 	cond   *sync.Cond
 	notify *ipn.Notify
+	netmap *netmap.NetworkMap // served by /localapi/v0/netmap
 }
 
 func (lc *localAPI) Start() error {
@@ -1502,8 +1536,44 @@ func (lc *localAPI) Notify(n *ipn.Notify) {
 	lc.cond.Broadcast()
 }
 
+// SetNetMap installs the netmap that the fake /localapi/v0/netmap endpoint
+// will return.
+func (lc *localAPI) SetNetMap(nm *netmap.NetworkMap) {
+	lc.Lock()
+	defer lc.Unlock()
+	lc.netmap = nm
+}
+
 func (lc *localAPI) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	switch r.URL.Path {
+	case "/localapi/v0/netmap":
+		w.Header().Set("Content-Type", "application/json")
+		lc.Lock()
+		nm := lc.netmap
+		lc.Unlock()
+		if nm == nil {
+			http.Error(w, "no netmap", http.StatusServiceUnavailable)
+			return
+		}
+		json.NewEncoder(w).Encode(nm)
+		return
+	case "/localapi/v0/debug":
+		// containerboot fetches the netmap via the "current-netmap"
+		// debug action; serve it like /localapi/v0/netmap above.
+		if r.URL.Query().Get("action") != "current-netmap" {
+			http.Error(w, "unsupported debug action", http.StatusNotFound)
+			return
+		}
+		w.Header().Set("Content-Type", "application/json")
+		lc.Lock()
+		nm := lc.netmap
+		lc.Unlock()
+		if nm == nil {
+			http.Error(w, "no netmap", http.StatusServiceUnavailable)
+			return
+		}
+		json.NewEncoder(w).Encode(nm)
+		return
 	case "/localapi/v0/serve-config":
 		switch r.Method {
 		case "GET":

cmd/derper/bootstrap_dns_test.go

@@ -41,8 +41,28 @@ func (b *bitbucketResponseWriter) Write(p []byte) (int, error) { return len(p),
 
 func (b *bitbucketResponseWriter) WriteHeader(statusCode int) {}
 
+// setDNSCache sets the published DNS cache for tests.
+func setDNSCache(tb testing.TB, m *dnsEntryMap) {
+	tb.Helper()
+	j, err := json.Marshal(m.IPs)
+	if err != nil {
+		tb.Fatal(err)
+	}
+	tstest.AssertNotParallel(tb)
+	dnsCache.Store(m)
+	dnsCacheBytes.Store(j)
+	tb.Cleanup(func() {
+		dnsCache.Store(nil)
+		dnsCacheBytes.Store(nil)
+	})
+}
+
 func getBootstrapDNS(t *testing.T, q string) map[string][]net.IP {
 	t.Helper()
+	tstest.AssertNotParallel(t)
+	if dnsCache.Load() == nil {
+		t.Fatal("dnsCache not initialized; call setDNSCache before getBootstrapDNS")
+	}
 	req, _ := http.NewRequest("GET", "https://localhost/bootstrap-dns?q="+url.QueryEscape(q), nil)
 	w := httptest.NewRecorder()
 	handleBootstrapDNS(w, req)
@@ -100,7 +120,8 @@ func TestUnpublishedDNS(t *testing.T) {
 	}
 }
 
-func resetMetrics() {
+func resetMetrics(tb testing.TB) {
+	tstest.AssertNotParallel(tb)
 	publishedDNSHits.Set(0)
 	publishedDNSMisses.Set(0)
 	unpublishedDNSHits.Set(0)
@@ -114,8 +135,7 @@ func TestUnpublishedDNSEmptyList(t *testing.T) {
 	pub := &dnsEntryMap{
 		IPs: map[string][]net.IP{"tailscale.com": {net.IPv4(10, 10, 10, 10)}},
 	}
-	dnsCache.Store(pub)
+	setDNSCache(t, pub)
-	dnsCacheBytes.Store([]byte(`{"tailscale.com":["10.10.10.10"]}`))
 
 	unpublishedDNSCache.Store(&dnsEntryMap{
 		IPs: map[string][]net.IP{
@@ -131,7 +151,7 @@ func TestUnpublishedDNSEmptyList(t *testing.T) {
 	t.Run("CacheMiss", func(t *testing.T) {
 		// One domain in map but empty, one not in map at all
 		for _, q := range []string{"log.tailscale.com", "login.tailscale.com"} {
-			resetMetrics()
+			resetMetrics(t)
 			ips := getBootstrapDNS(t, q)
 
 			// Expected our public map to be returned on a cache miss
@@ -149,7 +169,7 @@ func TestUnpublishedDNSEmptyList(t *testing.T) {
 
 	// Verify that we do get a valid response and metric.
 	t.Run("CacheHit", func(t *testing.T) {
-		resetMetrics()
+		resetMetrics(t)
 		ips := getBootstrapDNS(t, "controlplane.tailscale.com")
 		want := map[string][]net.IP{"controlplane.tailscale.com": {net.IPv4(1, 2, 3, 4)}}
 		if !reflect.DeepEqual(ips, want) {
@@ -166,8 +186,10 @@ func TestUnpublishedDNSEmptyList(t *testing.T) {
 }
 
 func TestLookupMetric(t *testing.T) {
+	setDNSCache(t, &dnsEntryMap{})
+
 	d := []string{"a.io", "b.io", "c.io", "d.io", "e.io", "e.io", "e.io", "a.io"}
-	resetMetrics()
+	resetMetrics(t)
 	for _, q := range d {
 		_ = getBootstrapDNS(t, q)
 	}

cmd/derper/depaware.txt

@@ -20,6 +20,7 @@ tailscale.com/cmd/derper dependencies: (generated by github.com/tailscale/depawa
         github.com/go-json-experiment/json/internal/jsonopts         from github.com/go-json-experiment/json+
         github.com/go-json-experiment/json/internal/jsonwire         from github.com/go-json-experiment/json+
         github.com/go-json-experiment/json/jsontext                  from github.com/go-json-experiment/json+
+     💣 github.com/go4org/hashtriemap                                from tailscale.com/derp/derpserver
         github.com/golang/groupcache/lru                             from tailscale.com/net/dnscache
         github.com/hdevalence/ed25519consensus                       from tailscale.com/tka
    L 💣 github.com/jsimonetti/rtnetlink                              from tailscale.com/net/netmon
@@ -310,7 +311,7 @@ tailscale.com/cmd/derper dependencies: (generated by github.com/tailscale/depawa
         hash                                                         from crypto+
         hash/crc32                                                   from compress/gzip+
         hash/fnv                                                     from google.golang.org/protobuf/internal/detrand
-        hash/maphash                                                 from go4.org/mem
+        hash/maphash                                                 from go4.org/mem+
         html                                                         from net/http/pprof+
         html/template                                                from tailscale.com/cmd/derper+
         internal/abi                                                 from crypto/x509/internal/macos+

cmd/derper/derper.go

@@ -87,8 +87,7 @@ var (
 	acceptConnLimit = flag.Float64("accept-connection-limit", math.Inf(+1), "rate limit for accepting new connection")
 	acceptConnBurst = flag.Int("accept-connection-burst", math.MaxInt, "burst limit for accepting new connection")
 
-	perClientRateLimit = flag.Uint("per-client-rate-limit", 0, "per-client receive rate limit in bytes/sec; 0 means unlimited. Mesh peers are exempt.")
+	rateConfigPath = flag.String("rate-config", "", "if non-empty, path to JSON rate limit config file. Rate limiting is experimental and subject to change. Configuration is reloaded on SIGHUP.")
-	perClientRateBurst = flag.Uint("per-client-rate-burst", 0, "per-client receive rate burst in bytes; 0 defaults to 2x the rate limit (only relevant when using nonzero --per-client-rate-limit)")
 
 	// tcpKeepAlive is intentionally long, to reduce battery cost. There is an L7 keepalive on a higher frequency schedule.
 	tcpKeepAlive = flag.Duration("tcp-keepalive-time", 10*time.Minute, "TCP keepalive time")
@@ -195,12 +194,11 @@ func main() {
 	s.SetVerifyClientURL(*verifyClientURL)
 	s.SetVerifyClientURLFailOpen(*verifyFailOpen)
 	s.SetTCPWriteTimeout(*tcpWriteTimeout)
-	if *perClientRateLimit > 0 {
+	if *rateConfigPath != "" {
-		burst := *perClientRateBurst
+		if err := s.LoadAndApplyRateConfig(*rateConfigPath); err != nil {
-		if burst < 1 {
+			log.Fatalf("derper: loading rate config: %v", err)
-			burst = *perClientRateLimit * 2
 		}
-		s.SetPerClientRateLimit(*perClientRateLimit, burst)
+		go watchRateConfig(ctx, s, *rateConfigPath)
 	}
 
 	var meshKey string
@@ -254,7 +252,7 @@ func main() {
 	if err := startMesh(s); err != nil {
 		log.Fatalf("startMesh: %v", err)
 	}
-	expvar.Publish("derp", s.ExpVar())
+	expvar.Publish("derp", s.ExpVar(*rateConfigPath != ""))
 
 	handleHome, ok := getHomeHandler(*flagHome)
 	if !ok {
@@ -436,6 +434,27 @@ func main() {
 	}
 }
 
+// watchRateConfig listens for SIGHUP signals and reloads the rate config
+// file on each signal, applying it to the server. It returns when ctx is done.
+func watchRateConfig(ctx context.Context, s *derpserver.Server, path string) {
+	sighup := make(chan os.Signal, 1)
+	signal.Notify(sighup, syscall.SIGHUP)
+	defer signal.Stop(sighup)
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-sighup:
+			log.Printf("derper: received SIGHUP, reloading rate config from %s", path)
+			if err := s.LoadAndApplyRateConfig(path); err != nil {
+				log.Printf("derper: rate config reload failed: %v", err)
+				continue
+			}
+			log.Printf("derper: rate config reloaded successfully")
+		}
+	}
+}
+
 var validProdHostname = regexp.MustCompile(`^derp([^.]*)\.tailscale\.com\.?$`)
 
 func prodAutocertHostPolicy(_ context.Context, host string) error {

cmd/gitops-pusher/gitops-pusher.go

@@ -26,7 +26,7 @@ import (
 	"github.com/tailscale/hujson"
 	"golang.org/x/oauth2/clientcredentials"
 	tsclient "tailscale.com/client/tailscale"
-	_ "tailscale.com/feature/condregister/identityfederation"
+	_ "tailscale.com/feature/identityfederation"
 	"tailscale.com/internal/client/tailscale"
 	"tailscale.com/util/httpm"
 )

cmd/hello/hello.go

@@ -5,212 +5,16 @@
 package main // import "tailscale.com/cmd/hello"
 
 import (
-	"context"
-	"crypto/tls"
-	_ "embed"
-	"encoding/json"
-	"errors"
-	"flag"
-	"html/template"
 	"log"
-	"net/http"
-	"os"
-	"strings"
-	"time"
 
-	"tailscale.com/client/local"
+	"tailscale.com/cmd/hello/helloserver"
-	"tailscale.com/client/tailscale/apitype"
-	"tailscale.com/tailcfg"
 )
 
-var (
-	httpAddr  = flag.String("http", ":80", "address to run an HTTP server on, or empty for none")
-	httpsAddr = flag.String("https", ":443", "address to run an HTTPS server on, or empty for none")
-	testIP    = flag.String("test-ip", "", "if non-empty, look up IP and exit before running a server")
-)
-
-//go:embed hello.tmpl.html
-var embeddedTemplate string
-
-var localClient local.Client
-
 func main() {
-	flag.Parse()
+	s := &helloserver.Server{
-	if *testIP != "" {
+		HTTPAddr:  ":80",
-		res, err := localClient.WhoIs(context.Background(), *testIP)
+		HTTPSAddr: ":443",
-		if err != nil {
-			log.Fatal(err)
 	}
-		e := json.NewEncoder(os.Stdout)
-		e.SetIndent("", "\t")
-		e.Encode(res)
-		return
-	}
-	if devMode() {
-		// Parse it optimistically
-		var err error
-		tmpl, err = template.New("home").Parse(embeddedTemplate)
-		if err != nil {
-			log.Printf("ignoring template error in dev mode: %v", err)
-		}
-	} else {
-		if embeddedTemplate == "" {
-			log.Fatalf("embeddedTemplate is empty; must be build with Go 1.16+")
-		}
-		tmpl = template.Must(template.New("home").Parse(embeddedTemplate))
-	}
-
-	http.HandleFunc("/", root)
 	log.Printf("Starting hello server.")
-
+	log.Fatal(s.Run())
-	errc := make(chan error, 1)
-	if *httpAddr != "" {
-		log.Printf("running HTTP server on %s", *httpAddr)
-		go func() {
-			errc <- http.ListenAndServe(*httpAddr, nil)
-		}()
-	}
-	if *httpsAddr != "" {
-		log.Printf("running HTTPS server on %s", *httpsAddr)
-		go func() {
-			hs := &http.Server{
-				Addr: *httpsAddr,
-				TLSConfig: &tls.Config{
-					GetCertificate: func(hi *tls.ClientHelloInfo) (*tls.Certificate, error) {
-						switch hi.ServerName {
-						case "hello.ts.net":
-							return localClient.GetCertificate(hi)
-						case "hello.ipn.dev":
-							c, err := tls.LoadX509KeyPair(
-								"/etc/hello/hello.ipn.dev.crt",
-								"/etc/hello/hello.ipn.dev.key",
-							)
-							if err != nil {
-								return nil, err
-							}
-							return &c, nil
-						}
-						return nil, errors.New("invalid SNI name")
-					},
-				},
-				IdleTimeout:       30 * time.Second,
-				ReadHeaderTimeout: 20 * time.Second,
-				MaxHeaderBytes:    10 << 10,
-			}
-			errc <- hs.ListenAndServeTLS("", "")
-		}()
-	}
-	log.Fatal(<-errc)
-}
-
-func devMode() bool { return *httpsAddr == "" && *httpAddr != "" }
-
-func getTmpl() (*template.Template, error) {
-	if devMode() {
-		tmplData, err := os.ReadFile("hello.tmpl.html")
-		if os.IsNotExist(err) {
-			log.Printf("using baked-in template in dev mode; can't find hello.tmpl.html in current directory")
-			return tmpl, nil
-		}
-		return template.New("home").Parse(string(tmplData))
-	}
-	return tmpl, nil
-}
-
-// tmpl is the template used in prod mode.
-// In dev mode it's only used if the template file doesn't exist on disk.
-// It's initialized by main after flag parsing.
-var tmpl *template.Template
-
-type tmplData struct {
-	DisplayName   string // "Foo Barberson"
-	LoginName     string // "foo@bar.com"
-	ProfilePicURL string // "https://..."
-	MachineName   string // "imac5k"
-	MachineOS     string // "Linux"
-	IP            string // "100.2.3.4"
-}
-
-func tailscaleIP(who *apitype.WhoIsResponse) string {
-	if who == nil {
-		return ""
-	}
-	vals, err := tailcfg.UnmarshalNodeCapJSON[string](who.Node.CapMap, tailcfg.NodeAttrNativeIPV4)
-	if err == nil && len(vals) > 0 {
-		return vals[0]
-	}
-	for _, nodeIP := range who.Node.Addresses {
-		if nodeIP.Addr().Is4() && nodeIP.IsSingleIP() {
-			return nodeIP.Addr().String()
-		}
-	}
-	for _, nodeIP := range who.Node.Addresses {
-		if nodeIP.IsSingleIP() {
-			return nodeIP.Addr().String()
-		}
-	}
-	return ""
-}
-
-func root(w http.ResponseWriter, r *http.Request) {
-	if r.TLS == nil && *httpsAddr != "" {
-		host := r.Host
-		if strings.Contains(r.Host, "100.101.102.103") ||
-			strings.Contains(r.Host, "hello.ipn.dev") {
-			host = "hello.ts.net"
-		}
-		http.Redirect(w, r, "https://"+host, http.StatusFound)
-		return
-	}
-	if r.RequestURI != "/" {
-		http.Redirect(w, r, "/", http.StatusFound)
-		return
-	}
-	if r.TLS != nil && *httpsAddr != "" && strings.Contains(r.Host, "hello.ipn.dev") {
-		http.Redirect(w, r, "https://hello.ts.net", http.StatusFound)
-		return
-	}
-	tmpl, err := getTmpl()
-	if err != nil {
-		w.Header().Set("Content-Type", "text/plain")
-		http.Error(w, "template error: "+err.Error(), 500)
-		return
-	}
-
-	who, err := localClient.WhoIs(r.Context(), r.RemoteAddr)
-	var data tmplData
-	if err != nil {
-		if devMode() {
-			log.Printf("warning: using fake data in dev mode due to whois lookup error: %v", err)
-			data = tmplData{
-				DisplayName:   "Taily Scalerson",
-				LoginName:     "taily@scaler.son",
-				ProfilePicURL: "https://placekitten.com/200/200",
-				MachineName:   "scaled",
-				MachineOS:     "Linux",
-				IP:            "100.1.2.3",
-			}
-		} else {
-			log.Printf("whois(%q) error: %v", r.RemoteAddr, err)
-			http.Error(w, "Your Tailscale works, but we failed to look you up.", 500)
-			return
-		}
-	} else {
-		data = tmplData{
-			DisplayName:   who.UserProfile.DisplayName,
-			LoginName:     who.UserProfile.LoginName,
-			ProfilePicURL: who.UserProfile.ProfilePicURL,
-			MachineName:   firstLabel(who.Node.ComputedName),
-			MachineOS:     who.Node.Hostinfo.OS(),
-			IP:            tailscaleIP(who),
-		}
-	}
-	w.Header().Set("Content-Type", "text/html; charset=utf-8")
-	tmpl.Execute(w, data)
-}
-
-// firstLabel s up until the first period, if any.
-func firstLabel(s string) string {
-	s, _, _ = strings.Cut(s, ".")
-	return s
 }

cmd/hello/hello.tmpl.html

@@ -1,438 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-  <meta charset="utf-8">
-  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
-  <title>Hello from Tailscale</title>
-  <style>
-    html,
-    body {
-      margin: 0;
-      padding: 0;
-    }
-
-    body {
-      font-family: Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
-      font-size: 100%;
-      -webkit-font-smoothing: antialiased;
-      -moz-osx-font-smoothing: grayscale;
-    }
-
-    html,
-    body,
-    main {
-      height: 100%;
-    }
-
-    *,
-    ::before,
-    ::after {
-      box-sizing: border-box;
-      border-width: 0;
-      border-style: solid;
-      border-color: #dad6d5;
-    }
-
-    h1,
-    h2,
-    h3,
-    h4,
-    h5,
-    h6 {
-      margin: 0;
-      font-size: 1rem;
-      font-weight: inherit;
-    }
-
-    a {
-      color: inherit;
-    }
-
-    p {
-      margin: 0;
-    }
-
-    main {
-      display: flex;
-      flex-direction: column;
-      justify-content: center;
-      align-items: center;
-      max-width: 24rem;
-      width: 95%;
-      margin-left: auto;
-      margin-right: auto;
-    }
-
-    .p-2 {
-      padding: 0.5rem;
-    }
-
-    .p-4 {
-      padding: 1rem;
-    }
-
-    .px-2 {
-      padding-left: 0.5rem;
-      padding-right: 0.5rem;
-    }
-
-    .pl-3 {
-      padding-left: 0.75rem;
-    }
-
-    .pr-3 {
-      padding-right: 0.75rem;
-    }
-
-    .pt-4 {
-      padding-top: 1rem;
-    }
-
-    .mr-2 {
-      margin-right: 0.5rem;
-      ;
-    }
-
-    .mb-1 {
-      margin-bottom: 0.25rem;
-    }
-
-    .mb-2 {
-      margin-bottom: 0.5rem;
-    }
-
-    .mb-4 {
-      margin-bottom: 1rem;
-    }
-
-    .mb-6 {
-      margin-bottom: 1.5rem;
-    }
-
-    .mb-8 {
-      margin-bottom: 2rem;
-    }
-
-    .mb-12 {
-      margin-bottom: 3rem;
-    }
-
-    .width-full {
-      width: 100%;
-    }
-
-    .min-width-0 {
-      min-width: 0;
-    }
-
-    .rounded-lg {
-      border-radius: 0.5rem;
-    }
-
-    .relative {
-      position: relative;
-    }
-
-    .flex {
-      display: flex;
-    }
-
-    .justify-between {
-      justify-content: space-between;
-    }
-
-    .items-center {
-      align-items: center;
-    }
-
-    .border {
-      border-width: 1px;
-    }
-
-    .border-t-1 {
-      border-top-width: 1px;
-    }
-
-    .border-gray-100 {
-      border-color: #f7f5f4;
-    }
-
-    .border-gray-200 {
-      border-color: #eeebea;
-    }
-
-    .border-gray-300 {
-      border-color: #dad6d5;
-    }
-
-    .bg-white {
-      background-color: white;
-    }
-
-    .bg-gray-0 {
-      background-color: #faf9f8;
-    }
-
-    .bg-gray-100 {
-      background-color: #f7f5f4;
-    }
-
-    .text-green-600 {
-      color: #0d4b3b;
-    }
-
-    .text-blue-600 {
-      color: #3f5db3;
-    }
-
-    .hover\:text-blue-800:hover {
-      color: #253570;
-    }
-
-    .text-gray-600 {
-      color: #444342;
-    }
-
-    .text-gray-700 {
-      color: #2e2d2d;
-    }
-
-    .text-gray-800 {
-      color: #232222;
-    }
-
-    .text-center {
-      text-align: center;
-    }
-
-    .text-sm {
-      font-size: 0.875rem;
-    }
-
-    .font-title {
-      font-size: 1.25rem;
-      letter-spacing: -0.025em;
-    }
-
-    .font-semibold {
-      font-weight: 600;
-    }
-
-    .font-medium {
-      font-weight: 500;
-    }
-
-    .font-regular {
-      font-weight: 400;
-    }
-
-    .truncate {
-      overflow: hidden;
-      text-overflow: ellipsis;
-      white-space: nowrap;
-    }
-
-    .overflow-hidden {
-      overflow: hidden;
-    }
-
-    .profile-pic {
-      width: 2.5rem;
-      height: 2.5rem;
-      border-radius: 9999px;
-      background-size: cover;
-      margin-right: 0.5rem;
-      flex-shrink: 0;
-    }
-
-    .panel {
-      box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04);
-    }
-
-    .animate .panel {
-      transform: translateY(10%);
-      box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.0), 0 10px 10px -5px rgba(0, 0, 0, 0.0);
-      transition: transform 1200ms ease, opacity 1200ms ease, box-shadow 1200ms ease;
-    }
-
-    .animate .panel-interior {
-      opacity: 0.0;
-      transition: opacity 1200ms ease;
-    }
-
-    .animate .logo {
-      transform: translateY(2rem);
-      opacity: 0.0;
-      transition: transform 1200ms ease, opacity 1200ms ease;
-    }
-
-    .animate .header-title {
-      transform: translateY(1.6rem);
-      opacity: 0.0;
-      transition: transform 1200ms ease, opacity 1200ms ease;
-    }
-
-    .animate .header-text {
-      transform: translateY(1.2rem);
-      opacity: 0.0;
-      transition: transform 1200ms ease, opacity 1200ms ease;
-    }
-
-    .animate .footer {
-      transform: translateY(-0.5rem);
-      opacity: 0.0;
-      transition: transform 1200ms ease, opacity 1200ms ease;
-    }
-
-    .animating .panel {
-      transform: translateY(0);
-      opacity: 1.0;
-      box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04);
-    }
-
-    .animating .panel-interior {
-      opacity: 1.0;
-    }
-
-    .animating .spinner {
-      opacity: 0.0;
-    }
-
-    .animating .logo,
-    .animating .header-title,
-    .animating .header-text,
-    .animating .footer {
-      transform: translateY(0);
-      opacity: 1.0;
-    }
-
-    .spinner {
-      display: inline-flex;
-      position: absolute;
-      top: 50%;
-      left: 50%;
-      transform: translate(-50%, -50%);
-      align-items: center;
-      transition: opacity 200ms ease;
-    }
-
-    .spinner span {
-      display: inline-block;
-      background-color: currentColor;
-      border-radius: 9999px;
-      animation-name: loading-dots-blink;
-      animation-duration: 1.4s;
-      animation-iteration-count: infinite;
-      animation-fill-mode: both;
-      width: 0.35em;
-      height: 0.35em;
-      margin: 0 0.15em;
-    }
-
-    .spinner span:nth-child(2) {
-      animation-delay: 200ms;
-    }
-
-    .spinner span:nth-child(3) {
-      animation-delay: 400ms;
-    }
-
-    .spinner {
-      display: none;
-    }
-
-    .animate .spinner {
-      display: inline-flex;
-    }
-
-    @keyframes loading-dots-blink {
-      0% {
-        opacity: 0.2;
-      }
-      20% {
-        opacity: 1;
-      }
-      100% {
-        opacity: 0.2;
-      }
-    }
-
-    @media (prefers-reduced-motion) {
-      * {
-        animation-duration: 0ms !important;
-        transition-duration: 0ms !important;
-        transition-delay: 0ms !important;
-      }
-    }
-  </style>
-</head>
-<body class="bg-gray-100">
-  <script>
-    (function() {
-      var lastSeen = localStorage.getItem("lastSeen");
-      if (!lastSeen) {
-        document.body.classList.add("animate");
-        window.addEventListener("load", function () {
-          setTimeout(function () {
-            document.body.classList.add("animating");
-            localStorage.setItem("lastSeen", Date.now());
-          }, 100);
-        });
-      }
-    })();
-  </script>
-  <main class="text-gray-800">
-    <svg class="logo mb-6" width="28" height="28" viewBox="0 0 22 22" fill="none" xmlns="http://www.w3.org/2000/svg">
-      <circle opacity="0.2" cx="3.4" cy="3.25" r="2.7" fill="currentColor" />
-      <circle cx="3.4" cy="11.3" r="2.7" fill="currentColor" />
-      <circle opacity="0.2" cx="3.4" cy="19.5" r="2.7" fill="currentColor" />
-      <circle cx="11.5" cy="11.3" r="2.7" fill="currentColor" />
-      <circle cx="11.5" cy="19.5" r="2.7" fill="currentColor" />
-      <circle opacity="0.2" cx="11.5" cy="3.25" r="2.7" fill="currentColor" />
-      <circle opacity="0.2" cx="19.5" cy="3.25" r="2.7" fill="currentColor" />
-      <circle cx="19.5" cy="11.3" r="2.7" fill="currentColor" />
-      <circle opacity="0.2" cx="19.5" cy="19.5" r="2.7" fill="currentColor" />
-    </svg>
-    <header class="mb-8 text-center">
-      <h1 class="header-title font-title font-semibold mb-2">You're connected over Tailscale!</h1>
-      <p class="header-text">This device is signed in as…</p>
-    </header>
-    <div class="panel relative bg-white rounded-lg width-full shadow-xl mb-8 p-4">
-      <div class="spinner text-gray-600">
-        <span></span>
-        <span></span>
-        <span></span>
-      </div>
-      <div class="panel-interior flex items-center width-full min-width-0 p-2 mb-4">
-        <div class="profile-pic bg-gray-100" style="background-image: url({{.ProfilePicURL}});"></div>
-        <div class="overflow-hidden">
-          {{ with .DisplayName }}
-          <h4 class="font-semibold truncate">{{.}}</h4>
-          {{ end }}
-          <h5 class="text-gray-600 truncate">{{.LoginName}}</h5>
-        </div>
-      </div>
-      <div
-        class="panel-interior border border-gray-200 bg-gray-0 rounded-lg p-2 pl-3 pr-3 mb-2 width-full flex justify-between items-center">
-        <div class="flex items-center min-width-0">
-          <svg class="text-gray-600 mr-2" xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none"
-            stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
-            <rect x="2" y="2" width="20" height="8" rx="2" ry="2"></rect>
-            <rect x="2" y="14" width="20" height="8" rx="2" ry="2"></rect>
-            <line x1="6" y1="6" x2="6.01" y2="6"></line>
-            <line x1="6" y1="18" x2="6.01" y2="18"></line>
-          </svg>
-          <h4 class="font-semibold truncate mr-2">{{.MachineName}}</h4>
-        </div>
-        <h5>{{.IP}}</h5>
-      </div>
-    </div>
-    <footer class="footer text-gray-600 text-center mb-12">
-      <p>Read about <a href="https://tailscale.com/kb/1017/install#advanced-features" class="text-blue-600 hover:text-blue-800"
-          target="_blank">what you can do next &rarr;</a></p>
-      <p>Read about <a href="https://tailscale.com/kb/1073/hello" class="text-blue-600 hover:text-blue-800"
-          target="_blank">the Hello service &rarr;</a></p>
-    </footer>
-  </main>
-</body>
-</html>

cmd/hello/helloserver/hello.tmpl.html

@@ -0,0 +1,71 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
+  <title>Hello from Tailscale</title>
+  <link type="text/css" rel="stylesheet" href="/static/style.css">
+  <script src="/static/script.js" defer></script>
+</head>
+<body class="bg-gray-100">
+  <main class="text-gray-800">
+    <svg class="logo mb-6" width="28" height="28" viewBox="0 0 22 22" fill="none" xmlns="http://www.w3.org/2000/svg">
+      <circle opacity="0.2" cx="3.4" cy="3.25" r="2.7" fill="currentColor" />
+      <circle cx="3.4" cy="11.3" r="2.7" fill="currentColor" />
+      <circle opacity="0.2" cx="3.4" cy="19.5" r="2.7" fill="currentColor" />
+      <circle cx="11.5" cy="11.3" r="2.7" fill="currentColor" />
+      <circle cx="11.5" cy="19.5" r="2.7" fill="currentColor" />
+      <circle opacity="0.2" cx="11.5" cy="3.25" r="2.7" fill="currentColor" />
+      <circle opacity="0.2" cx="19.5" cy="3.25" r="2.7" fill="currentColor" />
+      <circle cx="19.5" cy="11.3" r="2.7" fill="currentColor" />
+      <circle opacity="0.2" cx="19.5" cy="19.5" r="2.7" fill="currentColor" />
+    </svg>
+    <header class="mb-8 text-center">
+      <h1 class="header-title font-title font-semibold mb-2">You're connected over Tailscale!</h1>
+      <p class="header-text">This device is signed in as…</p>
+    </header>
+    <div class="panel relative bg-white rounded-lg width-full shadow-xl mb-8 p-4">
+      <div class="spinner text-gray-600">
+        <span></span>
+        <span></span>
+        <span></span>
+      </div>
+      <div class="panel-interior flex items-center width-full min-width-0 p-2 mb-4">
+        <div class="profile-pic bg-gray-100">
+            <img
+                    src="{{.ProfilePicURL}}"
+                    alt="Profile picture"
+                    class="profile-pic-img"
+            >
+        </div>
+        <div class="overflow-hidden">
+          {{ with .DisplayName }}
+          <h4 class="font-semibold truncate">{{.}}</h4>
+          {{ end }}
+          <h5 class="text-gray-600 truncate">{{.LoginName}}</h5>
+        </div>
+      </div>
+      <div
+        class="panel-interior border border-gray-200 bg-gray-0 rounded-lg p-2 pl-3 pr-3 mb-2 width-full flex justify-between items-center">
+        <div class="flex items-center min-width-0">
+          <svg class="text-gray-600 mr-2" xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none"
+            stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
+            <rect x="2" y="2" width="20" height="8" rx="2" ry="2"></rect>
+            <rect x="2" y="14" width="20" height="8" rx="2" ry="2"></rect>
+            <line x1="6" y1="6" x2="6.01" y2="6"></line>
+            <line x1="6" y1="18" x2="6.01" y2="18"></line>
+          </svg>
+          <h4 class="font-semibold truncate mr-2">{{.MachineName}}</h4>
+        </div>
+        <h5>{{.IP}}</h5>
+      </div>
+    </div>
+    <footer class="footer text-gray-600 text-center mb-12">
+      <p>Read about <a href="https://tailscale.com/kb/1017/install#advanced-features" class="text-blue-600 hover:text-blue-800"
+          target="_blank">what you can do next &rarr;</a></p>
+      <p>Read about <a href="https://tailscale.com/kb/1073/hello" class="text-blue-600 hover:text-blue-800"
+          target="_blank">the Hello service &rarr;</a></p>
+    </footer>
+  </main>
+</body>
+</html>

cmd/hello/helloserver/helloserver.go

@@ -0,0 +1,157 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+// Package helloserver implements the HTTP server behind hello.ts.net.
+package helloserver
+
+import (
+	"crypto/tls"
+	"embed"
+	"html/template"
+	"log"
+	"net/http"
+	"strings"
+	"time"
+
+	"tailscale.com/client/local"
+	"tailscale.com/client/tailscale/apitype"
+	"tailscale.com/tailcfg"
+)
+
+//go:embed hello.tmpl.html
+var embeddedTemplate string
+
+//go:embed static/*
+var staticFiles embed.FS
+
+var staticHandler = http.FileServerFS(staticFiles)
+
+var tmpl = template.Must(template.New("home").Parse(embeddedTemplate))
+
+// Server is an HTTP server for hello.ts.net.
+//
+// The zero value is not valid; populate at least one of HTTPAddr or HTTPSAddr
+// before calling Run.
+type Server struct {
+	// HTTPAddr is the address to run an HTTP server on, or empty for none.
+	HTTPAddr string
+
+	// HTTPSAddr is the address to run an HTTPS server on, or empty for none.
+	HTTPSAddr string
+
+	// LocalClient is used to look up the identity of incoming requests and
+	// to obtain TLS certificates. If nil, the zero value of local.Client is
+	// used.
+	LocalClient *local.Client
+}
+
+func (s *Server) localClient() *local.Client {
+	if s.LocalClient != nil {
+		return s.LocalClient
+	}
+	return &local.Client{}
+}
+
+// Run starts the configured HTTP and HTTPS servers and blocks until one of
+// them returns an error.
+func (s *Server) Run() error {
+	errc := make(chan error, 1)
+	if s.HTTPAddr != "" {
+		log.Printf("running HTTP server on %s", s.HTTPAddr)
+		go func() {
+			errc <- http.ListenAndServe(s.HTTPAddr, s)
+		}()
+	}
+	if s.HTTPSAddr != "" {
+		log.Printf("running HTTPS server on %s", s.HTTPSAddr)
+		go func() {
+			hs := &http.Server{
+				Addr:    s.HTTPSAddr,
+				Handler: s,
+				TLSConfig: &tls.Config{
+					GetCertificate: s.localClient().GetCertificate,
+				},
+				IdleTimeout:       30 * time.Second,
+				ReadHeaderTimeout: 20 * time.Second,
+				MaxHeaderBytes:    10 << 10,
+			}
+			errc <- hs.ListenAndServeTLS("", "")
+		}()
+	}
+	return <-errc
+}
+
+type tmplData struct {
+	DisplayName   string // "Foo Barberson"
+	LoginName     string // "foo@bar.com"
+	ProfilePicURL string // "https://..."
+	MachineName   string // "imac5k"
+	MachineOS     string // "Linux"
+	IP            string // "100.2.3.4"
+}
+
+func tailscaleIP(who *apitype.WhoIsResponse) string {
+	if who == nil {
+		return ""
+	}
+	vals, err := tailcfg.UnmarshalNodeCapJSON[string](who.Node.CapMap, tailcfg.NodeAttrNativeIPV4)
+	if err == nil && len(vals) > 0 {
+		return vals[0]
+	}
+	for _, nodeIP := range who.Node.Addresses {
+		if nodeIP.Addr().Is4() && nodeIP.IsSingleIP() {
+			return nodeIP.Addr().String()
+		}
+	}
+	for _, nodeIP := range who.Node.Addresses {
+		if nodeIP.IsSingleIP() {
+			return nodeIP.Addr().String()
+		}
+	}
+	return ""
+}
+
+// ServeHTTP implements http.Handler.
+func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	if r.TLS == nil && s.HTTPSAddr != "" {
+		host := r.Host
+		if strings.Contains(r.Host, "100.101.102.103") {
+			host = "hello.ts.net"
+		}
+		http.Redirect(w, r, "https://"+host, http.StatusFound)
+		return
+	}
+
+	if strings.HasPrefix(r.RequestURI, "/static/") {
+		staticHandler.ServeHTTP(w, r)
+		return
+	}
+
+	if r.RequestURI != "/" {
+		http.Redirect(w, r, "/", http.StatusFound)
+		return
+	}
+
+	who, err := s.localClient().WhoIs(r.Context(), r.RemoteAddr)
+	if err != nil {
+		log.Printf("whois(%q) error: %v", r.RemoteAddr, err)
+		http.Error(w, "Your Tailscale works, but we failed to look you up.", 500)
+		return
+	}
+	data := tmplData{
+		DisplayName:   who.UserProfile.DisplayName,
+		LoginName:     who.UserProfile.LoginName,
+		ProfilePicURL: who.UserProfile.ProfilePicURL,
+		MachineName:   firstLabel(who.Node.ComputedName),
+		MachineOS:     who.Node.Hostinfo.OS(),
+		IP:            tailscaleIP(who),
+	}
+	w.Header().Set("Content-Type", "text/html; charset=utf-8")
+	tmpl.Execute(w, data)
+}
+
+// firstLabel returns s up until the first period, if any.
+func firstLabel(s string) string {
+	s, _, _ = strings.Cut(s, ".")
+	return s
+}

cmd/hello/helloserver/static/script.js

@@ -0,0 +1,12 @@
+(function () {
+    var lastSeen = localStorage.getItem("lastSeen");
+    if (!lastSeen) {
+        document.body.classList.add("animate");
+        window.addEventListener("load", function () {
+            setTimeout(function () {
+                document.body.classList.add("animating");
+                localStorage.setItem("lastSeen", Date.now());
+            }, 100);
+        });
+    }
+})();

cmd/hello/helloserver/static/style.css

@@ -0,0 +1,366 @@
+html,
+body {
+    margin: 0;
+    padding: 0;
+}
+
+body {
+    font-family: Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+    font-size: 100%;
+    -webkit-font-smoothing: antialiased;
+    -moz-osx-font-smoothing: grayscale;
+}
+
+html,
+body,
+main {
+    height: 100%;
+}
+
+*,
+::before,
+::after {
+    box-sizing: border-box;
+    border-width: 0;
+    border-style: solid;
+    border-color: #dad6d5;
+}
+
+h1,
+h2,
+h3,
+h4,
+h5,
+h6 {
+    margin: 0;
+    font-size: 1rem;
+    font-weight: inherit;
+}
+
+a {
+    color: inherit;
+}
+
+p {
+    margin: 0;
+}
+
+main {
+    display: flex;
+    flex-direction: column;
+    justify-content: center;
+    align-items: center;
+    max-width: 24rem;
+    width: 95%;
+    margin-left: auto;
+    margin-right: auto;
+}
+
+.p-2 {
+    padding: 0.5rem;
+}
+
+.p-4 {
+    padding: 1rem;
+}
+
+.px-2 {
+    padding-left: 0.5rem;
+    padding-right: 0.5rem;
+}
+
+.pl-3 {
+    padding-left: 0.75rem;
+}
+
+.pr-3 {
+    padding-right: 0.75rem;
+}
+
+.pt-4 {
+    padding-top: 1rem;
+}
+
+.mr-2 {
+    margin-right: 0.5rem;
+;
+}
+
+.mb-1 {
+    margin-bottom: 0.25rem;
+}
+
+.mb-2 {
+    margin-bottom: 0.5rem;
+}
+
+.mb-4 {
+    margin-bottom: 1rem;
+}
+
+.mb-6 {
+    margin-bottom: 1.5rem;
+}
+
+.mb-8 {
+    margin-bottom: 2rem;
+}
+
+.mb-12 {
+    margin-bottom: 3rem;
+}
+
+.width-full {
+    width: 100%;
+}
+
+.min-width-0 {
+    min-width: 0;
+}
+
+.rounded-lg {
+    border-radius: 0.5rem;
+}
+
+.relative {
+    position: relative;
+}
+
+.flex {
+    display: flex;
+}
+
+.justify-between {
+    justify-content: space-between;
+}
+
+.items-center {
+    align-items: center;
+}
+
+.border {
+    border-width: 1px;
+}
+
+.border-t-1 {
+    border-top-width: 1px;
+}
+
+.border-gray-100 {
+    border-color: #f7f5f4;
+}
+
+.border-gray-200 {
+    border-color: #eeebea;
+}
+
+.border-gray-300 {
+    border-color: #dad6d5;
+}
+
+.bg-white {
+    background-color: white;
+}
+
+.bg-gray-0 {
+    background-color: #faf9f8;
+}
+
+.bg-gray-100 {
+    background-color: #f7f5f4;
+}
+
+.text-green-600 {
+    color: #0d4b3b;
+}
+
+.text-blue-600 {
+    color: #3f5db3;
+}
+
+.hover\:text-blue-800:hover {
+    color: #253570;
+}
+
+.text-gray-600 {
+    color: #444342;
+}
+
+.text-gray-700 {
+    color: #2e2d2d;
+}
+
+.text-gray-800 {
+    color: #232222;
+}
+
+.text-center {
+    text-align: center;
+}
+
+.text-sm {
+    font-size: 0.875rem;
+}
+
+.font-title {
+    font-size: 1.25rem;
+    letter-spacing: -0.025em;
+}
+
+.font-semibold {
+    font-weight: 600;
+}
+
+.font-medium {
+    font-weight: 500;
+}
+
+.font-regular {
+    font-weight: 400;
+}
+
+.truncate {
+    overflow: hidden;
+    text-overflow: ellipsis;
+    white-space: nowrap;
+}
+
+.overflow-hidden {
+    overflow: hidden;
+}
+
+.profile-pic {
+    width: 2.5rem;
+    height: 2.5rem;
+    background-size: cover;
+    margin-right: 0.5rem;
+    flex-shrink: 0;
+}
+
+.profile-pic-img {
+    width: 100%;
+    height: 100%;
+    object-fit: cover;
+    display: block;
+    border-radius: 9999px;
+}
+
+.panel {
+    box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04);
+}
+
+.animate .panel {
+    transform: translateY(10%);
+    box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.0), 0 10px 10px -5px rgba(0, 0, 0, 0.0);
+    transition: transform 1200ms ease, opacity 1200ms ease, box-shadow 1200ms ease;
+}
+
+.animate .panel-interior {
+    opacity: 0.0;
+    transition: opacity 1200ms ease;
+}
+
+.animate .logo {
+    transform: translateY(2rem);
+    opacity: 0.0;
+    transition: transform 1200ms ease, opacity 1200ms ease;
+}
+
+.animate .header-title {
+    transform: translateY(1.6rem);
+    opacity: 0.0;
+    transition: transform 1200ms ease, opacity 1200ms ease;
+}
+
+.animate .header-text {
+    transform: translateY(1.2rem);
+    opacity: 0.0;
+    transition: transform 1200ms ease, opacity 1200ms ease;
+}
+
+.animate .footer {
+    transform: translateY(-0.5rem);
+    opacity: 0.0;
+    transition: transform 1200ms ease, opacity 1200ms ease;
+}
+
+.animating .panel {
+    transform: translateY(0);
+    opacity: 1.0;
+    box-shadow: 0 20px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04);
+}
+
+.animating .panel-interior {
+    opacity: 1.0;
+}
+
+.animating .spinner {
+    opacity: 0.0;
+}
+
+.animating .logo,
+.animating .header-title,
+.animating .header-text,
+.animating .footer {
+    transform: translateY(0);
+    opacity: 1.0;
+}
+
+.spinner {
+    display: inline-flex;
+    position: absolute;
+    top: 50%;
+    left: 50%;
+    transform: translate(-50%, -50%);
+    align-items: center;
+    transition: opacity 200ms ease;
+}
+
+.spinner span {
+    display: inline-block;
+    background-color: currentColor;
+    border-radius: 9999px;
+    animation-name: loading-dots-blink;
+    animation-duration: 1.4s;
+    animation-iteration-count: infinite;
+    animation-fill-mode: both;
+    width: 0.35em;
+    height: 0.35em;
+    margin: 0 0.15em;
+}
+
+.spinner span:nth-child(2) {
+    animation-delay: 200ms;
+}
+
+.spinner span:nth-child(3) {
+    animation-delay: 400ms;
+}
+
+.spinner {
+    display: none;
+}
+
+.animate .spinner {
+    display: inline-flex;
+}
+
+@keyframes loading-dots-blink {
+    0% {
+        opacity: 0.2;
+    }
+    20% {
+        opacity: 1;
+    }
+    100% {
+        opacity: 0.2;
+    }
+}
+
+@media (prefers-reduced-motion) {
+    * {
+        animation-duration: 0ms !important;
+        transition-duration: 0ms !important;
+        transition-delay: 0ms !important;
+    }
+}

cmd/k8s-operator/api-server-proxy-pg.go

@@ -23,10 +23,11 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/internal/client/tailscale"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/k8s-proxy/conf"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
@@ -51,7 +52,7 @@ type KubeAPIServerTSServiceReconciler struct {
 	client.Client
 	recorder    record.EventRecorder
 	logger      *zap.SugaredLogger
-	tsClient    tsClient
+	clients     ClientProvider
 	tsNamespace string
 	defaultTags []string
 	operatorID  string // stableID of the operator's Tailscale device
@@ -77,15 +78,14 @@ func (r *KubeAPIServerTSServiceReconciler) Reconcile(ctx context.Context, req re
 
 	serviceName := serviceNameForAPIServerProxy(pg)
 	logger = logger.With("Tailscale Service", serviceName)
-
+	tsClient, err := r.clients.For(pg.Spec.Tailnet)
-	tailscaleClient, err := r.getClient(ctx, pg.Spec.Tailnet)
 	if err != nil {
 		return res, fmt.Errorf("failed to get tailscale client: %w", err)
 	}
 
 	if markedForDeletion(pg) {
 		logger.Debugf("ProxyGroup is being deleted, ensuring any created resources are cleaned up")
-		if err = r.maybeCleanup(ctx, serviceName, pg, logger, tailscaleClient); err != nil && strings.Contains(err.Error(), optimisticLockErrorMsg) {
+		if err = r.maybeCleanup(ctx, serviceName, pg, logger, tsClient); err != nil && strings.Contains(err.Error(), optimisticLockErrorMsg) {
 			logger.Infof("optimistic lock error, retrying: %s", err)
 			return res, nil
 		}
@@ -93,7 +93,7 @@ func (r *KubeAPIServerTSServiceReconciler) Reconcile(ctx context.Context, req re
 		return res, err
 	}
 
-	err = r.maybeProvision(ctx, serviceName, pg, logger, tailscaleClient)
+	err = r.maybeProvision(ctx, serviceName, pg, logger, tsClient)
 	if err != nil {
 		if strings.Contains(err.Error(), optimisticLockErrorMsg) {
 			logger.Infof("optimistic lock error, retrying: %s", err)
@@ -105,31 +105,15 @@ func (r *KubeAPIServerTSServiceReconciler) Reconcile(ctx context.Context, req re
 	return reconcile.Result{}, nil
 }
 
-// getClient returns the appropriate Tailscale client for the given tailnet.
-// If no tailnet is specified, returns the default client.
-func (r *KubeAPIServerTSServiceReconciler) getClient(ctx context.Context, tailnetName string) (tsClient,
-	error) {
-	if tailnetName == "" {
-		return r.tsClient, nil
-	}
-
-	tc, _, err := clientForTailnet(ctx, r.Client, r.tsNamespace, tailnetName)
-	if err != nil {
-		return nil, err
-	}
-
-	return tc, nil
-}
-
 // maybeProvision ensures that a Tailscale Service for this ProxyGroup exists
 // and is up to date.
 //
 // Returns true if the operation resulted in a Tailscale Service update.
-func (r *KubeAPIServerTSServiceReconciler) maybeProvision(ctx context.Context, serviceName tailcfg.ServiceName, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsClient) (err error) {
+func (r *KubeAPIServerTSServiceReconciler) maybeProvision(ctx context.Context, serviceName tailcfg.ServiceName, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsclient.Client) (err error) {
 	var dnsName string
 	oldPGStatus := pg.Status.DeepCopy()
 	defer func() {
-		podsAdvertising, podsErr := numberPodsAdvertising(ctx, r.Client, r.tsNamespace, pg.Name, serviceName)
+		podsAdvertising, podsErr := numberPodsAdvertising(ctx, r.Client, r.tsNamespace, pg.Name, serviceName.String())
 		if podsErr != nil {
 			err = errors.Join(err, fmt.Errorf("failed to get number of advertised Pods: %w", podsErr))
 			// Continue, updating the status with the best available information.
@@ -177,8 +161,8 @@ func (r *KubeAPIServerTSServiceReconciler) maybeProvision(ctx context.Context, s
 
 	// 1. Check there isn't a Tailscale Service with the same hostname
 	// already created and not owned by this ProxyGroup.
-	existingTSSvc, err := tsClient.GetVIPService(ctx, serviceName)
+	existingTSSvc, err := tsClient.VIPServices().Get(ctx, serviceName.String())
-	if err != nil && !isErrorTailscaleServiceNotFound(err) {
+	if err != nil && !tailscale.IsNotFound(err) {
 		return fmt.Errorf("error getting Tailscale Service %q: %w", serviceName, err)
 	}
 
@@ -202,8 +186,8 @@ func (r *KubeAPIServerTSServiceReconciler) maybeProvision(ctx context.Context, s
 		serviceTags = pg.Spec.Tags.Stringify()
 	}
 
-	tsSvc := &tailscale.VIPService{
+	tsSvc := tailscale.VIPService{
-		Name:        serviceName,
+		Name:        serviceName.String(),
 		Tags:        serviceTags,
 		Ports:       []string{"tcp:443"},
 		Comment:     managedTSServiceComment,
@@ -216,10 +200,10 @@ func (r *KubeAPIServerTSServiceReconciler) maybeProvision(ctx context.Context, s
 	// 2. Ensure the Tailscale Service exists and is up to date.
 	if existingTSSvc == nil ||
 		!slices.Equal(tsSvc.Tags, existingTSSvc.Tags) ||
-		!ownersAreSetAndEqual(tsSvc, existingTSSvc) ||
+		!ownersAreSetAndEqual(tsSvc, *existingTSSvc) ||
 		!slices.Equal(tsSvc.Ports, existingTSSvc.Ports) {
 		logger.Infof("Ensuring Tailscale Service exists and is up to date")
-		if err = tsClient.CreateOrUpdateVIPService(ctx, tsSvc); err != nil {
+		if err = tsClient.VIPServices().CreateOrUpdate(ctx, tsSvc); err != nil {
 			return fmt.Errorf("error creating Tailscale Service: %w", err)
 		}
 	}
@@ -248,10 +232,10 @@ func (r *KubeAPIServerTSServiceReconciler) maybeProvision(ctx context.Context, s
 }
 
 // maybeCleanup ensures that any resources, such as a Tailscale Service created for this Service, are cleaned up when the
-// Service is being deleted or is unexposed. The cleanup is safe for a multi-cluster setup- the Tailscale Service is only
+// Service is being deleted or is unexposed. The cleanup is safe for a multi-cluster setup. The Tailscale Service is only
 // deleted if it does not contain any other owner references. If it does, the cleanup only removes the owner reference
 // corresponding to this Service.
-func (r *KubeAPIServerTSServiceReconciler) maybeCleanup(ctx context.Context, serviceName tailcfg.ServiceName, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsClient) (err error) {
+func (r *KubeAPIServerTSServiceReconciler) maybeCleanup(ctx context.Context, serviceName tailcfg.ServiceName, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, client tsclient.Client) (err error) {
 	ix := slices.Index(pg.Finalizers, proxyPGFinalizerName)
 	if ix < 0 {
 		logger.Debugf("no finalizer, nothing to do")
@@ -265,7 +249,7 @@ func (r *KubeAPIServerTSServiceReconciler) maybeCleanup(ctx context.Context, ser
 		}
 	}()
 
-	if _, err = cleanupTailscaleService(ctx, tsClient, serviceName, r.operatorID, logger); err != nil {
+	if _, err = cleanupTailscaleService(ctx, client, serviceName.String(), r.operatorID, logger); err != nil {
 		return fmt.Errorf("error deleting Tailscale Service: %w", err)
 	}
 
@@ -278,16 +262,16 @@ func (r *KubeAPIServerTSServiceReconciler) maybeCleanup(ctx context.Context, ser
 
 // maybeDeleteStaleServices deletes Services that have previously been created for
 // this ProxyGroup but are no longer needed.
-func (r *KubeAPIServerTSServiceReconciler) maybeDeleteStaleServices(ctx context.Context, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsClient) error {
+func (r *KubeAPIServerTSServiceReconciler) maybeDeleteStaleServices(ctx context.Context, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsclient.Client) error {
 	serviceName := serviceNameForAPIServerProxy(pg)
 
-	svcs, err := tsClient.ListVIPServices(ctx)
+	svcs, err := tsClient.VIPServices().List(ctx)
 	if err != nil {
 		return fmt.Errorf("error listing Tailscale Services: %w", err)
 	}
 
-	for _, svc := range svcs.VIPServices {
+	for _, svc := range svcs {
-		if svc.Name == serviceName {
+		if svc.Name == serviceName.String() {
 			continue
 		}
 
@@ -306,11 +290,11 @@ func (r *KubeAPIServerTSServiceReconciler) maybeDeleteStaleServices(ctx context.
 		}
 
 		logger.Infof("Deleting Tailscale Service %s", svc.Name)
-		if err = tsClient.DeleteVIPService(ctx, svc.Name); err != nil && !isErrorTailscaleServiceNotFound(err) {
+		if err = tsClient.VIPServices().Delete(ctx, svc.Name); err != nil && !tailscale.IsNotFound(err) {
 			return fmt.Errorf("error deleting Tailscale Service %s: %w", svc.Name, err)
 		}
 
-		if err = cleanupCertResources(ctx, r.Client, r.tsNamespace, svc.Name, pg); err != nil {
+		if err = cleanupCertResources(ctx, r.Client, r.tsNamespace, tailcfg.ServiceName(svc.Name), pg); err != nil {
 			return fmt.Errorf("failed to clean up cert resources: %w", err)
 		}
 	}

cmd/k8s-operator/api-server-proxy-pg_test.go

@@ -16,10 +16,11 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/internal/client/tailscale"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/k8s-proxy/conf"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
@@ -93,8 +94,10 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 		expectEqual(t, fc, pgCfgSecret)
 	}
 
-	ft := &fakeTSClient{}
+	ft := &fakeTSClient{
-	ingressTSSvc := &tailscale.VIPService{
+		vipServices: make(map[string]tailscale.VIPService),
+	}
+	ingressTSSvc := tailscale.VIPService{
 		Name:    "svc:some-ingress-hostname",
 		Comment: managedTSServiceComment,
 		Annotations: map[string]string{
@@ -105,11 +108,11 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 		Tags:  []string{"tag:k8s"},
 		Addrs: []string{"5.6.7.8"},
 	}
-	ft.CreateOrUpdateVIPService(t.Context(), ingressTSSvc)
+	ft.VIPServices().CreateOrUpdate(t.Context(), ingressTSSvc)
 
 	r := &KubeAPIServerTSServiceReconciler{
 		Client:      fc,
-		tsClient:    ft,
+		clients:     tsclient.NewProvider(ft),
 		defaultTags: []string{"tag:k8s"},
 		tsNamespace: ns,
 		logger:      zap.Must(zap.NewDevelopment()).Sugar(),
@@ -119,7 +122,7 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 	}
 
 	// Create a Tailscale Service that will conflict with the initial config.
-	if err := ft.CreateOrUpdateVIPService(t.Context(), &tailscale.VIPService{
+	if err := ft.VIPServices().CreateOrUpdate(t.Context(), tailscale.VIPService{
 		Name: "svc:" + pgName,
 	}); err != nil {
 		t.Fatalf("creating initial Tailscale Service: %v", err)
@@ -135,7 +138,7 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 	expectEqual(t, fc, pgCfgSecret) // Unchanged.
 
 	// Delete Tailscale Service; should see Service created and valid condition updated to true.
-	if err := ft.DeleteVIPService(t.Context(), "svc:"+pgName); err != nil {
+	if err := ft.VIPServices().Delete(t.Context(), "svc:"+pgName); err != nil {
 		t.Fatalf("deleting initial Tailscale Service: %v", err)
 	}
 
@@ -154,7 +157,7 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 
 	expectReconciled(t, r, "", pgName)
 
-	tsSvc, err := ft.GetVIPService(t.Context(), "svc:"+pgName)
+	tsSvc, err := ft.VIPServices().Get(t.Context(), "svc:"+pgName)
 	if err != nil {
 		t.Fatalf("getting Tailscale Service: %v", err)
 	}
@@ -223,15 +226,15 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 		p.Spec.KubeAPIServer = pg.Spec.KubeAPIServer
 	})
 	expectReconciled(t, r, "", pgName)
-	_, err = ft.GetVIPService(t.Context(), "svc:"+pgName)
+	_, err = ft.VIPServices().Get(t.Context(), "svc:"+pgName)
-	if !isErrorTailscaleServiceNotFound(err) {
+	if !tailscale.IsNotFound(err) {
 		t.Fatalf("Expected 404, got: %v", err)
 	}
-	tsSvc, err = ft.GetVIPService(t.Context(), updatedServiceName)
+	tsSvc, err = ft.VIPServices().Get(t.Context(), updatedServiceName.String())
 	if err != nil {
 		t.Fatalf("Expected renamed svc, got error: %v", err)
 	}
-	expectedTSSvc.Name = updatedServiceName
+	expectedTSSvc.Name = updatedServiceName.String()
 	if !reflect.DeepEqual(tsSvc, expectedTSSvc) {
 		t.Fatalf("expected Tailscale Service to be %+v, got %+v", expectedTSSvc, tsSvc)
 	}
@@ -269,17 +272,17 @@ func TestAPIServerProxyReconciler(t *testing.T) {
 	expectMissing[corev1.Secret](t, fc, ns, updatedDomain)
 	expectMissing[rbacv1.Role](t, fc, ns, updatedDomain)
 	expectMissing[rbacv1.RoleBinding](t, fc, ns, updatedDomain)
-	_, err = ft.GetVIPService(t.Context(), updatedServiceName)
+	_, err = ft.VIPServices().Get(t.Context(), updatedServiceName.String())
-	if !isErrorTailscaleServiceNotFound(err) {
+	if !tailscale.IsNotFound(err) {
 		t.Fatalf("Expected 404, got: %v", err)
 	}
 
 	// Ingress Tailscale Service should not be affected.
-	svc, err := ft.GetVIPService(t.Context(), ingressTSSvc.Name)
+	svc, err := ft.VIPServices().Get(t.Context(), ingressTSSvc.Name)
 	if err != nil {
 		t.Fatalf("getting ingress Tailscale Service: %v", err)
 	}
-	if !reflect.DeepEqual(svc, ingressTSSvc) {
+	if !reflect.DeepEqual(svc, &ingressTSSvc) {
 		t.Fatalf("expected ingress Tailscale Service to be unmodified %+v, got %+v", ingressTSSvc, svc)
 	}
 }
@@ -292,7 +295,6 @@ func TestExclusiveOwnerAnnotations(t *testing.T) {
 		},
 	}
 	const (
-		selfOperatorID = "self-id"
 		pg1Owner = `{"ownerRefs":[{"operatorID":"self-id","resource":{"kind":"ProxyGroup","name":"pg1","uid":"pg1-uid"}}]}`
 	)
 

cmd/k8s-operator/connector_test.go

@@ -19,7 +19,9 @@ import (
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tstest"
 	"tailscale.com/util/mak"
@@ -62,7 +64,7 @@ func TestConnector(t *testing.T) {
 		recorder: record.NewFakeRecorder(10),
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -252,7 +254,7 @@ func TestConnectorWithProxyClass(t *testing.T) {
 		clock:  cl,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -346,7 +348,7 @@ func TestConnectorWithAppConnector(t *testing.T) {
 		clock:  cl,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -446,7 +448,7 @@ func TestConnectorWithMultipleReplicas(t *testing.T) {
 		clock:  cl,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",

cmd/k8s-operator/depaware.txt

@@ -6,77 +6,6 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
    W 💣 github.com/alexbrainman/sspi                                 from github.com/alexbrainman/sspi/internal/common+
    W    github.com/alexbrainman/sspi/internal/common                 from github.com/alexbrainman/sspi/negotiate
    W 💣 github.com/alexbrainman/sspi/negotiate                       from tailscale.com/net/tshttpproxy
-        github.com/aws/aws-sdk-go-v2/aws                             from github.com/aws/aws-sdk-go-v2/aws/defaults+
-        github.com/aws/aws-sdk-go-v2/aws/defaults                    from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/aws/middleware                  from github.com/aws/aws-sdk-go-v2/aws/retry+
-        github.com/aws/aws-sdk-go-v2/aws/protocol/query              from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/aws-sdk-go-v2/aws/protocol/restjson           from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/aws/protocol/xml                from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/aws-sdk-go-v2/aws/ratelimit                   from github.com/aws/aws-sdk-go-v2/aws/retry
-        github.com/aws/aws-sdk-go-v2/aws/retry                       from github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client+
-        github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4          from github.com/aws/aws-sdk-go-v2/aws/signer/v4
-        github.com/aws/aws-sdk-go-v2/aws/signer/v4                   from github.com/aws/aws-sdk-go-v2/internal/auth/smithy+
-        github.com/aws/aws-sdk-go-v2/aws/transport/http              from github.com/aws/aws-sdk-go-v2/config+
-        github.com/aws/aws-sdk-go-v2/config                          from tailscale.com/wif
-        github.com/aws/aws-sdk-go-v2/credentials                     from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds        from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/credentials/endpointcreds       from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client from github.com/aws/aws-sdk-go-v2/credentials/endpointcreds
-        github.com/aws/aws-sdk-go-v2/credentials/processcreds        from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/credentials/ssocreds            from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/credentials/stscreds            from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/feature/ec2/imds                from github.com/aws/aws-sdk-go-v2/config+
-        github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config from github.com/aws/aws-sdk-go-v2/feature/ec2/imds
-        github.com/aws/aws-sdk-go-v2/internal/auth                   from github.com/aws/aws-sdk-go-v2/aws/signer/v4+
-        github.com/aws/aws-sdk-go-v2/internal/auth/smithy            from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/internal/configsources          from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/internal/context                from github.com/aws/aws-sdk-go-v2/aws/retry+
-        github.com/aws/aws-sdk-go-v2/internal/endpoints              from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/internal/endpoints/awsrulesfn   from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/internal/endpoints/v2           from github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints+
-        github.com/aws/aws-sdk-go-v2/internal/ini                    from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/aws-sdk-go-v2/internal/middleware             from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/aws-sdk-go-v2/internal/rand                   from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/aws-sdk-go-v2/internal/sdk                    from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/aws-sdk-go-v2/internal/sdkio                  from github.com/aws/aws-sdk-go-v2/credentials/processcreds
-        github.com/aws/aws-sdk-go-v2/internal/shareddefaults         from github.com/aws/aws-sdk-go-v2/config+
-        github.com/aws/aws-sdk-go-v2/internal/strings                from github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4
-        github.com/aws/aws-sdk-go-v2/internal/sync/singleflight      from github.com/aws/aws-sdk-go-v2/aws
-        github.com/aws/aws-sdk-go-v2/internal/timeconv               from github.com/aws/aws-sdk-go-v2/aws/retry
-        github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/aws-sdk-go-v2/service/internal/presigned-url  from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/aws-sdk-go-v2/service/sso                     from github.com/aws/aws-sdk-go-v2/config+
-        github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints  from github.com/aws/aws-sdk-go-v2/service/sso
-        github.com/aws/aws-sdk-go-v2/service/sso/types               from github.com/aws/aws-sdk-go-v2/service/sso
-        github.com/aws/aws-sdk-go-v2/service/ssooidc                 from github.com/aws/aws-sdk-go-v2/config+
-        github.com/aws/aws-sdk-go-v2/service/ssooidc/internal/endpoints from github.com/aws/aws-sdk-go-v2/service/ssooidc
-        github.com/aws/aws-sdk-go-v2/service/ssooidc/types           from github.com/aws/aws-sdk-go-v2/service/ssooidc
-        github.com/aws/aws-sdk-go-v2/service/sts                     from github.com/aws/aws-sdk-go-v2/config+
-        github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints  from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/aws-sdk-go-v2/service/sts/types               from github.com/aws/aws-sdk-go-v2/credentials/stscreds+
-        github.com/aws/smithy-go                                     from github.com/aws/aws-sdk-go-v2/aws/protocol/restjson+
-        github.com/aws/smithy-go/auth                                from github.com/aws/aws-sdk-go-v2/internal/auth+
-        github.com/aws/smithy-go/auth/bearer                         from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/smithy-go/context                             from github.com/aws/smithy-go/auth/bearer
-        github.com/aws/smithy-go/document                            from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/smithy-go/encoding                            from github.com/aws/smithy-go/encoding/json+
-        github.com/aws/smithy-go/encoding/httpbinding                from github.com/aws/aws-sdk-go-v2/aws/protocol/query+
-        github.com/aws/smithy-go/encoding/json                       from github.com/aws/aws-sdk-go-v2/service/ssooidc
-        github.com/aws/smithy-go/encoding/xml                        from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/smithy-go/endpoints                           from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/smithy-go/endpoints/private/rulesfn           from github.com/aws/aws-sdk-go-v2/service/sts
-        github.com/aws/smithy-go/internal/sync/singleflight          from github.com/aws/smithy-go/auth/bearer
-        github.com/aws/smithy-go/io                                  from github.com/aws/aws-sdk-go-v2/feature/ec2/imds+
-        github.com/aws/smithy-go/logging                             from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/smithy-go/metrics                             from github.com/aws/aws-sdk-go-v2/aws/retry+
-        github.com/aws/smithy-go/middleware                          from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/smithy-go/private/requestcompression          from github.com/aws/aws-sdk-go-v2/config
-        github.com/aws/smithy-go/ptr                                 from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/smithy-go/rand                                from github.com/aws/aws-sdk-go-v2/aws/middleware
-        github.com/aws/smithy-go/time                                from github.com/aws/aws-sdk-go-v2/service/sso+
-        github.com/aws/smithy-go/tracing                             from github.com/aws/aws-sdk-go-v2/aws/middleware+
-        github.com/aws/smithy-go/transport/http                      from github.com/aws/aws-sdk-go-v2/aws+
-        github.com/aws/smithy-go/transport/http/internal/io          from github.com/aws/smithy-go/transport/http
         github.com/beorn7/perks/quantile                             from github.com/prometheus/client_golang/prometheus
         github.com/blang/semver/v4                                   from k8s.io/component-base/metrics
      💣 github.com/cespare/xxhash/v2                                 from github.com/prometheus/client_golang/prometheus+
@@ -130,7 +59,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         github.com/google/gnostic-models/jsonschema                  from github.com/google/gnostic-models/compiler
         github.com/google/gnostic-models/openapiv2                   from k8s.io/client-go/discovery+
         github.com/google/gnostic-models/openapiv3                   from k8s.io/kube-openapi/pkg/handler3+
-        github.com/google/uuid                                       from github.com/prometheus-community/pro-bing+
+        github.com/google/uuid                                       from k8s.io/apimachinery/pkg/util/uuid+
         github.com/hdevalence/ed25519consensus                       from tailscale.com/tka
         github.com/huin/goupnp                                       from github.com/huin/goupnp/dcps/internetgateway2+
         github.com/huin/goupnp/dcps/internetgateway2                 from tailscale.com/net/portmapper
@@ -164,7 +93,6 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         github.com/pires/go-proxyproto                               from tailscale.com/ipn/ipnlocal+
         github.com/pkg/errors                                        from github.com/evanphx/json-patch/v5+
         github.com/pmezard/go-difflib/difflib                        from k8s.io/apimachinery/pkg/util/diff
-   D    github.com/prometheus-community/pro-bing                     from tailscale.com/wgengine/netstack
         github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil from github.com/prometheus/client_golang/prometheus/promhttp
         github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil/header from github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil
      💣 github.com/prometheus/client_golang/prometheus               from github.com/prometheus/client_golang/prometheus/collectors+
@@ -180,7 +108,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
   LD    github.com/prometheus/procfs/internal/util                   from github.com/prometheus/procfs
    L 💣 github.com/safchain/ethtool                                  from tailscale.com/net/netkernelconf
         github.com/spf13/pflag                                       from k8s.io/client-go/tools/clientcmd+
-   W 💣 github.com/tailscale/certstore                               from tailscale.com/control/controlclient
+  DW 💣 github.com/tailscale/certstore                               from tailscale.com/control/controlclient
    W 💣 github.com/tailscale/go-winio                                from tailscale.com/safesocket
    W 💣 github.com/tailscale/go-winio/internal/fs                    from github.com/tailscale/go-winio
    W 💣 github.com/tailscale/go-winio/internal/socket                from github.com/tailscale/go-winio
@@ -784,8 +712,9 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/appc                                           from tailscale.com/ipn/ipnlocal
      💣 tailscale.com/atomicfile                                     from tailscale.com/ipn+
         tailscale.com/client/local                                   from tailscale.com/client/tailscale+
-        tailscale.com/client/tailscale                               from tailscale.com/cmd/k8s-operator+
+        tailscale.com/client/tailscale                               from tailscale.com/internal/client/tailscale
         tailscale.com/client/tailscale/apitype                       from tailscale.com/client/tailscale+
+        tailscale.com/client/tailscale/v2                            from tailscale.com/cmd/k8s-operator+
         tailscale.com/client/web                                     from tailscale.com/ipn/ipnlocal
         tailscale.com/control/controlbase                            from tailscale.com/control/controlhttp+
         tailscale.com/control/controlclient                          from tailscale.com/ipn/ipnlocal+
@@ -804,11 +733,9 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/feature/buildfeatures                          from tailscale.com/wgengine/magicsock+
         tailscale.com/feature/c2n                                    from tailscale.com/tsnet
         tailscale.com/feature/condlite/expvar                        from tailscale.com/wgengine/magicsock
-        tailscale.com/feature/condregister/identityfederation        from tailscale.com/tsnet
         tailscale.com/feature/condregister/oauthkey                  from tailscale.com/tsnet
         tailscale.com/feature/condregister/portmapper                from tailscale.com/tsnet
         tailscale.com/feature/condregister/useproxy                  from tailscale.com/tsnet
-        tailscale.com/feature/identityfederation                     from tailscale.com/feature/condregister/identityfederation
         tailscale.com/feature/oauthkey                               from tailscale.com/feature/condregister/oauthkey
         tailscale.com/feature/portmapper                             from tailscale.com/feature/condregister/portmapper
         tailscale.com/feature/syspolicy                              from tailscale.com/logpolicy
@@ -816,7 +743,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/health                                         from tailscale.com/control/controlclient+
         tailscale.com/health/healthmsg                               from tailscale.com/ipn/ipnlocal
         tailscale.com/hostinfo                                       from tailscale.com/client/web+
-        tailscale.com/internal/client/tailscale                      from tailscale.com/cmd/k8s-operator+
+        tailscale.com/internal/client/tailscale                      from tailscale.com/feature/oauthkey+
         tailscale.com/ipn                                            from tailscale.com/client/local+
         tailscale.com/ipn/conffile                                   from tailscale.com/ipn/ipnlocal+
      💣 tailscale.com/ipn/ipnauth                                    from tailscale.com/ipn/ipnlocal+
@@ -839,6 +766,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/k8s-operator/sessionrecording/spdy             from tailscale.com/k8s-operator/sessionrecording
         tailscale.com/k8s-operator/sessionrecording/tsrecorder       from tailscale.com/k8s-operator/sessionrecording+
         tailscale.com/k8s-operator/sessionrecording/ws               from tailscale.com/k8s-operator/sessionrecording
+        tailscale.com/k8s-operator/tsclient                          from tailscale.com/cmd/k8s-operator+
         tailscale.com/kube/egressservices                            from tailscale.com/cmd/k8s-operator
         tailscale.com/kube/ingressservices                           from tailscale.com/cmd/k8s-operator
         tailscale.com/kube/k8s-proxy/conf                            from tailscale.com/cmd/k8s-operator
@@ -908,7 +836,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/tstime                                         from tailscale.com/cmd/k8s-operator+
         tailscale.com/tstime/mono                                    from tailscale.com/net/tstun+
         tailscale.com/tstime/rate                                    from tailscale.com/wgengine/filter
-        tailscale.com/tsweb                                          from tailscale.com/util/eventbus
+        tailscale.com/tsweb                                          from tailscale.com/util/eventbus+
         tailscale.com/tsweb/varz                                     from tailscale.com/util/usermetric+
         tailscale.com/types/appctype                                 from tailscale.com/ipn/ipnlocal+
         tailscale.com/types/bools                                    from tailscale.com/tsnet+
@@ -998,7 +926,6 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         tailscale.com/wgengine/wgcfg/nmcfg                           from tailscale.com/ipn/ipnlocal
      💣 tailscale.com/wgengine/wgint                                 from tailscale.com/wgengine+
         tailscale.com/wgengine/wglog                                 from tailscale.com/wgengine
-        tailscale.com/wif                                            from tailscale.com/feature/identityfederation
         golang.org/x/crypto/argon2                                   from tailscale.com/tka
         golang.org/x/crypto/blake2b                                  from golang.org/x/crypto/argon2+
         golang.org/x/crypto/blake2s                                  from github.com/tailscale/wireguard-go/device+
@@ -1021,14 +948,15 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         golang.org/x/net/http/httpproxy                              from tailscale.com/net/tshttpproxy
         golang.org/x/net/http2                                       from k8s.io/apimachinery/pkg/util/net+
         golang.org/x/net/http2/hpack                                 from golang.org/x/net/http2+
-        golang.org/x/net/icmp                                        from github.com/prometheus-community/pro-bing+
+        golang.org/x/net/icmp                                        from tailscale.com/net/ping
         golang.org/x/net/idna                                        from golang.org/x/net/http/httpguts+
         golang.org/x/net/internal/httpcommon                         from golang.org/x/net/http2
+        golang.org/x/net/internal/httpsfv                            from golang.org/x/net/http2
         golang.org/x/net/internal/iana                               from golang.org/x/net/icmp+
-        golang.org/x/net/internal/socket                             from golang.org/x/net/icmp+
+        golang.org/x/net/internal/socket                             from golang.org/x/net/ipv4+
         golang.org/x/net/internal/socks                              from golang.org/x/net/proxy
-        golang.org/x/net/ipv4                                        from github.com/prometheus-community/pro-bing+
+        golang.org/x/net/ipv4                                        from github.com/tailscale/wireguard-go/conn+
-        golang.org/x/net/ipv6                                        from github.com/prometheus-community/pro-bing+
+        golang.org/x/net/ipv6                                        from github.com/tailscale/wireguard-go/conn+
         golang.org/x/net/proxy                                       from tailscale.com/net/netns
    D    golang.org/x/net/route                                       from tailscale.com/net/netmon+
         golang.org/x/net/websocket                                   from tailscale.com/k8s-operator/sessionrecording/ws
@@ -1135,7 +1063,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         crypto/sha3                                                  from crypto/internal/fips140hash+
         crypto/sha512                                                from crypto/ecdsa+
         crypto/subtle                                                from crypto/cipher+
-        crypto/tls                                                   from github.com/prometheus-community/pro-bing+
+        crypto/tls                                                   from github.com/prometheus/client_golang/prometheus/promhttp+
         crypto/tls/internal/fips140tls                               from crypto/tls
         crypto/x509                                                  from crypto/tls+
    D    crypto/x509/internal/macos                                   from crypto/x509
@@ -1244,7 +1172,7 @@ tailscale.com/cmd/k8s-operator dependencies: (generated by github.com/tailscale/
         mime/quotedprintable                                         from mime/multipart
         net                                                          from crypto/tls+
         net/http                                                     from expvar+
-        net/http/httptrace                                           from github.com/prometheus-community/pro-bing+
+        net/http/httptrace                                           from github.com/prometheus/client_golang/prometheus/promhttp+
         net/http/httputil                                            from tailscale.com/client/web+
         net/http/internal                                            from net/http+
         net/http/internal/ascii                                      from net/http+

cmd/k8s-operator/deploy/chart/templates/deployment.yaml

@@ -146,3 +146,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with .Values.operatorConfig.priorityClassName }}
+      priorityClassName: {{ . }}
+      {{- end }}

cmd/k8s-operator/deploy/chart/values.yaml

@@ -72,6 +72,8 @@ operatorConfig:
 
   affinity: {}
 
+  priorityClassName: ""
+
   podSecurityContext: {}
 
   securityContext: {}

cmd/k8s-operator/deploy/crds/tailscale.com_dnsconfigs.yaml

@@ -104,6 +104,884 @@ spec:
                       description: Pod configuration.
                       type: object
                       properties:
+                        affinity:
+                          description: If specified, applies affinity rules to the pods deployed by the DNSConfig resource.
+                          type: object
+                          properties:
+                            nodeAffinity:
+                              description: Describes node affinity scheduling rules for the pod.
+                              type: object
+                              properties:
+                                preferredDuringSchedulingIgnoredDuringExecution:
+                                  description: |-
+                                    The scheduler will prefer to schedule pods to nodes that satisfy
+                                    the affinity expressions specified by this field, but it may choose
+                                    a node that violates one or more of the expressions. The node that is
+                                    most preferred is the one with the greatest sum of weights, i.e.
+                                    for each node that meets all of the scheduling requirements (resource
+                                    request, requiredDuringScheduling affinity expressions, etc.),
+                                    compute a sum by iterating through the elements of this field and adding
+                                    "weight" to the sum if the node matches the corresponding matchExpressions; the
+                                    node(s) with the highest sum are the most preferred.
+                                  type: array
+                                  items:
+                                    description: |-
+                                      An empty preferred scheduling term matches all objects with implicit weight 0
+                                      (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op).
+                                    type: object
+                                    required:
+                                      - preference
+                                      - weight
+                                    properties:
+                                      preference:
+                                        description: A node selector term, associated with the corresponding weight.
+                                        type: object
+                                        properties:
+                                          matchExpressions:
+                                            description: A list of node selector requirements by node's labels.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A node selector requirement is a selector that contains values, a key, and an operator
+                                                that relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: The label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    Represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    An array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                    array must have a single element, which will be interpreted as an integer.
+                                                    This array is replaced during a strategic merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                          matchFields:
+                                            description: A list of node selector requirements by node's fields.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A node selector requirement is a selector that contains values, a key, and an operator
+                                                that relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: The label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    Represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    An array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                    array must have a single element, which will be interpreted as an integer.
+                                                    This array is replaced during a strategic merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                        x-kubernetes-map-type: atomic
+                                      weight:
+                                        description: Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100.
+                                        type: integer
+                                        format: int32
+                                  x-kubernetes-list-type: atomic
+                                requiredDuringSchedulingIgnoredDuringExecution:
+                                  description: |-
+                                    If the affinity requirements specified by this field are not met at
+                                    scheduling time, the pod will not be scheduled onto the node.
+                                    If the affinity requirements specified by this field cease to be met
+                                    at some point during pod execution (e.g. due to an update), the system
+                                    may or may not try to eventually evict the pod from its node.
+                                  type: object
+                                  required:
+                                    - nodeSelectorTerms
+                                  properties:
+                                    nodeSelectorTerms:
+                                      description: Required. A list of node selector terms. The terms are ORed.
+                                      type: array
+                                      items:
+                                        description: |-
+                                          A null or empty node selector term matches no objects. The requirements of
+                                          them are ANDed.
+                                          The TopologySelectorTerm type implements a subset of the NodeSelectorTerm.
+                                        type: object
+                                        properties:
+                                          matchExpressions:
+                                            description: A list of node selector requirements by node's labels.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A node selector requirement is a selector that contains values, a key, and an operator
+                                                that relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: The label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    Represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    An array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                    array must have a single element, which will be interpreted as an integer.
+                                                    This array is replaced during a strategic merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                          matchFields:
+                                            description: A list of node selector requirements by node's fields.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A node selector requirement is a selector that contains values, a key, and an operator
+                                                that relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: The label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    Represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    An array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                    array must have a single element, which will be interpreted as an integer.
+                                                    This array is replaced during a strategic merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                        x-kubernetes-map-type: atomic
+                                      x-kubernetes-list-type: atomic
+                                  x-kubernetes-map-type: atomic
+                            podAffinity:
+                              description: Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)).
+                              type: object
+                              properties:
+                                preferredDuringSchedulingIgnoredDuringExecution:
+                                  description: |-
+                                    The scheduler will prefer to schedule pods to nodes that satisfy
+                                    the affinity expressions specified by this field, but it may choose
+                                    a node that violates one or more of the expressions. The node that is
+                                    most preferred is the one with the greatest sum of weights, i.e.
+                                    for each node that meets all of the scheduling requirements (resource
+                                    request, requiredDuringScheduling affinity expressions, etc.),
+                                    compute a sum by iterating through the elements of this field and adding
+                                    "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the
+                                    node(s) with the highest sum are the most preferred.
+                                  type: array
+                                  items:
+                                    description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
+                                    type: object
+                                    required:
+                                      - podAffinityTerm
+                                      - weight
+                                    properties:
+                                      podAffinityTerm:
+                                        description: Required. A pod affinity term, associated with the corresponding weight.
+                                        type: object
+                                        required:
+                                          - topologyKey
+                                        properties:
+                                          labelSelector:
+                                            description: |-
+                                              A label query over a set of resources, in this case pods.
+                                              If it's null, this PodAffinityTerm matches with no Pods.
+                                            type: object
+                                            properties:
+                                              matchExpressions:
+                                                description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                type: array
+                                                items:
+                                                  description: |-
+                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                    relates the key and values.
+                                                  type: object
+                                                  required:
+                                                    - key
+                                                    - operator
+                                                  properties:
+                                                    key:
+                                                      description: key is the label key that the selector applies to.
+                                                      type: string
+                                                    operator:
+                                                      description: |-
+                                                        operator represents a key's relationship to a set of values.
+                                                        Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                      type: string
+                                                    values:
+                                                      description: |-
+                                                        values is an array of string values. If the operator is In or NotIn,
+                                                        the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                        the values array must be empty. This array is replaced during a strategic
+                                                        merge patch.
+                                                      type: array
+                                                      items:
+                                                        type: string
+                                                      x-kubernetes-list-type: atomic
+                                                x-kubernetes-list-type: atomic
+                                              matchLabels:
+                                                description: |-
+                                                  matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                  map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                  operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                type: object
+                                                additionalProperties:
+                                                  type: string
+                                            x-kubernetes-map-type: atomic
+                                          matchLabelKeys:
+                                            description: |-
+                                              MatchLabelKeys is a set of pod label keys to select which pods will
+                                              be taken into consideration. The keys are used to lookup values from the
+                                              incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                              to select the group of existing pods which pods will be taken into consideration
+                                              for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                              pod labels will be ignored. The default value is empty.
+                                              The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                              Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                            type: array
+                                            items:
+                                              type: string
+                                            x-kubernetes-list-type: atomic
+                                          mismatchLabelKeys:
+                                            description: |-
+                                              MismatchLabelKeys is a set of pod label keys to select which pods will
+                                              be taken into consideration. The keys are used to lookup values from the
+                                              incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                              to select the group of existing pods which pods will be taken into consideration
+                                              for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                              pod labels will be ignored. The default value is empty.
+                                              The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                              Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                            type: array
+                                            items:
+                                              type: string
+                                            x-kubernetes-list-type: atomic
+                                          namespaceSelector:
+                                            description: |-
+                                              A label query over the set of namespaces that the term applies to.
+                                              The term is applied to the union of the namespaces selected by this field
+                                              and the ones listed in the namespaces field.
+                                              null selector and null or empty namespaces list means "this pod's namespace".
+                                              An empty selector ({}) matches all namespaces.
+                                            type: object
+                                            properties:
+                                              matchExpressions:
+                                                description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                type: array
+                                                items:
+                                                  description: |-
+                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                    relates the key and values.
+                                                  type: object
+                                                  required:
+                                                    - key
+                                                    - operator
+                                                  properties:
+                                                    key:
+                                                      description: key is the label key that the selector applies to.
+                                                      type: string
+                                                    operator:
+                                                      description: |-
+                                                        operator represents a key's relationship to a set of values.
+                                                        Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                      type: string
+                                                    values:
+                                                      description: |-
+                                                        values is an array of string values. If the operator is In or NotIn,
+                                                        the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                        the values array must be empty. This array is replaced during a strategic
+                                                        merge patch.
+                                                      type: array
+                                                      items:
+                                                        type: string
+                                                      x-kubernetes-list-type: atomic
+                                                x-kubernetes-list-type: atomic
+                                              matchLabels:
+                                                description: |-
+                                                  matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                  map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                  operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                type: object
+                                                additionalProperties:
+                                                  type: string
+                                            x-kubernetes-map-type: atomic
+                                          namespaces:
+                                            description: |-
+                                              namespaces specifies a static list of namespace names that the term applies to.
+                                              The term is applied to the union of the namespaces listed in this field
+                                              and the ones selected by namespaceSelector.
+                                              null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                            type: array
+                                            items:
+                                              type: string
+                                            x-kubernetes-list-type: atomic
+                                          topologyKey:
+                                            description: |-
+                                              This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                              the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                              whose value of the label with key topologyKey matches that of any node on which any of the
+                                              selected pods is running.
+                                              Empty topologyKey is not allowed.
+                                            type: string
+                                      weight:
+                                        description: |-
+                                          weight associated with matching the corresponding podAffinityTerm,
+                                          in the range 1-100.
+                                        type: integer
+                                        format: int32
+                                  x-kubernetes-list-type: atomic
+                                requiredDuringSchedulingIgnoredDuringExecution:
+                                  description: |-
+                                    If the affinity requirements specified by this field are not met at
+                                    scheduling time, the pod will not be scheduled onto the node.
+                                    If the affinity requirements specified by this field cease to be met
+                                    at some point during pod execution (e.g. due to a pod label update), the
+                                    system may or may not try to eventually evict the pod from its node.
+                                    When there are multiple elements, the lists of nodes corresponding to each
+                                    podAffinityTerm are intersected, i.e. all terms must be satisfied.
+                                  type: array
+                                  items:
+                                    description: |-
+                                      Defines a set of pods (namely those matching the labelSelector
+                                      relative to the given namespace(s)) that this pod should be
+                                      co-located (affinity) or not co-located (anti-affinity) with,
+                                      where co-located is defined as running on a node whose value of
+                                      the label with key <topologyKey> matches that of any node on which
+                                      a pod of the set of pods is running
+                                    type: object
+                                    required:
+                                      - topologyKey
+                                    properties:
+                                      labelSelector:
+                                        description: |-
+                                          A label query over a set of resources, in this case pods.
+                                          If it's null, this PodAffinityTerm matches with no Pods.
+                                        type: object
+                                        properties:
+                                          matchExpressions:
+                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A label selector requirement is a selector that contains values, a key, and an operator that
+                                                relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: key is the label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    operator represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    values is an array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. This array is replaced during a strategic
+                                                    merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                          matchLabels:
+                                            description: |-
+                                              matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                              map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                              operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                            type: object
+                                            additionalProperties:
+                                              type: string
+                                        x-kubernetes-map-type: atomic
+                                      matchLabelKeys:
+                                        description: |-
+                                          MatchLabelKeys is a set of pod label keys to select which pods will
+                                          be taken into consideration. The keys are used to lookup values from the
+                                          incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                          to select the group of existing pods which pods will be taken into consideration
+                                          for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                          pod labels will be ignored. The default value is empty.
+                                          The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                          Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                        type: array
+                                        items:
+                                          type: string
+                                        x-kubernetes-list-type: atomic
+                                      mismatchLabelKeys:
+                                        description: |-
+                                          MismatchLabelKeys is a set of pod label keys to select which pods will
+                                          be taken into consideration. The keys are used to lookup values from the
+                                          incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                          to select the group of existing pods which pods will be taken into consideration
+                                          for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                          pod labels will be ignored. The default value is empty.
+                                          The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                          Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                        type: array
+                                        items:
+                                          type: string
+                                        x-kubernetes-list-type: atomic
+                                      namespaceSelector:
+                                        description: |-
+                                          A label query over the set of namespaces that the term applies to.
+                                          The term is applied to the union of the namespaces selected by this field
+                                          and the ones listed in the namespaces field.
+                                          null selector and null or empty namespaces list means "this pod's namespace".
+                                          An empty selector ({}) matches all namespaces.
+                                        type: object
+                                        properties:
+                                          matchExpressions:
+                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A label selector requirement is a selector that contains values, a key, and an operator that
+                                                relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: key is the label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    operator represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    values is an array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. This array is replaced during a strategic
+                                                    merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                          matchLabels:
+                                            description: |-
+                                              matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                              map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                              operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                            type: object
+                                            additionalProperties:
+                                              type: string
+                                        x-kubernetes-map-type: atomic
+                                      namespaces:
+                                        description: |-
+                                          namespaces specifies a static list of namespace names that the term applies to.
+                                          The term is applied to the union of the namespaces listed in this field
+                                          and the ones selected by namespaceSelector.
+                                          null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                        type: array
+                                        items:
+                                          type: string
+                                        x-kubernetes-list-type: atomic
+                                      topologyKey:
+                                        description: |-
+                                          This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                          the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                          whose value of the label with key topologyKey matches that of any node on which any of the
+                                          selected pods is running.
+                                          Empty topologyKey is not allowed.
+                                        type: string
+                                  x-kubernetes-list-type: atomic
+                            podAntiAffinity:
+                              description: Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)).
+                              type: object
+                              properties:
+                                preferredDuringSchedulingIgnoredDuringExecution:
+                                  description: |-
+                                    The scheduler will prefer to schedule pods to nodes that satisfy
+                                    the anti-affinity expressions specified by this field, but it may choose
+                                    a node that violates one or more of the expressions. The node that is
+                                    most preferred is the one with the greatest sum of weights, i.e.
+                                    for each node that meets all of the scheduling requirements (resource
+                                    request, requiredDuringScheduling anti-affinity expressions, etc.),
+                                    compute a sum by iterating through the elements of this field and subtracting
+                                    "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the
+                                    node(s) with the highest sum are the most preferred.
+                                  type: array
+                                  items:
+                                    description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
+                                    type: object
+                                    required:
+                                      - podAffinityTerm
+                                      - weight
+                                    properties:
+                                      podAffinityTerm:
+                                        description: Required. A pod affinity term, associated with the corresponding weight.
+                                        type: object
+                                        required:
+                                          - topologyKey
+                                        properties:
+                                          labelSelector:
+                                            description: |-
+                                              A label query over a set of resources, in this case pods.
+                                              If it's null, this PodAffinityTerm matches with no Pods.
+                                            type: object
+                                            properties:
+                                              matchExpressions:
+                                                description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                type: array
+                                                items:
+                                                  description: |-
+                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                    relates the key and values.
+                                                  type: object
+                                                  required:
+                                                    - key
+                                                    - operator
+                                                  properties:
+                                                    key:
+                                                      description: key is the label key that the selector applies to.
+                                                      type: string
+                                                    operator:
+                                                      description: |-
+                                                        operator represents a key's relationship to a set of values.
+                                                        Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                      type: string
+                                                    values:
+                                                      description: |-
+                                                        values is an array of string values. If the operator is In or NotIn,
+                                                        the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                        the values array must be empty. This array is replaced during a strategic
+                                                        merge patch.
+                                                      type: array
+                                                      items:
+                                                        type: string
+                                                      x-kubernetes-list-type: atomic
+                                                x-kubernetes-list-type: atomic
+                                              matchLabels:
+                                                description: |-
+                                                  matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                  map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                  operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                type: object
+                                                additionalProperties:
+                                                  type: string
+                                            x-kubernetes-map-type: atomic
+                                          matchLabelKeys:
+                                            description: |-
+                                              MatchLabelKeys is a set of pod label keys to select which pods will
+                                              be taken into consideration. The keys are used to lookup values from the
+                                              incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                              to select the group of existing pods which pods will be taken into consideration
+                                              for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                              pod labels will be ignored. The default value is empty.
+                                              The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                              Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                            type: array
+                                            items:
+                                              type: string
+                                            x-kubernetes-list-type: atomic
+                                          mismatchLabelKeys:
+                                            description: |-
+                                              MismatchLabelKeys is a set of pod label keys to select which pods will
+                                              be taken into consideration. The keys are used to lookup values from the
+                                              incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                              to select the group of existing pods which pods will be taken into consideration
+                                              for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                              pod labels will be ignored. The default value is empty.
+                                              The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                              Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                            type: array
+                                            items:
+                                              type: string
+                                            x-kubernetes-list-type: atomic
+                                          namespaceSelector:
+                                            description: |-
+                                              A label query over the set of namespaces that the term applies to.
+                                              The term is applied to the union of the namespaces selected by this field
+                                              and the ones listed in the namespaces field.
+                                              null selector and null or empty namespaces list means "this pod's namespace".
+                                              An empty selector ({}) matches all namespaces.
+                                            type: object
+                                            properties:
+                                              matchExpressions:
+                                                description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                type: array
+                                                items:
+                                                  description: |-
+                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                    relates the key and values.
+                                                  type: object
+                                                  required:
+                                                    - key
+                                                    - operator
+                                                  properties:
+                                                    key:
+                                                      description: key is the label key that the selector applies to.
+                                                      type: string
+                                                    operator:
+                                                      description: |-
+                                                        operator represents a key's relationship to a set of values.
+                                                        Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                      type: string
+                                                    values:
+                                                      description: |-
+                                                        values is an array of string values. If the operator is In or NotIn,
+                                                        the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                        the values array must be empty. This array is replaced during a strategic
+                                                        merge patch.
+                                                      type: array
+                                                      items:
+                                                        type: string
+                                                      x-kubernetes-list-type: atomic
+                                                x-kubernetes-list-type: atomic
+                                              matchLabels:
+                                                description: |-
+                                                  matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                  map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                  operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                type: object
+                                                additionalProperties:
+                                                  type: string
+                                            x-kubernetes-map-type: atomic
+                                          namespaces:
+                                            description: |-
+                                              namespaces specifies a static list of namespace names that the term applies to.
+                                              The term is applied to the union of the namespaces listed in this field
+                                              and the ones selected by namespaceSelector.
+                                              null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                            type: array
+                                            items:
+                                              type: string
+                                            x-kubernetes-list-type: atomic
+                                          topologyKey:
+                                            description: |-
+                                              This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                              the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                              whose value of the label with key topologyKey matches that of any node on which any of the
+                                              selected pods is running.
+                                              Empty topologyKey is not allowed.
+                                            type: string
+                                      weight:
+                                        description: |-
+                                          weight associated with matching the corresponding podAffinityTerm,
+                                          in the range 1-100.
+                                        type: integer
+                                        format: int32
+                                  x-kubernetes-list-type: atomic
+                                requiredDuringSchedulingIgnoredDuringExecution:
+                                  description: |-
+                                    If the anti-affinity requirements specified by this field are not met at
+                                    scheduling time, the pod will not be scheduled onto the node.
+                                    If the anti-affinity requirements specified by this field cease to be met
+                                    at some point during pod execution (e.g. due to a pod label update), the
+                                    system may or may not try to eventually evict the pod from its node.
+                                    When there are multiple elements, the lists of nodes corresponding to each
+                                    podAffinityTerm are intersected, i.e. all terms must be satisfied.
+                                  type: array
+                                  items:
+                                    description: |-
+                                      Defines a set of pods (namely those matching the labelSelector
+                                      relative to the given namespace(s)) that this pod should be
+                                      co-located (affinity) or not co-located (anti-affinity) with,
+                                      where co-located is defined as running on a node whose value of
+                                      the label with key <topologyKey> matches that of any node on which
+                                      a pod of the set of pods is running
+                                    type: object
+                                    required:
+                                      - topologyKey
+                                    properties:
+                                      labelSelector:
+                                        description: |-
+                                          A label query over a set of resources, in this case pods.
+                                          If it's null, this PodAffinityTerm matches with no Pods.
+                                        type: object
+                                        properties:
+                                          matchExpressions:
+                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A label selector requirement is a selector that contains values, a key, and an operator that
+                                                relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: key is the label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    operator represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    values is an array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. This array is replaced during a strategic
+                                                    merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                          matchLabels:
+                                            description: |-
+                                              matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                              map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                              operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                            type: object
+                                            additionalProperties:
+                                              type: string
+                                        x-kubernetes-map-type: atomic
+                                      matchLabelKeys:
+                                        description: |-
+                                          MatchLabelKeys is a set of pod label keys to select which pods will
+                                          be taken into consideration. The keys are used to lookup values from the
+                                          incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                          to select the group of existing pods which pods will be taken into consideration
+                                          for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                          pod labels will be ignored. The default value is empty.
+                                          The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                          Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                        type: array
+                                        items:
+                                          type: string
+                                        x-kubernetes-list-type: atomic
+                                      mismatchLabelKeys:
+                                        description: |-
+                                          MismatchLabelKeys is a set of pod label keys to select which pods will
+                                          be taken into consideration. The keys are used to lookup values from the
+                                          incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                          to select the group of existing pods which pods will be taken into consideration
+                                          for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                          pod labels will be ignored. The default value is empty.
+                                          The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                          Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                        type: array
+                                        items:
+                                          type: string
+                                        x-kubernetes-list-type: atomic
+                                      namespaceSelector:
+                                        description: |-
+                                          A label query over the set of namespaces that the term applies to.
+                                          The term is applied to the union of the namespaces selected by this field
+                                          and the ones listed in the namespaces field.
+                                          null selector and null or empty namespaces list means "this pod's namespace".
+                                          An empty selector ({}) matches all namespaces.
+                                        type: object
+                                        properties:
+                                          matchExpressions:
+                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                            type: array
+                                            items:
+                                              description: |-
+                                                A label selector requirement is a selector that contains values, a key, and an operator that
+                                                relates the key and values.
+                                              type: object
+                                              required:
+                                                - key
+                                                - operator
+                                              properties:
+                                                key:
+                                                  description: key is the label key that the selector applies to.
+                                                  type: string
+                                                operator:
+                                                  description: |-
+                                                    operator represents a key's relationship to a set of values.
+                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                  type: string
+                                                values:
+                                                  description: |-
+                                                    values is an array of string values. If the operator is In or NotIn,
+                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                    the values array must be empty. This array is replaced during a strategic
+                                                    merge patch.
+                                                  type: array
+                                                  items:
+                                                    type: string
+                                                  x-kubernetes-list-type: atomic
+                                            x-kubernetes-list-type: atomic
+                                          matchLabels:
+                                            description: |-
+                                              matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                              map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                              operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                            type: object
+                                            additionalProperties:
+                                              type: string
+                                        x-kubernetes-map-type: atomic
+                                      namespaces:
+                                        description: |-
+                                          namespaces specifies a static list of namespace names that the term applies to.
+                                          The term is applied to the union of the namespaces listed in this field
+                                          and the ones selected by namespaceSelector.
+                                          null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                        type: array
+                                        items:
+                                          type: string
+                                        x-kubernetes-list-type: atomic
+                                      topologyKey:
+                                        description: |-
+                                          This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                          the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                          whose value of the label with key topologyKey matches that of any node on which any of the
+                                          selected pods is running.
+                                          Empty topologyKey is not allowed.
+                                        type: string
+                                  x-kubernetes-list-type: atomic
+                        nodeSelector:
+                          description: If specified, applies node selector rules to the pods deployed by the DNSConfig resource.
+                          type: object
+                          additionalProperties:
+                            type: string
                         tolerations:
                           description: If specified, applies tolerations to the pods deployed by the DNSConfig resource.
                           type: array

cmd/k8s-operator/deploy/manifests/operator.yaml

@@ -442,6 +442,884 @@ spec:
                                     pod:
                                         description: Pod configuration.
                                         properties:
+                                            affinity:
+                                                description: If specified, applies affinity rules to the pods deployed by the DNSConfig resource.
+                                                properties:
+                                                    nodeAffinity:
+                                                        description: Describes node affinity scheduling rules for the pod.
+                                                        properties:
+                                                            preferredDuringSchedulingIgnoredDuringExecution:
+                                                                description: |-
+                                                                    The scheduler will prefer to schedule pods to nodes that satisfy
+                                                                    the affinity expressions specified by this field, but it may choose
+                                                                    a node that violates one or more of the expressions. The node that is
+                                                                    most preferred is the one with the greatest sum of weights, i.e.
+                                                                    for each node that meets all of the scheduling requirements (resource
+                                                                    request, requiredDuringScheduling affinity expressions, etc.),
+                                                                    compute a sum by iterating through the elements of this field and adding
+                                                                    "weight" to the sum if the node matches the corresponding matchExpressions; the
+                                                                    node(s) with the highest sum are the most preferred.
+                                                                items:
+                                                                    description: |-
+                                                                        An empty preferred scheduling term matches all objects with implicit weight 0
+                                                                        (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op).
+                                                                    properties:
+                                                                        preference:
+                                                                            description: A node selector term, associated with the corresponding weight.
+                                                                            properties:
+                                                                                matchExpressions:
+                                                                                    description: A list of node selector requirements by node's labels.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A node selector requirement is a selector that contains values, a key, and an operator
+                                                                                            that relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: The label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    Represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    An array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                                                                    array must have a single element, which will be interpreted as an integer.
+                                                                                                    This array is replaced during a strategic merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                matchFields:
+                                                                                    description: A list of node selector requirements by node's fields.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A node selector requirement is a selector that contains values, a key, and an operator
+                                                                                            that relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: The label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    Represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    An array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                                                                    array must have a single element, which will be interpreted as an integer.
+                                                                                                    This array is replaced during a strategic merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                            type: object
+                                                                            x-kubernetes-map-type: atomic
+                                                                        weight:
+                                                                            description: Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100.
+                                                                            format: int32
+                                                                            type: integer
+                                                                    required:
+                                                                        - preference
+                                                                        - weight
+                                                                    type: object
+                                                                type: array
+                                                                x-kubernetes-list-type: atomic
+                                                            requiredDuringSchedulingIgnoredDuringExecution:
+                                                                description: |-
+                                                                    If the affinity requirements specified by this field are not met at
+                                                                    scheduling time, the pod will not be scheduled onto the node.
+                                                                    If the affinity requirements specified by this field cease to be met
+                                                                    at some point during pod execution (e.g. due to an update), the system
+                                                                    may or may not try to eventually evict the pod from its node.
+                                                                properties:
+                                                                    nodeSelectorTerms:
+                                                                        description: Required. A list of node selector terms. The terms are ORed.
+                                                                        items:
+                                                                            description: |-
+                                                                                A null or empty node selector term matches no objects. The requirements of
+                                                                                them are ANDed.
+                                                                                The TopologySelectorTerm type implements a subset of the NodeSelectorTerm.
+                                                                            properties:
+                                                                                matchExpressions:
+                                                                                    description: A list of node selector requirements by node's labels.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A node selector requirement is a selector that contains values, a key, and an operator
+                                                                                            that relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: The label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    Represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    An array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                                                                    array must have a single element, which will be interpreted as an integer.
+                                                                                                    This array is replaced during a strategic merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                matchFields:
+                                                                                    description: A list of node selector requirements by node's fields.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A node selector requirement is a selector that contains values, a key, and an operator
+                                                                                            that relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: The label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    Represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    An array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. If the operator is Gt or Lt, the values
+                                                                                                    array must have a single element, which will be interpreted as an integer.
+                                                                                                    This array is replaced during a strategic merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                            type: object
+                                                                            x-kubernetes-map-type: atomic
+                                                                        type: array
+                                                                        x-kubernetes-list-type: atomic
+                                                                required:
+                                                                    - nodeSelectorTerms
+                                                                type: object
+                                                                x-kubernetes-map-type: atomic
+                                                        type: object
+                                                    podAffinity:
+                                                        description: Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)).
+                                                        properties:
+                                                            preferredDuringSchedulingIgnoredDuringExecution:
+                                                                description: |-
+                                                                    The scheduler will prefer to schedule pods to nodes that satisfy
+                                                                    the affinity expressions specified by this field, but it may choose
+                                                                    a node that violates one or more of the expressions. The node that is
+                                                                    most preferred is the one with the greatest sum of weights, i.e.
+                                                                    for each node that meets all of the scheduling requirements (resource
+                                                                    request, requiredDuringScheduling affinity expressions, etc.),
+                                                                    compute a sum by iterating through the elements of this field and adding
+                                                                    "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the
+                                                                    node(s) with the highest sum are the most preferred.
+                                                                items:
+                                                                    description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
+                                                                    properties:
+                                                                        podAffinityTerm:
+                                                                            description: Required. A pod affinity term, associated with the corresponding weight.
+                                                                            properties:
+                                                                                labelSelector:
+                                                                                    description: |-
+                                                                                        A label query over a set of resources, in this case pods.
+                                                                                        If it's null, this PodAffinityTerm matches with no Pods.
+                                                                                    properties:
+                                                                                        matchExpressions:
+                                                                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                            items:
+                                                                                                description: |-
+                                                                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                                    relates the key and values.
+                                                                                                properties:
+                                                                                                    key:
+                                                                                                        description: key is the label key that the selector applies to.
+                                                                                                        type: string
+                                                                                                    operator:
+                                                                                                        description: |-
+                                                                                                            operator represents a key's relationship to a set of values.
+                                                                                                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                        type: string
+                                                                                                    values:
+                                                                                                        description: |-
+                                                                                                            values is an array of string values. If the operator is In or NotIn,
+                                                                                                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                            the values array must be empty. This array is replaced during a strategic
+                                                                                                            merge patch.
+                                                                                                        items:
+                                                                                                            type: string
+                                                                                                        type: array
+                                                                                                        x-kubernetes-list-type: atomic
+                                                                                                required:
+                                                                                                    - key
+                                                                                                    - operator
+                                                                                                type: object
+                                                                                            type: array
+                                                                                            x-kubernetes-list-type: atomic
+                                                                                        matchLabels:
+                                                                                            additionalProperties:
+                                                                                                type: string
+                                                                                            description: |-
+                                                                                                matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                                map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                                operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                            type: object
+                                                                                    type: object
+                                                                                    x-kubernetes-map-type: atomic
+                                                                                matchLabelKeys:
+                                                                                    description: |-
+                                                                                        MatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                        be taken into consideration. The keys are used to lookup values from the
+                                                                                        incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                                                                        to select the group of existing pods which pods will be taken into consideration
+                                                                                        for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                        pod labels will be ignored. The default value is empty.
+                                                                                        The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                                                                        Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                                                                    items:
+                                                                                        type: string
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                mismatchLabelKeys:
+                                                                                    description: |-
+                                                                                        MismatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                        be taken into consideration. The keys are used to lookup values from the
+                                                                                        incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                                                                        to select the group of existing pods which pods will be taken into consideration
+                                                                                        for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                        pod labels will be ignored. The default value is empty.
+                                                                                        The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                                                                        Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                                                                    items:
+                                                                                        type: string
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                namespaceSelector:
+                                                                                    description: |-
+                                                                                        A label query over the set of namespaces that the term applies to.
+                                                                                        The term is applied to the union of the namespaces selected by this field
+                                                                                        and the ones listed in the namespaces field.
+                                                                                        null selector and null or empty namespaces list means "this pod's namespace".
+                                                                                        An empty selector ({}) matches all namespaces.
+                                                                                    properties:
+                                                                                        matchExpressions:
+                                                                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                            items:
+                                                                                                description: |-
+                                                                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                                    relates the key and values.
+                                                                                                properties:
+                                                                                                    key:
+                                                                                                        description: key is the label key that the selector applies to.
+                                                                                                        type: string
+                                                                                                    operator:
+                                                                                                        description: |-
+                                                                                                            operator represents a key's relationship to a set of values.
+                                                                                                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                        type: string
+                                                                                                    values:
+                                                                                                        description: |-
+                                                                                                            values is an array of string values. If the operator is In or NotIn,
+                                                                                                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                            the values array must be empty. This array is replaced during a strategic
+                                                                                                            merge patch.
+                                                                                                        items:
+                                                                                                            type: string
+                                                                                                        type: array
+                                                                                                        x-kubernetes-list-type: atomic
+                                                                                                required:
+                                                                                                    - key
+                                                                                                    - operator
+                                                                                                type: object
+                                                                                            type: array
+                                                                                            x-kubernetes-list-type: atomic
+                                                                                        matchLabels:
+                                                                                            additionalProperties:
+                                                                                                type: string
+                                                                                            description: |-
+                                                                                                matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                                map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                                operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                            type: object
+                                                                                    type: object
+                                                                                    x-kubernetes-map-type: atomic
+                                                                                namespaces:
+                                                                                    description: |-
+                                                                                        namespaces specifies a static list of namespace names that the term applies to.
+                                                                                        The term is applied to the union of the namespaces listed in this field
+                                                                                        and the ones selected by namespaceSelector.
+                                                                                        null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                                                                    items:
+                                                                                        type: string
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                topologyKey:
+                                                                                    description: |-
+                                                                                        This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                                                                        the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                                                                        whose value of the label with key topologyKey matches that of any node on which any of the
+                                                                                        selected pods is running.
+                                                                                        Empty topologyKey is not allowed.
+                                                                                    type: string
+                                                                            required:
+                                                                                - topologyKey
+                                                                            type: object
+                                                                        weight:
+                                                                            description: |-
+                                                                                weight associated with matching the corresponding podAffinityTerm,
+                                                                                in the range 1-100.
+                                                                            format: int32
+                                                                            type: integer
+                                                                    required:
+                                                                        - podAffinityTerm
+                                                                        - weight
+                                                                    type: object
+                                                                type: array
+                                                                x-kubernetes-list-type: atomic
+                                                            requiredDuringSchedulingIgnoredDuringExecution:
+                                                                description: |-
+                                                                    If the affinity requirements specified by this field are not met at
+                                                                    scheduling time, the pod will not be scheduled onto the node.
+                                                                    If the affinity requirements specified by this field cease to be met
+                                                                    at some point during pod execution (e.g. due to a pod label update), the
+                                                                    system may or may not try to eventually evict the pod from its node.
+                                                                    When there are multiple elements, the lists of nodes corresponding to each
+                                                                    podAffinityTerm are intersected, i.e. all terms must be satisfied.
+                                                                items:
+                                                                    description: |-
+                                                                        Defines a set of pods (namely those matching the labelSelector
+                                                                        relative to the given namespace(s)) that this pod should be
+                                                                        co-located (affinity) or not co-located (anti-affinity) with,
+                                                                        where co-located is defined as running on a node whose value of
+                                                                        the label with key <topologyKey> matches that of any node on which
+                                                                        a pod of the set of pods is running
+                                                                    properties:
+                                                                        labelSelector:
+                                                                            description: |-
+                                                                                A label query over a set of resources, in this case pods.
+                                                                                If it's null, this PodAffinityTerm matches with no Pods.
+                                                                            properties:
+                                                                                matchExpressions:
+                                                                                    description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                            relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: key is the label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    operator represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    values is an array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. This array is replaced during a strategic
+                                                                                                    merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                matchLabels:
+                                                                                    additionalProperties:
+                                                                                        type: string
+                                                                                    description: |-
+                                                                                        matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                        map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                        operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                    type: object
+                                                                            type: object
+                                                                            x-kubernetes-map-type: atomic
+                                                                        matchLabelKeys:
+                                                                            description: |-
+                                                                                MatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                be taken into consideration. The keys are used to lookup values from the
+                                                                                incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                                                                to select the group of existing pods which pods will be taken into consideration
+                                                                                for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                pod labels will be ignored. The default value is empty.
+                                                                                The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                                                                Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                                                            items:
+                                                                                type: string
+                                                                            type: array
+                                                                            x-kubernetes-list-type: atomic
+                                                                        mismatchLabelKeys:
+                                                                            description: |-
+                                                                                MismatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                be taken into consideration. The keys are used to lookup values from the
+                                                                                incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                                                                to select the group of existing pods which pods will be taken into consideration
+                                                                                for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                pod labels will be ignored. The default value is empty.
+                                                                                The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                                                                Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                                                            items:
+                                                                                type: string
+                                                                            type: array
+                                                                            x-kubernetes-list-type: atomic
+                                                                        namespaceSelector:
+                                                                            description: |-
+                                                                                A label query over the set of namespaces that the term applies to.
+                                                                                The term is applied to the union of the namespaces selected by this field
+                                                                                and the ones listed in the namespaces field.
+                                                                                null selector and null or empty namespaces list means "this pod's namespace".
+                                                                                An empty selector ({}) matches all namespaces.
+                                                                            properties:
+                                                                                matchExpressions:
+                                                                                    description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                            relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: key is the label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    operator represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    values is an array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. This array is replaced during a strategic
+                                                                                                    merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                matchLabels:
+                                                                                    additionalProperties:
+                                                                                        type: string
+                                                                                    description: |-
+                                                                                        matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                        map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                        operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                    type: object
+                                                                            type: object
+                                                                            x-kubernetes-map-type: atomic
+                                                                        namespaces:
+                                                                            description: |-
+                                                                                namespaces specifies a static list of namespace names that the term applies to.
+                                                                                The term is applied to the union of the namespaces listed in this field
+                                                                                and the ones selected by namespaceSelector.
+                                                                                null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                                                            items:
+                                                                                type: string
+                                                                            type: array
+                                                                            x-kubernetes-list-type: atomic
+                                                                        topologyKey:
+                                                                            description: |-
+                                                                                This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                                                                the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                                                                whose value of the label with key topologyKey matches that of any node on which any of the
+                                                                                selected pods is running.
+                                                                                Empty topologyKey is not allowed.
+                                                                            type: string
+                                                                    required:
+                                                                        - topologyKey
+                                                                    type: object
+                                                                type: array
+                                                                x-kubernetes-list-type: atomic
+                                                        type: object
+                                                    podAntiAffinity:
+                                                        description: Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)).
+                                                        properties:
+                                                            preferredDuringSchedulingIgnoredDuringExecution:
+                                                                description: |-
+                                                                    The scheduler will prefer to schedule pods to nodes that satisfy
+                                                                    the anti-affinity expressions specified by this field, but it may choose
+                                                                    a node that violates one or more of the expressions. The node that is
+                                                                    most preferred is the one with the greatest sum of weights, i.e.
+                                                                    for each node that meets all of the scheduling requirements (resource
+                                                                    request, requiredDuringScheduling anti-affinity expressions, etc.),
+                                                                    compute a sum by iterating through the elements of this field and subtracting
+                                                                    "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the
+                                                                    node(s) with the highest sum are the most preferred.
+                                                                items:
+                                                                    description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
+                                                                    properties:
+                                                                        podAffinityTerm:
+                                                                            description: Required. A pod affinity term, associated with the corresponding weight.
+                                                                            properties:
+                                                                                labelSelector:
+                                                                                    description: |-
+                                                                                        A label query over a set of resources, in this case pods.
+                                                                                        If it's null, this PodAffinityTerm matches with no Pods.
+                                                                                    properties:
+                                                                                        matchExpressions:
+                                                                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                            items:
+                                                                                                description: |-
+                                                                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                                    relates the key and values.
+                                                                                                properties:
+                                                                                                    key:
+                                                                                                        description: key is the label key that the selector applies to.
+                                                                                                        type: string
+                                                                                                    operator:
+                                                                                                        description: |-
+                                                                                                            operator represents a key's relationship to a set of values.
+                                                                                                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                        type: string
+                                                                                                    values:
+                                                                                                        description: |-
+                                                                                                            values is an array of string values. If the operator is In or NotIn,
+                                                                                                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                            the values array must be empty. This array is replaced during a strategic
+                                                                                                            merge patch.
+                                                                                                        items:
+                                                                                                            type: string
+                                                                                                        type: array
+                                                                                                        x-kubernetes-list-type: atomic
+                                                                                                required:
+                                                                                                    - key
+                                                                                                    - operator
+                                                                                                type: object
+                                                                                            type: array
+                                                                                            x-kubernetes-list-type: atomic
+                                                                                        matchLabels:
+                                                                                            additionalProperties:
+                                                                                                type: string
+                                                                                            description: |-
+                                                                                                matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                                map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                                operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                            type: object
+                                                                                    type: object
+                                                                                    x-kubernetes-map-type: atomic
+                                                                                matchLabelKeys:
+                                                                                    description: |-
+                                                                                        MatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                        be taken into consideration. The keys are used to lookup values from the
+                                                                                        incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                                                                        to select the group of existing pods which pods will be taken into consideration
+                                                                                        for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                        pod labels will be ignored. The default value is empty.
+                                                                                        The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                                                                        Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                                                                    items:
+                                                                                        type: string
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                mismatchLabelKeys:
+                                                                                    description: |-
+                                                                                        MismatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                        be taken into consideration. The keys are used to lookup values from the
+                                                                                        incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                                                                        to select the group of existing pods which pods will be taken into consideration
+                                                                                        for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                        pod labels will be ignored. The default value is empty.
+                                                                                        The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                                                                        Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                                                                    items:
+                                                                                        type: string
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                namespaceSelector:
+                                                                                    description: |-
+                                                                                        A label query over the set of namespaces that the term applies to.
+                                                                                        The term is applied to the union of the namespaces selected by this field
+                                                                                        and the ones listed in the namespaces field.
+                                                                                        null selector and null or empty namespaces list means "this pod's namespace".
+                                                                                        An empty selector ({}) matches all namespaces.
+                                                                                    properties:
+                                                                                        matchExpressions:
+                                                                                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                            items:
+                                                                                                description: |-
+                                                                                                    A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                                    relates the key and values.
+                                                                                                properties:
+                                                                                                    key:
+                                                                                                        description: key is the label key that the selector applies to.
+                                                                                                        type: string
+                                                                                                    operator:
+                                                                                                        description: |-
+                                                                                                            operator represents a key's relationship to a set of values.
+                                                                                                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                        type: string
+                                                                                                    values:
+                                                                                                        description: |-
+                                                                                                            values is an array of string values. If the operator is In or NotIn,
+                                                                                                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                            the values array must be empty. This array is replaced during a strategic
+                                                                                                            merge patch.
+                                                                                                        items:
+                                                                                                            type: string
+                                                                                                        type: array
+                                                                                                        x-kubernetes-list-type: atomic
+                                                                                                required:
+                                                                                                    - key
+                                                                                                    - operator
+                                                                                                type: object
+                                                                                            type: array
+                                                                                            x-kubernetes-list-type: atomic
+                                                                                        matchLabels:
+                                                                                            additionalProperties:
+                                                                                                type: string
+                                                                                            description: |-
+                                                                                                matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                                map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                                operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                            type: object
+                                                                                    type: object
+                                                                                    x-kubernetes-map-type: atomic
+                                                                                namespaces:
+                                                                                    description: |-
+                                                                                        namespaces specifies a static list of namespace names that the term applies to.
+                                                                                        The term is applied to the union of the namespaces listed in this field
+                                                                                        and the ones selected by namespaceSelector.
+                                                                                        null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                                                                    items:
+                                                                                        type: string
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                topologyKey:
+                                                                                    description: |-
+                                                                                        This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                                                                        the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                                                                        whose value of the label with key topologyKey matches that of any node on which any of the
+                                                                                        selected pods is running.
+                                                                                        Empty topologyKey is not allowed.
+                                                                                    type: string
+                                                                            required:
+                                                                                - topologyKey
+                                                                            type: object
+                                                                        weight:
+                                                                            description: |-
+                                                                                weight associated with matching the corresponding podAffinityTerm,
+                                                                                in the range 1-100.
+                                                                            format: int32
+                                                                            type: integer
+                                                                    required:
+                                                                        - podAffinityTerm
+                                                                        - weight
+                                                                    type: object
+                                                                type: array
+                                                                x-kubernetes-list-type: atomic
+                                                            requiredDuringSchedulingIgnoredDuringExecution:
+                                                                description: |-
+                                                                    If the anti-affinity requirements specified by this field are not met at
+                                                                    scheduling time, the pod will not be scheduled onto the node.
+                                                                    If the anti-affinity requirements specified by this field cease to be met
+                                                                    at some point during pod execution (e.g. due to a pod label update), the
+                                                                    system may or may not try to eventually evict the pod from its node.
+                                                                    When there are multiple elements, the lists of nodes corresponding to each
+                                                                    podAffinityTerm are intersected, i.e. all terms must be satisfied.
+                                                                items:
+                                                                    description: |-
+                                                                        Defines a set of pods (namely those matching the labelSelector
+                                                                        relative to the given namespace(s)) that this pod should be
+                                                                        co-located (affinity) or not co-located (anti-affinity) with,
+                                                                        where co-located is defined as running on a node whose value of
+                                                                        the label with key <topologyKey> matches that of any node on which
+                                                                        a pod of the set of pods is running
+                                                                    properties:
+                                                                        labelSelector:
+                                                                            description: |-
+                                                                                A label query over a set of resources, in this case pods.
+                                                                                If it's null, this PodAffinityTerm matches with no Pods.
+                                                                            properties:
+                                                                                matchExpressions:
+                                                                                    description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                            relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: key is the label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    operator represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    values is an array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. This array is replaced during a strategic
+                                                                                                    merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                matchLabels:
+                                                                                    additionalProperties:
+                                                                                        type: string
+                                                                                    description: |-
+                                                                                        matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                        map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                        operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                    type: object
+                                                                            type: object
+                                                                            x-kubernetes-map-type: atomic
+                                                                        matchLabelKeys:
+                                                                            description: |-
+                                                                                MatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                be taken into consideration. The keys are used to lookup values from the
+                                                                                incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)`
+                                                                                to select the group of existing pods which pods will be taken into consideration
+                                                                                for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                pod labels will be ignored. The default value is empty.
+                                                                                The same key is forbidden to exist in both matchLabelKeys and labelSelector.
+                                                                                Also, matchLabelKeys cannot be set when labelSelector isn't set.
+                                                                            items:
+                                                                                type: string
+                                                                            type: array
+                                                                            x-kubernetes-list-type: atomic
+                                                                        mismatchLabelKeys:
+                                                                            description: |-
+                                                                                MismatchLabelKeys is a set of pod label keys to select which pods will
+                                                                                be taken into consideration. The keys are used to lookup values from the
+                                                                                incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)`
+                                                                                to select the group of existing pods which pods will be taken into consideration
+                                                                                for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+                                                                                pod labels will be ignored. The default value is empty.
+                                                                                The same key is forbidden to exist in both mismatchLabelKeys and labelSelector.
+                                                                                Also, mismatchLabelKeys cannot be set when labelSelector isn't set.
+                                                                            items:
+                                                                                type: string
+                                                                            type: array
+                                                                            x-kubernetes-list-type: atomic
+                                                                        namespaceSelector:
+                                                                            description: |-
+                                                                                A label query over the set of namespaces that the term applies to.
+                                                                                The term is applied to the union of the namespaces selected by this field
+                                                                                and the ones listed in the namespaces field.
+                                                                                null selector and null or empty namespaces list means "this pod's namespace".
+                                                                                An empty selector ({}) matches all namespaces.
+                                                                            properties:
+                                                                                matchExpressions:
+                                                                                    description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                                                                    items:
+                                                                                        description: |-
+                                                                                            A label selector requirement is a selector that contains values, a key, and an operator that
+                                                                                            relates the key and values.
+                                                                                        properties:
+                                                                                            key:
+                                                                                                description: key is the label key that the selector applies to.
+                                                                                                type: string
+                                                                                            operator:
+                                                                                                description: |-
+                                                                                                    operator represents a key's relationship to a set of values.
+                                                                                                    Valid operators are In, NotIn, Exists and DoesNotExist.
+                                                                                                type: string
+                                                                                            values:
+                                                                                                description: |-
+                                                                                                    values is an array of string values. If the operator is In or NotIn,
+                                                                                                    the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                                                                                    the values array must be empty. This array is replaced during a strategic
+                                                                                                    merge patch.
+                                                                                                items:
+                                                                                                    type: string
+                                                                                                type: array
+                                                                                                x-kubernetes-list-type: atomic
+                                                                                        required:
+                                                                                            - key
+                                                                                            - operator
+                                                                                        type: object
+                                                                                    type: array
+                                                                                    x-kubernetes-list-type: atomic
+                                                                                matchLabels:
+                                                                                    additionalProperties:
+                                                                                        type: string
+                                                                                    description: |-
+                                                                                        matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                                                                        map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                                                                        operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                                                                    type: object
+                                                                            type: object
+                                                                            x-kubernetes-map-type: atomic
+                                                                        namespaces:
+                                                                            description: |-
+                                                                                namespaces specifies a static list of namespace names that the term applies to.
+                                                                                The term is applied to the union of the namespaces listed in this field
+                                                                                and the ones selected by namespaceSelector.
+                                                                                null or empty namespaces list and null namespaceSelector means "this pod's namespace".
+                                                                            items:
+                                                                                type: string
+                                                                            type: array
+                                                                            x-kubernetes-list-type: atomic
+                                                                        topologyKey:
+                                                                            description: |-
+                                                                                This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching
+                                                                                the labelSelector in the specified namespaces, where co-located is defined as running on a node
+                                                                                whose value of the label with key topologyKey matches that of any node on which any of the
+                                                                                selected pods is running.
+                                                                                Empty topologyKey is not allowed.
+                                                                            type: string
+                                                                    required:
+                                                                        - topologyKey
+                                                                    type: object
+                                                                type: array
+                                                                x-kubernetes-list-type: atomic
+                                                        type: object
+                                                type: object
+                                            nodeSelector:
+                                                additionalProperties:
+                                                    type: string
+                                                description: If specified, applies node selector rules to the pods deployed by the DNSConfig resource.
+                                                type: object
                                             tolerations:
                                                 description: If specified, applies tolerations to the pods deployed by the DNSConfig resource.
                                                 items:

cmd/k8s-operator/e2e/ingress_test.go

@@ -17,9 +17,11 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
+	"tailscale.com/client/tailscale/v2"
 	kube "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
 	"tailscale.com/kube/kubetypes"
+	"tailscale.com/tsnet"
 	"tailscale.com/tstest"
 	"tailscale.com/util/httpm"
 )
@@ -31,12 +33,12 @@ func TestL3Ingress(t *testing.T) {
 	}
 
 	// Apply nginx
-	createAndCleanup(t, kubeClient, nginxDeployment(ns, "nginx"))
+	nginx := nginxDeployment(ns)
+	createAndCleanup(t, kubeClient, nginx)
 	// Apply service to expose it as ingress
-	name := generateName("test-ingress")
 	svc := &corev1.Service{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      name,
+			Name:      generateName("test-ingress"),
 			Namespace: ns,
 			Annotations: map[string]string{
 				"tailscale.com/expose": "true",
@@ -44,7 +46,7 @@ func TestL3Ingress(t *testing.T) {
 		},
 		Spec: corev1.ServiceSpec{
 			Selector: map[string]string{
-				"app.kubernetes.io/name": "nginx",
+				"app.kubernetes.io/name": nginx.Name,
 			},
 			Ports: []corev1.ServicePort{
 				{
@@ -58,7 +60,7 @@ func TestL3Ingress(t *testing.T) {
 	createAndCleanup(t, kubeClient, svc)
 
 	if err := tstest.WaitFor(time.Minute, func() error {
-		maybeReadySvc := &corev1.Service{ObjectMeta: objectMeta(ns, name)}
+		maybeReadySvc := &corev1.Service{ObjectMeta: objectMeta(ns, svc.Name)}
 		if err := get(t.Context(), kubeClient, maybeReadySvc); err != nil {
 			return err
 		}
@@ -79,7 +81,7 @@ func TestL3Ingress(t *testing.T) {
 		if err := kubeClient.List(t.Context(), &secrets,
 			client.InNamespace("tailscale"),
 			client.MatchingLabels{
-				"tailscale.com/parent-resource":    name,
+				"tailscale.com/parent-resource":    svc.Name,
 				"tailscale.com/parent-resource-ns": ns,
 			},
 		); err != nil {
@@ -109,33 +111,34 @@ func TestL3HAIngress(t *testing.T) {
 	}
 
 	// Apply nginx.
-	createAndCleanup(t, kubeClient, nginxDeployment(ns, "nginx"))
+	nginx := nginxDeployment(ns)
+	createAndCleanup(t, kubeClient, nginx)
 
 	// Create an ingress ProxyGroup.
-	createAndCleanup(t, kubeClient, &tsapi.ProxyGroup{
+	pg := &tsapi.ProxyGroup{
 		ObjectMeta: metav1.ObjectMeta{
-			Name: "ingress",
+			Name: generateName("ingress"),
 		},
 		Spec: tsapi.ProxyGroupSpec{
 			Type: tsapi.ProxyGroupTypeIngress,
 		},
-	})
+	}
+	createAndCleanup(t, kubeClient, pg)
 
 	// Apply a Service to expose nginx via the ProxyGroup.
-	name := generateName("test-ingress")
 	svc := &corev1.Service{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      name,
+			Name:      generateName("test-ingress"),
 			Namespace: ns,
 			Annotations: map[string]string{
-				"tailscale.com/proxy-group": "ingress",
+				"tailscale.com/proxy-group": pg.Name,
 			},
 		},
 		Spec: corev1.ServiceSpec{
 			Type:              corev1.ServiceTypeLoadBalancer,
 			LoadBalancerClass: new("tailscale"),
 			Selector: map[string]string{
-				"app.kubernetes.io/name": "nginx",
+				"app.kubernetes.io/name": nginx.Name,
 			},
 			Ports: []corev1.ServicePort{
 				{
@@ -150,12 +153,12 @@ func TestL3HAIngress(t *testing.T) {
 
 	var svcIPv4 string
 	forceReconcile := triggerReconcile(t,
-		client.ObjectKey{Namespace: ns, Name: name},
+		client.ObjectKey{Namespace: ns, Name: svc.Name},
 		&corev1.Service{}, 30*time.Second)
 
 	// Wait for Service to be ready
 	if err := tstest.WaitFor(5*time.Minute, func() error {
-		maybeReadySvc := &corev1.Service{ObjectMeta: objectMeta(ns, name)}
+		maybeReadySvc := &corev1.Service{ObjectMeta: objectMeta(ns, svc.Name)}
 		forceReconcile()
 		if err := get(t.Context(), kubeClient, maybeReadySvc); err != nil {
 			return err
@@ -186,15 +189,16 @@ func TestL7Ingress(t *testing.T) {
 	}
 
 	// Apply nginx Deployment and Service.
-	createAndCleanup(t, kubeClient, nginxDeployment(ns, "nginx"))
+	nginx := nginxDeployment(ns)
+	createAndCleanup(t, kubeClient, nginx)
 	createAndCleanup(t, kubeClient, &corev1.Service{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "nginx",
+			Name:      nginx.Name,
 			Namespace: ns,
 		},
 		Spec: corev1.ServiceSpec{
 			Selector: map[string]string{
-				"app.kubernetes.io/name": "nginx",
+				"app.kubernetes.io/name": nginx.Name,
 			},
 			Ports: []corev1.ServicePort{
 				{
@@ -206,13 +210,12 @@ func TestL7Ingress(t *testing.T) {
 	})
 
 	// Apply Ingress to expose nginx.
-	name := generateName("test-ingress")
+	ingress := l7Ingress(ns, nginx.Name, map[string]string{})
-	ingress := l7Ingress(ns, name, map[string]string{})
 	createAndCleanup(t, kubeClient, ingress)
 
 	t.Log("Waiting for the Ingress to be ready...")
 
-	hostname, err := waitForIngressHostname(t, ns, name)
+	hostname, err := waitForIngressHostname(t, ns, ingress.Name)
 	if err != nil {
 		t.Fatalf("error waiting for Ingress hostname: %v", err)
 	}
@@ -228,15 +231,16 @@ func TestL7HAIngress(t *testing.T) {
 	}
 
 	// Apply nginx Deployment and Service.
-	createAndCleanup(t, kubeClient, nginxDeployment(ns, "nginx"))
+	nginx := nginxDeployment(ns)
+	createAndCleanup(t, kubeClient, nginx)
 	createAndCleanup(t, kubeClient, &corev1.Service{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "nginx",
+			Name:      nginx.Name,
 			Namespace: ns,
 		},
 		Spec: corev1.ServiceSpec{
 			Selector: map[string]string{
-				"app.kubernetes.io/name": "nginx",
+				"app.kubernetes.io/name": nginx.Name,
 			},
 			Ports: []corev1.ServicePort{
 				{
@@ -248,23 +252,23 @@ func TestL7HAIngress(t *testing.T) {
 	})
 
 	// Create ProxyGroup that the Ingress will reference.
-	createAndCleanup(t, kubeClient, &tsapi.ProxyGroup{
+	pg := &tsapi.ProxyGroup{
 		ObjectMeta: metav1.ObjectMeta{
-			Name: "ingress",
+			Name: generateName("ingress"),
 		},
 		Spec: tsapi.ProxyGroupSpec{
 			Type: tsapi.ProxyGroupTypeIngress,
 		},
-	})
+	}
+	createAndCleanup(t, kubeClient, pg)
 
 	// Apply Ingress to expose nginx.
-	name := generateName("test-ingress")
+	ingress := l7Ingress(ns, nginx.Name, map[string]string{"tailscale.com/proxy-group": pg.Name})
-	ingress := l7Ingress(ns, name, map[string]string{"tailscale.com/proxy-group": "ingress"})
 	createAndCleanup(t, kubeClient, ingress)
 
 	t.Log("Waiting for the Ingress to be ready...")
 
-	hostname, err := waitForIngressHostname(t, ns, name)
+	hostname, err := waitForIngressHostname(t, ns, ingress.Name)
 	if err != nil {
 		t.Fatalf("error waiting for Ingress hostname: %v", err)
 	}
@@ -274,7 +278,88 @@ func TestL7HAIngress(t *testing.T) {
 	}
 }
 
-func l7Ingress(namespace, name string, annotations map[string]string) *networkingv1.Ingress {
+func TestL7HAIngressMultiTailnet(t *testing.T) {
+	if tnClient == nil || secondTNClient == nil {
+		t.Skip("TestL7HAIngressMultiTailnet requires a working tailnet client for a first and second tailnet")
+	}
+
+	// Apply nginx Deployment and Service.
+	nginx := nginxDeployment(ns)
+	createAndCleanup(t, kubeClient, nginx)
+	createAndCleanup(t, kubeClient, &corev1.Service{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      nginx.Name,
+			Namespace: ns,
+		},
+		Spec: corev1.ServiceSpec{
+			Selector: map[string]string{
+				"app.kubernetes.io/name": nginx.Name,
+			},
+			Ports: []corev1.ServicePort{
+				{
+					Name: "http",
+					Port: 80,
+				},
+			},
+		},
+	})
+
+	// Create Ingress ProxyGroup for each Tailnet.
+	firstTailnetPG := &tsapi.ProxyGroup{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: generateName("first-tailnet"),
+		},
+		Spec: tsapi.ProxyGroupSpec{
+			Type: tsapi.ProxyGroupTypeIngress,
+		},
+	}
+	createAndCleanup(t, kubeClient, firstTailnetPG)
+	secondTailnetPG := &tsapi.ProxyGroup{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: generateName("second-tailnet"),
+		},
+		Spec: tsapi.ProxyGroupSpec{
+			Type:    tsapi.ProxyGroupTypeIngress,
+			Tailnet: "second-tailnet",
+		},
+	}
+	createAndCleanup(t, kubeClient, secondTailnetPG)
+
+	if err := verifyProxyGroupTailnet(t, firstTailnetPG, tnClient); err != nil {
+		t.Fatalf("verifying ProxyGroup %s is registered to the correct tailnet: %v", firstTailnetPG.Name, err)
+	}
+	if err := verifyProxyGroupTailnet(t, secondTailnetPG, secondTNClient); err != nil {
+		t.Fatalf("verifying ProxyGroup %s is registered to the correct tailnet: %v", secondTailnetPG.Name, err)
+	}
+
+	// Apply Ingress to expose nginx.
+	ingress := l7Ingress(ns, nginx.Name, map[string]string{
+		"tailscale.com/proxy-group": secondTailnetPG.Name,
+	})
+	createAndCleanup(t, kubeClient, ingress)
+
+	// Check that the tailscale (VIP) Service has been created in the expected Tailnet.
+	svcName := "svc:" + ingress.Name
+	if err := tstest.WaitFor(3*time.Minute, func() error {
+		_, err := secondTSClient.VIPServices().Get(t.Context(), svcName)
+		if tailscale.IsNotFound(err) {
+			return fmt.Errorf("Tailscale service %q not yet in expected tailnet", svcName)
+		}
+		return err
+	}); err != nil {
+		t.Fatalf("Tailscale service %q never appeared in expected tailnet: %v", svcName, err)
+	}
+	hostname, err := waitForIngressHostname(t, ns, ingress.Name)
+	if err != nil {
+		t.Fatalf("error waiting for Ingress hostname: %v", err)
+	}
+	if err := testIngressIsReachable(t, newHTTPClient(secondTNClient), fmt.Sprintf("https://%s:443", hostname)); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func l7Ingress(namespace, svc string, annotations map[string]string) *networkingv1.Ingress {
+	name := generateName("test-ingress")
 	ingress := &networkingv1.Ingress{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:        name,
@@ -296,7 +381,7 @@ func l7Ingress(namespace, name string, annotations map[string]string) *networkin
 									PathType: new(networkingv1.PathTypePrefix),
 									Backend: networkingv1.IngressBackend{
 										Service: &networkingv1.IngressServiceBackend{
-											Name: "nginx",
+											Name: svc,
 											Port: networkingv1.ServiceBackendPort{
 												Number: 80,
 											},
@@ -313,26 +398,27 @@ func l7Ingress(namespace, name string, annotations map[string]string) *networkin
 	return ingress
 }
 
-func nginxDeployment(namespace, name string) *appsv1.Deployment {
+func nginxDeployment(namespace string) *appsv1.Deployment {
+	name := generateName("nginx")
 	return &appsv1.Deployment{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      name,
 			Namespace: namespace,
 			Labels: map[string]string{
-				"app.kubernetes.io/name": "nginx",
+				"app.kubernetes.io/name": name,
 			},
 		},
 		Spec: appsv1.DeploymentSpec{
 			Replicas: new(int32(1)),
 			Selector: &metav1.LabelSelector{
 				MatchLabels: map[string]string{
-					"app.kubernetes.io/name": "nginx",
+					"app.kubernetes.io/name": name,
 				},
 			},
 			Template: corev1.PodTemplateSpec{
 				ObjectMeta: metav1.ObjectMeta{
 					Labels: map[string]string{
-						"app.kubernetes.io/name": "nginx",
+						"app.kubernetes.io/name": name,
 					},
 				},
 				Spec: corev1.PodSpec{
@@ -406,6 +492,56 @@ func testIngressIsReachable(t *testing.T, httpClient *http.Client, url string) e
 	return nil
 }
 
+// verifyProxyGroupTailnet verifies that a ProxyGroup is registered to the correct tailnet.
+// This is done by getting the expected tailnet domain for the tailnet client,
+// and comparing this with the actual device fqdn in the ProxyGroup state secret.
+func verifyProxyGroupTailnet(t *testing.T, pg *tsapi.ProxyGroup, cl *tsnet.Server) error {
+	t.Helper()
+	// Determine the expected tailnet Magic DNS Name.
+	lc, err := cl.LocalClient()
+	if err != nil {
+		return err
+	}
+	status, err := lc.Status(t.Context())
+	if err != nil {
+		return err
+	}
+	_, expectedTailnet, ok := strings.Cut(strings.TrimSuffix(status.Self.DNSName, "."), ".")
+	if !ok {
+		return fmt.Errorf("unexpected DNSName format %q", status.Self.DNSName)
+	}
+	// Read the device FQDN from the first state secret for the ProxyGroup,
+	// and verify that this matches the expected tailnet.
+	if err := tstest.WaitFor(3*time.Minute, func() error {
+		var secrets corev1.SecretList
+		if err := kubeClient.List(t.Context(), &secrets,
+			client.InNamespace("tailscale"),
+			client.MatchingLabels{
+				kubetypes.LabelSecretType:            kubetypes.LabelSecretTypeState,
+				"tailscale.com/parent-resource-type": "proxygroup",
+				"tailscale.com/parent-resource":      pg.Name,
+			},
+		); err != nil {
+			return err
+		}
+		if len(secrets.Items) == 0 {
+			return fmt.Errorf("no state secrets found for ProxyGroup %q yet", pg.Name)
+		}
+		fqdn := strings.TrimSuffix(string(secrets.Items[0].Data[kubetypes.KeyDeviceFQDN]), ".")
+		_, tailnet, ok := strings.Cut(fqdn, ".")
+		if !ok {
+			return fmt.Errorf("ProxyGroup %q: device FQDN %q has no domain yet", pg.Name, fqdn)
+		}
+		if tailnet != expectedTailnet {
+			return fmt.Errorf("ProxyGroup %q on wrong tailnet: got domain %q, want %q", pg.Name, tailnet, expectedTailnet)
+		}
+		return nil
+	}); err != nil {
+		return fmt.Errorf("ProxyGroup %q not on expected tailnet: %v", pg.Name, err)
+	}
+	return nil
+}
+
 func waitForIngressHostname(t *testing.T, namespace, name string) (string, error) {
 	t.Helper()
 	var hostname string

cmd/k8s-operator/e2e/main_test.go

@@ -54,7 +54,7 @@ func createAndCleanup(t *testing.T, cl client.Client, obj client.Object) {
 	t.Cleanup(func() {
 		// Use context.Background() for cleanup, as t.Context() is cancelled
 		// just before cleanup functions are called.
-		if err = cl.Delete(context.Background(), obj); err != nil {
+		if err := cl.Delete(context.Background(), obj); err != nil {
 			t.Errorf("error cleaning up %s %s/%s: %s", obj.GetObjectKind().GroupVersionKind(), obj.GetNamespace(), obj.GetName(), err)
 		}
 	})
@@ -69,7 +69,7 @@ func createAndCleanupErr(t *testing.T, cl client.Client, obj client.Object) erro
 	}
 
 	t.Cleanup(func() {
-		if err = cl.Delete(context.Background(), obj); err != nil {
+		if err := cl.Delete(context.Background(), obj); err != nil {
 			t.Errorf("error cleaning up %s %s/%s: %s", obj.GetObjectKind().GroupVersionKind(), obj.GetNamespace(), obj.GetName(), err)
 		}
 	})

cmd/k8s-operator/e2e/setup.go

@@ -40,6 +40,7 @@ import (
 	"helm.sh/helm/v3/pkg/release"
 	"helm.sh/helm/v3/pkg/storage/driver"
 	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/watch"
 	"k8s.io/client-go/rest"
@@ -53,12 +54,13 @@ import (
 	"sigs.k8s.io/kind/pkg/cluster/nodeutils"
 	"sigs.k8s.io/kind/pkg/cmd"
 
-	"tailscale.com/internal/client/tailscale"
+	"tailscale.com/client/tailscale/v2"
 	"tailscale.com/ipn"
 	"tailscale.com/ipn/store/mem"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
 	"tailscale.com/tsnet"
+	"tailscale.com/util/must"
 )
 
 const (
@@ -70,9 +72,12 @@ const (
 
 var (
 	tsClient           *tailscale.Client // For API calls to control.
-	tnClient   *tsnet.Server     // For testing real tailnet traffic.
+	tnClient           *tsnet.Server     // For testing real tailnet traffic on first tailnet.
+	secondTSClient     *tailscale.Client // For API calls to the secondary tailnet (_second_tailnet).
+	secondTNClient     *tsnet.Server     // For testing real tailnet traffic on second tailnet.
 	restCfg            *rest.Config      // For constructing a client-go client if necessary.
 	kubeClient         client.WithWatch  // For k8s API calls.
+	clusterLoginServer string
 
 	//go:embed certs/pebble.minica.crt
 	pebbleMiniCACert []byte
@@ -106,7 +111,8 @@ func runTests(m *testing.M) (int, error) {
 	if err != nil {
 		return 0, err
 	}
-	if err := os.MkdirAll(tmp, 0755); err != nil {
+
+	if err = os.MkdirAll(tmp, 0755); err != nil {
 		return 0, fmt.Errorf("failed to create temp dir: %w", err)
 	}
 
@@ -122,10 +128,12 @@ func runTests(m *testing.M) (int, error) {
 		kindProvider = cluster.NewProvider(
 			cluster.ProviderWithLogger(cmd.NewLogger()),
 		)
+
 		clusters, err := kindProvider.List()
 		if err != nil {
 			return 0, fmt.Errorf("failed to list kind clusters: %w", err)
 		}
+
 		if !slices.Contains(clusters, kindClusterName) {
 			if err := kindProvider.Create(kindClusterName,
 				cluster.CreateWithWaitForReady(5*time.Minute),
@@ -147,34 +155,39 @@ func runTests(m *testing.M) (int, error) {
 	if err != nil {
 		return 0, fmt.Errorf("error loading kubeconfig: %w", err)
 	}
+
 	kubeClient, err = client.NewWithWatch(restCfg, client.Options{Scheme: tsapi.GlobalScheme})
 	if err != nil {
 		return 0, fmt.Errorf("error creating Kubernetes client: %w", err)
 	}
 
 	var (
-		clusterLoginServer     string   // Login server from cluster Pod point of view.
+		clientID, clientSecret string   // OAuth client for the first tailnet (for the operator to use).
-		clientID, clientSecret string   // OAuth client for the operator to use.
 		caPaths                []string // Extra CA cert file paths to add to images.
 
-		certsDir string = filepath.Join(tmp, "certs") // Directory containing extra CA certs to add to images.
+		certsDir                           = filepath.Join(tmp, "certs") // Directory containing extra CA certs to add to images.
+		secondClientID, secondClientSecret string                        // OAuth client for the second tailnet (for the operator to use).
 	)
 	if *fDevcontrol {
 		// Deploy pebble and get its certs.
-		if err := applyPebbleResources(ctx, kubeClient); err != nil {
+		if err = applyPebbleResources(ctx, kubeClient); err != nil {
 			return 0, fmt.Errorf("failed to apply pebble resources: %w", err)
 		}
+
 		pebblePod, err := waitForPodReady(ctx, logger, kubeClient, ns, client.MatchingLabels{"app": "pebble"})
 		if err != nil {
 			return 0, fmt.Errorf("pebble pod not ready: %w", err)
 		}
-		if err := forwardLocalPortToPod(ctx, logger, restCfg, ns, pebblePod, 15000); err != nil {
+
+		if err = forwardLocalPortToPod(ctx, logger, restCfg, ns, pebblePod, 15000); err != nil {
 			return 0, fmt.Errorf("failed to set up port forwarding to pebble: %w", err)
 		}
+
 		testCAs = x509.NewCertPool()
 		if ok := testCAs.AppendCertsFromPEM(pebbleMiniCACert); !ok {
 			return 0, fmt.Errorf("failed to parse pebble minica cert")
 		}
+
 		var pebbleCAChain []byte
 		for _, path := range []string{"/intermediates/0", "/roots/0"} {
 			pem, err := pebbleGet(ctx, 15000, path)
@@ -183,20 +196,25 @@ func runTests(m *testing.M) (int, error) {
 			}
 			pebbleCAChain = append(pebbleCAChain, pem...)
 		}
+
 		if ok := testCAs.AppendCertsFromPEM(pebbleCAChain); !ok {
 			return 0, fmt.Errorf("failed to parse pebble ca chain cert")
 		}
-		if err := os.MkdirAll(certsDir, 0755); err != nil {
+
+		if err = os.MkdirAll(certsDir, 0755); err != nil {
 			return 0, fmt.Errorf("failed to create certs dir: %w", err)
 		}
+
 		pebbleCAChainPath := filepath.Join(certsDir, "pebble-ca-chain.crt")
-		if err := os.WriteFile(pebbleCAChainPath, pebbleCAChain, 0644); err != nil {
+		if err = os.WriteFile(pebbleCAChainPath, pebbleCAChain, 0644); err != nil {
 			return 0, fmt.Errorf("failed to write pebble CA chain: %w", err)
 		}
+
 		pebbleMiniCACertPath := filepath.Join(certsDir, "pebble.minica.crt")
-		if err := os.WriteFile(pebbleMiniCACertPath, pebbleMiniCACert, 0644); err != nil {
+		if err = os.WriteFile(pebbleMiniCACertPath, pebbleMiniCACert, 0644); err != nil {
 			return 0, fmt.Errorf("failed to write pebble minica: %w", err)
 		}
+
 		caPaths = []string{pebbleCAChainPath, pebbleMiniCACertPath}
 		if !*fSkipCleanup {
 			defer os.RemoveAll(certsDir)
@@ -210,13 +228,15 @@ func runTests(m *testing.M) (int, error) {
 		// For Pods -> devcontrol (tailscale clients joining the tailnet):
 		// * Create ssh-server Deployment in cluster.
 		// * Create reverse ssh tunnel that goes from ssh-server port 31544 to localhost:31544.
-		if err := forwardLocalPortToPod(ctx, logger, restCfg, ns, pebblePod, 8055); err != nil {
+		if err = forwardLocalPortToPod(ctx, logger, restCfg, ns, pebblePod, 8055); err != nil {
 			return 0, fmt.Errorf("failed to set up port forwarding to pebble: %w", err)
 		}
+
 		privateKey, publicKey, err := readOrGenerateSSHKey(tmp)
 		if err != nil {
 			return 0, fmt.Errorf("failed to read or generate SSH key: %w", err)
 		}
+
 		if !*fSkipCleanup {
 			defer os.Remove(privateKeyPath)
 		}
@@ -225,6 +245,7 @@ func runTests(m *testing.M) (int, error) {
 		if err != nil {
 			return 0, fmt.Errorf("failed to set up cluster->devcontrol connection: %w", err)
 		}
+
 		if !*fSkipCleanup {
 			defer func() {
 				if err := cleanupSSHResources(context.Background(), kubeClient); err != nil {
@@ -245,7 +266,7 @@ func runTests(m *testing.M) (int, error) {
 		var apiKeyData struct {
 			APIKey string `json:"apiKey"`
 		}
-		if err := json.Unmarshal(b, &apiKeyData); err != nil {
+		if err = json.Unmarshal(b, &apiKeyData); err != nil {
 			return 0, fmt.Errorf("failed to parse api-key.json: %w", err)
 		}
 		if apiKeyData.APIKey == "" {
@@ -253,74 +274,96 @@ func runTests(m *testing.M) (int, error) {
 		}
 
 		// Finish setting up tsClient.
-		tsClient = tailscale.NewClient("-", tailscale.APIKey(apiKeyData.APIKey))
+		tsClient = &tailscale.Client{
-		tsClient.BaseURL = "http://localhost:31544"
+			APIKey:  apiKeyData.APIKey,
+			BaseURL: must.Get(url.Parse("http://localhost:31544")),
+		}
 
 		// Set ACLs and create OAuth client.
-		req, _ := http.NewRequest("POST", tsClient.BuildTailnetURL("acl"), bytes.NewReader(requiredACLs))
+		if err = tsClient.PolicyFile().Set(ctx, string(requiredACLs), ""); err != nil {
-		resp, err := tsClient.Do(req)
+			return 0, fmt.Errorf("failed to set policy file: %w", err)
-		if err != nil {
-			return 0, fmt.Errorf("failed to set ACLs: %w", err)
 		}
-		defer resp.Body.Close()
-		if resp.StatusCode != http.StatusOK {
-			b, _ := io.ReadAll(resp.Body)
-			return 0, fmt.Errorf("HTTP %d setting ACLs: %s", resp.StatusCode, string(b))
-		}
-		logger.Infof("ACLs configured")
 
-		reqBody, err := json.Marshal(map[string]any{
+		logger.Info("ACLs configured for first tailnet")
-			"keyType":     "client",
+
-			"scopes":      []string{"auth_keys", "devices:core", "services"},
+		key, err := tsClient.Keys().CreateOAuthClient(ctx, tailscale.CreateOAuthClientRequest{
-			"tags":        []string{"tag:k8s-operator"},
+			Scopes:      []string{"auth_keys", "devices:core", "services"},
-			"description": "k8s-operator client for e2e tests",
+			Tags:        []string{"tag:k8s-operator"},
+			Description: "k8s-operator client for e2e tests",
 		})
 		if err != nil {
-			return 0, fmt.Errorf("failed to marshal OAuth client creation request: %w", err)
+			return 0, fmt.Errorf("failed to create OAuth client for first tailnet: %w", err)
-		}
-		req, _ = http.NewRequest("POST", tsClient.BuildTailnetURL("keys"), bytes.NewReader(reqBody))
-		resp, err = tsClient.Do(req)
-		if err != nil {
-			return 0, fmt.Errorf("failed to create OAuth client: %w", err)
-		}
-		defer resp.Body.Close()
-		if resp.StatusCode != http.StatusOK {
-			b, _ := io.ReadAll(resp.Body)
-			return 0, fmt.Errorf("HTTP %d creating OAuth client: %s", resp.StatusCode, string(b))
-		}
-		var key struct {
-			ID  string `json:"id"`
-			Key string `json:"key"`
-		}
-		if err := json.NewDecoder(resp.Body).Decode(&key); err != nil {
-			return 0, fmt.Errorf("failed to decode OAuth client creation response: %w", err)
 		}
 		clientID = key.ID
 		clientSecret = key.Key
+
+		logger.Info("OAuth credentials set for first tailnet")
+
+		// Create second tailnet. The bootstrap credentials returned have 'all' permissions-
+		// they are used for administrative actions and to create a separately scoped
+		// Oauth client for the k8s operator.
+		bootstrapClient, err := createTailnet(ctx, tsClient)
+		if err != nil {
+			return 0, fmt.Errorf("failed to create second tailnet: %w", err)
+		}
+
+		// Set HTTPS on second tailnet.
+		err = bootstrapClient.TailnetSettings().Update(ctx, tailscale.UpdateTailnetSettingsRequest{HTTPSEnabled: new(true)})
+		if err != nil {
+			return 0, fmt.Errorf("failed to configure https for second tailnet: %w", err)
+		}
+		logger.Info("HTTPS settings configured for second tailnet")
+
+		// Set ACLs for second tailnet.
+		if err = bootstrapClient.PolicyFile().Set(ctx, string(requiredACLs), ""); err != nil {
+			return 0, fmt.Errorf("failed to set policy file: %w", err)
+		}
+
+		logger.Info("ACLs configured for second tailnet")
+
+		// Create an OAuth client for the second tailnet to be used
+		// by the k8s-operator.
+		secondKey, err := bootstrapClient.Keys().CreateOAuthClient(ctx, tailscale.CreateOAuthClientRequest{
+			Scopes:      []string{"auth_keys", "devices:core", "services"},
+			Tags:        []string{"tag:k8s-operator"},
+			Description: "k8s-operator client for e2e tests",
+		})
+		if err != nil {
+			return 0, fmt.Errorf("failed to create OAuth client for second tailnet: %w", err)
+		}
+		secondClientID = secondKey.ID
+		secondClientSecret = secondKey.Key
+
+		secondTSClient, err = tailscaleClientFromSecret(ctx, "http://localhost:31544", secondClientID, secondClientSecret)
+		if err != nil {
+			return 0, fmt.Errorf("failed to set up second tailnet client: %w", err)
+		}
+
 	} else {
 		clientSecret = os.Getenv("TS_API_CLIENT_SECRET")
 		if clientSecret == "" {
 			return 0, fmt.Errorf("must use --devcontrol or set TS_API_CLIENT_SECRET to an OAuth client suitable for the operator")
 		}
-		// Format is "tskey-client-<id>-<random>".
+		clientID, err = clientIDFromSecret(clientSecret)
-		parts := strings.Split(clientSecret, "-")
-		if len(parts) != 4 {
-			return 0, fmt.Errorf("TS_API_CLIENT_SECRET is not valid")
-		}
-		clientID = parts[2]
-		credentials := clientcredentials.Config{
-			ClientID:     clientID,
-			ClientSecret: clientSecret,
-			TokenURL:     fmt.Sprintf("%s/api/v2/oauth/token", ipn.DefaultControlURL),
-			Scopes:       []string{"auth_keys"},
-		}
-		tk, err := credentials.Token(ctx)
 		if err != nil {
-			return 0, fmt.Errorf("failed to get OAuth token: %w", err)
+			return 0, fmt.Errorf("failed to get client id from secret: %w", err)
+		}
+		tsClient, err = tailscaleClientFromSecret(ctx, ipn.DefaultControlURL, clientID, clientSecret)
+		if err != nil {
+			return 0, fmt.Errorf("failed to set up first tailnet client: %w", err)
+		}
+		secondClientSecret = os.Getenv("SECOND_TS_API_CLIENT_SECRET")
+		if secondClientSecret == "" {
+			return 0, fmt.Errorf("must use --devcontrol or set SECOND_TS_API_CLIENT_SECRET to an OAuth client suitable for the operator")
+		}
+		secondClientID, err = clientIDFromSecret(secondClientSecret)
+		if err != nil {
+			return 0, fmt.Errorf("failed to get client id from secret: %w", err)
+		}
+		secondTSClient, err = tailscaleClientFromSecret(ctx, ipn.DefaultControlURL, secondClientID, secondClientSecret)
+		if err != nil {
+			return 0, fmt.Errorf("failed to set up second tailnet client: %w", err)
 		}
-		// An access token will last for an hour which is plenty of time for
-		// the tests to run. No need for token refresh logic.
-		tsClient = tailscale.NewClient("-", tailscale.APIKey(tk.AccessToken))
 	}
 
 	var ossTag string
@@ -447,18 +490,24 @@ func runTests(m *testing.M) (int, error) {
 	caps.Devices.Create.Ephemeral = true
 	caps.Devices.Create.Tags = []string{"tag:k8s"}
 
-	authKey, authKeyMeta, err := tsClient.CreateKey(ctx, caps)
+	authKey, err := tsClient.Keys().CreateAuthKey(ctx, tailscale.CreateKeyRequest{Capabilities: caps})
 	if err != nil {
-		return 0, err
+		return 0, fmt.Errorf("failed to create auth key for first tailnet: %w", err)
 	}
-	defer tsClient.DeleteKey(context.Background(), authKeyMeta.ID)
+	defer tsClient.Keys().Delete(context.Background(), authKey.ID)
+
+	secondAuthKey, err := secondTSClient.Keys().CreateAuthKey(ctx, tailscale.CreateKeyRequest{Capabilities: caps})
+	if err != nil {
+		return 0, fmt.Errorf("failed to create auth key for second tailnet: %w", err)
+	}
+	defer secondTSClient.Keys().Delete(context.Background(), secondAuthKey.ID)
 
 	tnClient = &tsnet.Server{
-		ControlURL: tsClient.BaseURL,
+		ControlURL: tsClient.BaseURL.String(),
 		Hostname:   "test-proxy",
 		Ephemeral:  true,
 		Store:      &mem.Store{},
-		AuthKey:    authKey,
+		AuthKey:    authKey.Key,
 	}
 	_, err = tnClient.Up(ctx)
 	if err != nil {
@@ -466,9 +515,64 @@ func runTests(m *testing.M) (int, error) {
 	}
 	defer tnClient.Close()
 
+	secondTNClient = &tsnet.Server{
+		ControlURL: secondTSClient.BaseURL.String(),
+		Hostname:   "test-proxy",
+		Ephemeral:  true,
+		Store:      &mem.Store{},
+		AuthKey:    secondAuthKey.Key,
+	}
+	_, err = secondTNClient.Up(ctx)
+	if err != nil {
+		return 0, err
+	}
+	defer secondTNClient.Close()
+
+	// Create the tailnet Secret in the tailscale namespace.
+	secret := &corev1.Secret{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "second-tailnet-credentials",
+			Namespace: "tailscale",
+		},
+		Data: map[string][]byte{
+			"client_id":     []byte(secondClientID),
+			"client_secret": []byte(secondClientSecret),
+		},
+	}
+	if err := createOrUpdate(ctx, kubeClient, secret); err != nil {
+		return 0, fmt.Errorf("failed to create second-tailnet-credentials Secret: %w", err)
+	}
+	defer kubeClient.Delete(context.Background(), secret)
+
+	// Create the Tailnet resource.
+	tn := &tsapi.Tailnet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "second-tailnet",
+		},
+		Spec: tsapi.TailnetSpec{
+			LoginURL: clusterLoginServer,
+			Credentials: tsapi.TailnetCredentials{
+				SecretName: "second-tailnet-credentials",
+			},
+		},
+	}
+	if err := createOrUpdate(ctx, kubeClient, tn); err != nil {
+		return 0, fmt.Errorf("failed to create second-tailnet Tailnet: %w", err)
+	}
+	defer kubeClient.Delete(context.Background(), tn)
+
 	return m.Run(), nil
 }
 
+func clientIDFromSecret(clientSecret string) (string, error) {
+	// Format is "tskey-client-<id>-<random>".
+	parts := strings.Split(clientSecret, "-")
+	if len(parts) != 4 {
+		return "", fmt.Errorf("secret is not valid")
+	}
+	return parts[2], nil
+}
+
 func upgraderOrInstaller(cfg *action.Configuration, releaseName string) helmInstallerFunc {
 	hist := action.NewHistory(cfg)
 	hist.Max = 1
@@ -727,3 +831,65 @@ func buildImage(ctx context.Context, dir, repo, target, tag string, extraCACerts
 
 	return nil
 }
+
+func createOrUpdate(ctx context.Context, cl client.Client, obj client.Object) error {
+	if err := cl.Create(ctx, obj); err != nil {
+		if !apierrors.IsAlreadyExists(err) {
+			return err
+		}
+		return cl.Update(ctx, obj)
+	}
+	return nil
+}
+
+// createTailnet creates a new tailnet and returns a tailscale.Client
+// authenticated against it using the bootstrap credentials included in the
+// creation response.
+func createTailnet(ctx context.Context, tsClient *tailscale.Client) (*tailscale.Client, error) {
+	tailnetName := fmt.Sprintf("second-tailnet-%d", time.Now().Unix())
+	body, err := json.Marshal(map[string]any{"displayName": tailnetName})
+	if err != nil {
+		return nil, fmt.Errorf("failed to marshal tailnet creation request: %w", err)
+	}
+	// TODO(beckypauley): change to use a method on tailscale.Client once this is available.
+	req, _ := http.NewRequestWithContext(ctx, "POST", tsClient.BaseURL.String()+"/api/v2/organizations/-/tailnets", bytes.NewBuffer(body))
+	req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", tsClient.APIKey))
+	resp, err := tsClient.HTTP.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create tailnet: %w", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nil, fmt.Errorf("HTTP %d creating tailnet: %s", resp.StatusCode, string(b))
+	}
+	var result struct {
+		OauthClient struct {
+			ID     string `json:"id"`
+			Secret string `json:"secret"`
+		} `json:"oauthClient"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
+		return nil, fmt.Errorf("failed to decode response: %w", err)
+	}
+	return tailscaleClientFromSecret(ctx, tsClient.BaseURL.String(), result.OauthClient.ID, result.OauthClient.Secret)
+}
+
+// tailscaleClientFromSecret exchanges OAuth client credentials for an access token and
+// returns a tailscale.Client configured to use it. The token is valid for
+// one hour, which is sufficient for the tests to run. No need for refresh logic.
+func tailscaleClientFromSecret(ctx context.Context, baseURL, clientID, clientSecret string) (*tailscale.Client, error) {
+	cfg := clientcredentials.Config{
+		ClientID:     clientID,
+		ClientSecret: clientSecret,
+		TokenURL:     fmt.Sprintf("%s/api/v2/oauth/token", baseURL),
+	}
+	tk, err := cfg.Token(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get OAuth token for client %q: %w", clientID, err)
+	}
+	return &tailscale.Client{
+		APIKey:  tk.AccessToken,
+		BaseURL: must.Get(url.Parse(baseURL)),
+	}, nil
+}

cmd/k8s-operator/egress-eps.go

@@ -20,6 +20,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
 	"tailscale.com/kube/egressservices"
 )
 
@@ -90,7 +91,7 @@ func (er *egressEpsReconciler) Reconcile(ctx context.Context, req reconcile.Requ
 		lg.Debugf("No egress config found, likely because ProxyGroup has not been created")
 		return res, nil
 	}
-	cfg, ok := (*cfgs)[tailnetSvc]
+	cfg, ok := cfgs[tailnetSvc]
 	if !ok {
 		lg.Infof("[unexpected] configuration for tailnet service %s not found", tailnetSvc)
 		return res, nil

cmd/k8s-operator/egress-services.go

@@ -30,6 +30,7 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
 	"tailscale.com/kube/egressservices"
@@ -347,11 +348,11 @@ func (esr *egressSvcsReconciler) provision(ctx context.Context, proxyGroupName s
 		return nil, false, nil
 	}
 	tailnetSvc := tailnetSvcName(svc)
-	gotCfg := (*cfgs)[tailnetSvc]
+	gotCfg := cfgs[tailnetSvc]
 	wantsCfg := egressSvcCfg(svc, clusterIPSvc, esr.tsNamespace, lg)
 	if !reflect.DeepEqual(gotCfg, wantsCfg) {
 		lg.Debugf("updating egress services ConfigMap %s", cm.Name)
-		mak.Set(cfgs, tailnetSvc, wantsCfg)
+		mak.Set(&cfgs, tailnetSvc, wantsCfg)
 		bs, err := json.Marshal(cfgs)
 		if err != nil {
 			return nil, false, fmt.Errorf("error marshalling egress services configs: %w", err)
@@ -458,6 +459,7 @@ func (esr *egressSvcsReconciler) clusterIPSvcForEgress(crl map[string]string) *c
 		},
 		Spec: corev1.ServiceSpec{
 			Type:           corev1.ServiceTypeClusterIP,
+			IPFamilyPolicy: new(corev1.IPFamilyPolicyPreferDualStack),
 		},
 	}
 }
@@ -484,19 +486,19 @@ func (esr *egressSvcsReconciler) ensureEgressSvcCfgDeleted(ctx context.Context,
 		lggr.Debugf("ConfigMap does not contain egress service configs")
 		return nil
 	}
-	cfgs := &egressservices.Configs{}
+	cfgs := egressservices.Configs{}
-	if err := json.Unmarshal(bs, cfgs); err != nil {
+	if err := json.Unmarshal(bs, &cfgs); err != nil {
 		return fmt.Errorf("error unmarshalling egress services configs")
 	}
 	tailnetSvc := tailnetSvcName(svc)
-	_, ok := (*cfgs)[tailnetSvc]
+	_, ok := cfgs[tailnetSvc]
 	if !ok {
 		lggr.Debugf("ConfigMap does not contain egress service config, likely because it was already deleted")
 		return nil
 	}
-	lggr.Infof("before deleting config %+#v", *cfgs)
+	lggr.Infof("before deleting config %+#v", cfgs)
-	delete(*cfgs, tailnetSvc)
+	delete(cfgs, tailnetSvc)
-	lggr.Infof("after deleting config %+#v", *cfgs)
+	lggr.Infof("after deleting config %+#v", cfgs)
 	bs, err := json.Marshal(cfgs)
 	if err != nil {
 		return fmt.Errorf("error marshalling egress services configs: %w", err)
@@ -648,7 +650,7 @@ func isEgressSvcForProxyGroup(obj client.Object) bool {
 
 // egressSvcConfig returns a ConfigMap that contains egress services configuration for the provided ProxyGroup as well
 // as unmarshalled configuration from the ConfigMap.
-func egressSvcsConfigs(ctx context.Context, cl client.Client, proxyGroupName, tsNamespace string) (cm *corev1.ConfigMap, cfgs *egressservices.Configs, err error) {
+func egressSvcsConfigs(ctx context.Context, cl client.Client, proxyGroupName, tsNamespace string) (cm *corev1.ConfigMap, cfgs egressservices.Configs, err error) {
 	name := pgEgressCMName(proxyGroupName)
 	cm = &corev1.ConfigMap{
 		ObjectMeta: metav1.ObjectMeta{
@@ -663,9 +665,9 @@ func egressSvcsConfigs(ctx context.Context, cl client.Client, proxyGroupName, ts
 	if err != nil {
 		return nil, nil, fmt.Errorf("error retrieving egress services ConfigMap %s: %v", name, err)
 	}
-	cfgs = &egressservices.Configs{}
+	cfgs = egressservices.Configs{}
 	if len(cm.BinaryData[egressservices.KeyEgressServices]) != 0 {
-		if err := json.Unmarshal(cm.BinaryData[egressservices.KeyEgressServices], cfgs); err != nil {
+		if err := json.Unmarshal(cm.BinaryData[egressservices.KeyEgressServices], &cfgs); err != nil {
 			return nil, nil, fmt.Errorf("error unmarshaling egress services config %v: %w", cm.BinaryData[egressservices.KeyEgressServices], err)
 		}
 	}

cmd/k8s-operator/egress-services_test.go

@@ -21,6 +21,7 @@ import (
 	"k8s.io/apimachinery/pkg/util/intstr"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
 	"tailscale.com/kube/egressservices"
 	"tailscale.com/tstest"
@@ -204,6 +205,7 @@ func clusterIPSvc(name string, extNSvc *corev1.Service) *corev1.Service {
 		},
 		Spec: corev1.ServiceSpec{
 			Type:           corev1.ServiceTypeClusterIP,
+			IPFamilyPolicy: new(corev1.IPFamilyPolicyPreferDualStack),
 			Ports:          ports,
 		},
 	}
@@ -283,11 +285,11 @@ func configFromCM(t *testing.T, cm *corev1.ConfigMap, svcName string) *egressser
 	if !ok {
 		return nil
 	}
-	cfgs := &egressservices.Configs{}
+	cfgs := egressservices.Configs{}
-	if err := json.Unmarshal(cfgBs, cfgs); err != nil {
+	if err := json.Unmarshal(cfgBs, &cfgs); err != nil {
 		t.Fatalf("error unmarshalling config: %v", err)
 	}
-	cfg, ok := (*cfgs)[svcName]
+	cfg, ok := cfgs[svcName]
 	if ok {
 		return &cfg
 	}

cmd/k8s-operator/ingress-for-pg.go

@@ -12,7 +12,6 @@ import (
 	"fmt"
 	"maps"
 	"math/rand/v2"
-	"net/http"
 	"reflect"
 	"slices"
 	"strings"
@@ -30,11 +29,12 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/internal/client/tailscale"
 	"tailscale.com/ipn"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
 	"tailscale.com/util/clientmetric"
@@ -64,7 +64,7 @@ type HAIngressReconciler struct {
 
 	recorder         record.EventRecorder
 	logger           *zap.SugaredLogger
-	tsClient         tsClient
+	clients          ClientProvider
 	tsnetServer      tsnetServer
 	tsNamespace      string
 	defaultTags      []string
@@ -127,7 +127,7 @@ func (r *HAIngressReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 		return res, fmt.Errorf("getting ProxyGroup %q: %w", pgName, err)
 	}
 
-	tailscaleClient, err := clientFromProxyGroup(ctx, r.Client, pg, r.tsNamespace, r.tsClient)
+	tsClient, err := r.clients.For(pg.Spec.Tailnet)
 	if err != nil {
 		return res, fmt.Errorf("failed to get tailscale client: %w", err)
 	}
@@ -139,9 +139,9 @@ func (r *HAIngressReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 	// resulted in another actor overwriting our Tailscale Service update.
 	needsRequeue := false
 	if !ing.DeletionTimestamp.IsZero() || !r.shouldExpose(ing) {
-		needsRequeue, err = r.maybeCleanup(ctx, hostname, ing, logger, tailscaleClient, pg)
+		needsRequeue, err = r.maybeCleanup(ctx, hostname, ing, logger, tsClient, pg)
 	} else {
-		needsRequeue, err = r.maybeProvision(ctx, hostname, ing, logger, tailscaleClient, pg)
+		needsRequeue, err = r.maybeProvision(ctx, hostname, ing, logger, tsClient, pg)
 	}
 	if err != nil {
 		return res, err
@@ -160,12 +160,12 @@ func (r *HAIngressReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 // If a Tailscale Service exists, but does not have an owner reference from any operator, we error
 // out assuming that this is an owner reference created by an unknown actor.
 // Returns true if the operation resulted in a Tailscale Service update.
-func (r *HAIngressReconciler) maybeProvision(ctx context.Context, hostname string, ing *networkingv1.Ingress, logger *zap.SugaredLogger, tsClient tsClient, pg *tsapi.ProxyGroup) (svcsChanged bool, err error) {
+func (r *HAIngressReconciler) maybeProvision(ctx context.Context, hostname string, ing *networkingv1.Ingress, logger *zap.SugaredLogger, tsClient tsclient.Client, pg *tsapi.ProxyGroup) (svcsChanged bool, err error) {
 	// Currently (2025-05) Tailscale Services are behind an alpha feature flag that
 	// needs to be explicitly enabled for a tailnet to be able to use them.
 	serviceName := tailcfg.ServiceName("svc:" + hostname)
-	existingTSSvc, err := tsClient.GetVIPService(ctx, serviceName)
+	existingTSSvc, err := tsClient.VIPServices().Get(ctx, serviceName.String())
-	if err != nil && !isErrorTailscaleServiceNotFound(err) {
+	if err != nil && !tailscale.IsNotFound(err) {
 		return false, fmt.Errorf("error getting Tailscale Service %q: %w", hostname, err)
 	}
 
@@ -341,8 +341,8 @@ func (r *HAIngressReconciler) maybeProvision(ctx context.Context, hostname strin
 		tsSvcPorts = append(tsSvcPorts, "tcp:80")
 	}
 
-	tsSvc := &tailscale.VIPService{
+	tsSvc := tailscale.VIPService{
-		Name:        serviceName,
+		Name:        serviceName.String(),
 		Tags:        tags,
 		Ports:       tsSvcPorts,
 		Comment:     managedTSServiceComment,
@@ -357,9 +357,9 @@ func (r *HAIngressReconciler) maybeProvision(ctx context.Context, hostname strin
 	if existingTSSvc == nil ||
 		!reflect.DeepEqual(tsSvc.Tags, existingTSSvc.Tags) ||
 		!reflect.DeepEqual(tsSvc.Ports, existingTSSvc.Ports) ||
-		!ownersAreSetAndEqual(tsSvc, existingTSSvc) {
+		!ownersAreSetAndEqual(tsSvc, *existingTSSvc) {
 		logger.Infof("Ensuring Tailscale Service exists and is up to date")
-		if err := tsClient.CreateOrUpdateVIPService(ctx, tsSvc); err != nil {
+		if err := tsClient.VIPServices().CreateOrUpdate(ctx, tsSvc); err != nil {
 			return false, fmt.Errorf("error creating Tailscale Service: %w", err)
 		}
 	}
@@ -375,7 +375,7 @@ func (r *HAIngressReconciler) maybeProvision(ctx context.Context, hostname strin
 	}
 
 	// 6. Update Ingress status if ProxyGroup Pods are ready.
-	count, err := numberPodsAdvertising(ctx, r.Client, r.tsNamespace, pg.Name, serviceName)
+	count, err := numberPodsAdvertising(ctx, r.Client, r.tsNamespace, pg.Name, serviceName.String())
 	if err != nil {
 		return false, fmt.Errorf("failed to check if any Pods are configured: %w", err)
 	}
@@ -440,7 +440,7 @@ func (r *HAIngressReconciler) maybeProvision(ctx context.Context, hostname strin
 // operator instances, else the owner reference is cleaned up.  Returns true if
 // the operation resulted in an existing Tailscale Service updates (owner
 // reference removal).
-func (r *HAIngressReconciler) maybeCleanupProxyGroup(ctx context.Context, logger *zap.SugaredLogger, tsClient tsClient, pg *tsapi.ProxyGroup) (svcsChanged bool, err error) {
+func (r *HAIngressReconciler) maybeCleanupProxyGroup(ctx context.Context, logger *zap.SugaredLogger, tsClient tsclient.Client, pg *tsapi.ProxyGroup) (svcsChanged bool, err error) {
 	// Get serve config for the ProxyGroup
 	cm, cfg, err := r.proxyGroupServeConfig(ctx, pg.Name)
 	if err != nil {
@@ -470,11 +470,11 @@ func (r *HAIngressReconciler) maybeCleanupProxyGroup(ctx context.Context, logger
 
 		if !found {
 			logger.Infof("Tailscale Service %q is not owned by any Ingress, cleaning up", tsSvcName)
-			tsService, err := tsClient.GetVIPService(ctx, tsSvcName)
+			tsService, err := tsClient.VIPServices().Get(ctx, tsSvcName.String())
-			if isErrorTailscaleServiceNotFound(err) {
+			switch {
+			case tailscale.IsNotFound(err):
 				return false, nil
-			}
+			case err != nil:
-			if err != nil {
 				return false, fmt.Errorf("getting Tailscale Service %q: %w", tsSvcName, err)
 			}
 
@@ -519,17 +519,19 @@ func (r *HAIngressReconciler) maybeCleanupProxyGroup(ctx context.Context, logger
 // Ingress is being deleted or is unexposed. The cleanup is safe for a multi-cluster setup- the Tailscale Service is only
 // deleted if it does not contain any other owner references. If it does the cleanup only removes the owner reference
 // corresponding to this Ingress.
-func (r *HAIngressReconciler) maybeCleanup(ctx context.Context, hostname string, ing *networkingv1.Ingress, logger *zap.SugaredLogger, tsClient tsClient, pg *tsapi.ProxyGroup) (svcChanged bool, err error) {
+func (r *HAIngressReconciler) maybeCleanup(ctx context.Context, hostname string, ing *networkingv1.Ingress, logger *zap.SugaredLogger, tsClient tsclient.Client, pg *tsapi.ProxyGroup) (svcChanged bool, err error) {
 	logger.Debugf("Ensuring any resources for Ingress are cleaned up")
 	ix := slices.Index(ing.Finalizers, FinalizerNamePG)
 	if ix < 0 {
 		logger.Debugf("no finalizer, nothing to do")
 		return false, nil
 	}
+
 	logger.Infof("Ensuring that Tailscale Service %q configuration is cleaned up", hostname)
 	serviceName := tailcfg.ServiceName("svc:" + hostname)
-	svc, err := tsClient.GetVIPService(ctx, serviceName)
+
-	if err != nil && !isErrorTailscaleServiceNotFound(err) {
+	svc, err := tsClient.VIPServices().Get(ctx, serviceName.String())
+	if err != nil && !tailscale.IsNotFound(err) {
 		return false, fmt.Errorf("error getting Tailscale Service: %w", err)
 	}
 
@@ -698,10 +700,7 @@ func (r *HAIngressReconciler) validateIngress(ctx context.Context, ing *networki
 // If a Tailscale Service is found, but contains other owner references, only removes this operator's owner reference.
 // If a Tailscale Service by the given name is not found or does not contain this operator's owner reference, do nothing.
 // It returns true if an existing Tailscale Service was updated to remove owner reference, as well as any error that occurred.
-func (r *HAIngressReconciler) cleanupTailscaleService(ctx context.Context, svc *tailscale.VIPService, logger *zap.SugaredLogger, tsClient tsClient) (updated bool, _ error) {
+func (r *HAIngressReconciler) cleanupTailscaleService(ctx context.Context, svc *tailscale.VIPService, logger *zap.SugaredLogger, tsClient tsclient.Client) (updated bool, _ error) {
-	if svc == nil {
-		return false, nil
-	}
 	o, err := parseOwnerAnnotation(svc)
 	if err != nil {
 		return false, fmt.Errorf("error parsing Tailscale Service's owner annotation")
@@ -721,7 +720,7 @@ func (r *HAIngressReconciler) cleanupTailscaleService(ctx context.Context, svc *
 	}
 	if len(o.OwnerRefs) == 1 {
 		logger.Infof("Deleting Tailscale Service %q", svc.Name)
-		if err = tsClient.DeleteVIPService(ctx, svc.Name); err != nil && !isErrorTailscaleServiceNotFound(err) {
+		if err = tsClient.VIPServices().Delete(ctx, svc.Name); err != nil && !tailscale.IsNotFound(err) {
 			return false, err
 		}
 
@@ -735,7 +734,7 @@ func (r *HAIngressReconciler) cleanupTailscaleService(ctx context.Context, svc *
 		return false, fmt.Errorf("error marshalling updated Tailscale Service owner reference: %w", err)
 	}
 	svc.Annotations[ownerAnnotation] = string(json)
-	return true, tsClient.CreateOrUpdateVIPService(ctx, svc)
+	return true, tsClient.VIPServices().CreateOrUpdate(ctx, *svc)
 }
 
 // isHTTPEndpointEnabled returns true if the Ingress has been configured to expose an HTTP endpoint to tailnet.
@@ -819,7 +818,7 @@ func (r *HAIngressReconciler) maybeUpdateAdvertiseServicesConfig(ctx context.Con
 	return nil
 }
 
-func numberPodsAdvertising(ctx context.Context, cl client.Client, tsNamespace, pgName string, serviceName tailcfg.ServiceName) (int, error) {
+func numberPodsAdvertising(ctx context.Context, cl client.Client, tsNamespace, pgName string, serviceName string) (int, error) {
 	// Get all state Secrets for this ProxyGroup.
 	secrets := &corev1.SecretList{}
 	if err := cl.List(ctx, secrets, client.InNamespace(tsNamespace), client.MatchingLabels(pgSecretLabels(pgName, kubetypes.LabelSecretTypeState))); err != nil {
@@ -835,7 +834,7 @@ func numberPodsAdvertising(ctx context.Context, cl client.Client, tsNamespace, p
 		if !ok {
 			continue
 		}
-		if slices.Contains(prefs.AdvertiseServices, serviceName.String()) {
+		if slices.Contains(prefs.AdvertiseServices, serviceName) {
 			count++
 		}
 	}
@@ -912,6 +911,10 @@ func ownerAnnotations(operatorID string, svc *tailscale.VIPService) (map[string]
 
 // parseOwnerAnnotation returns nil if no valid owner found.
 func parseOwnerAnnotation(tsSvc *tailscale.VIPService) (*ownerAnnotationValue, error) {
+	if tsSvc == nil {
+		return nil, nil
+	}
+
 	if tsSvc.Annotations == nil || tsSvc.Annotations[ownerAnnotation] == "" {
 		return nil, nil
 	}
@@ -922,9 +925,8 @@ func parseOwnerAnnotation(tsSvc *tailscale.VIPService) (*ownerAnnotationValue, e
 	return o, nil
 }
 
-func ownersAreSetAndEqual(a, b *tailscale.VIPService) bool {
+func ownersAreSetAndEqual(a, b tailscale.VIPService) bool {
-	return a != nil && b != nil &&
+	return a.Annotations != nil && b.Annotations != nil &&
-		a.Annotations != nil && b.Annotations != nil &&
 		a.Annotations[ownerAnnotation] != "" &&
 		b.Annotations[ownerAnnotation] != "" &&
 		strings.EqualFold(a.Annotations[ownerAnnotation], b.Annotations[ownerAnnotation])
@@ -1079,7 +1081,7 @@ func certResourceLabels(pgName, domain string) map[string]string {
 	return map[string]string{
 		kubetypes.LabelManaged: "true",
 		labelProxyGroup:        pgName,
-		labelDomain:            domain,
+		labelDomain:            tsoperator.TruncateLabelValue(domain),
 	}
 }
 
@@ -1107,11 +1109,6 @@ func hasCerts(ctx context.Context, cl client.Client, ns string, svc tailcfg.Serv
 	return len(cert) > 0 && len(key) > 0, nil
 }
 
-func isErrorTailscaleServiceNotFound(err error) bool {
-	errResp, ok := errors.AsType[tailscale.ErrResponse](err)
-	return ok && errResp.Status == http.StatusNotFound
-}
-
 func tagViolations(obj client.Object) []string {
 	var violations []string
 	if obj == nil {

cmd/k8s-operator/ingress-for-pg_test.go

@@ -25,11 +25,12 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/internal/client/tailscale"
 	"tailscale.com/ipn"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
 )
@@ -88,7 +89,7 @@ func TestIngressPGReconciler(t *testing.T) {
 	expectReconciled(t, ingPGR, "default", "test-ingress")
 
 	// Verify Tailscale Service uses custom tags
-	tsSvc, err := ft.GetVIPService(t.Context(), "svc:my-svc")
+	tsSvc, err := ft.VIPServices().Get(t.Context(), "svc:my-svc")
 	if err != nil {
 		t.Fatalf("getting Tailscale Service: %v", err)
 	}
@@ -259,7 +260,7 @@ func TestIngressPGReconciler(t *testing.T) {
 	expectReconciled(t, ingPGR, ing3.Namespace, ing3.Name)
 
 	// Delete the service from "control"
-	ft.vipServices = make(map[tailcfg.ServiceName]*tailscale.VIPService)
+	ft.vipServices = make(map[string]tailscale.VIPService)
 
 	// Delete the ingress and confirm we don't get stuck due to the VIP service not existing.
 	if err = fc.Delete(t.Context(), ing3); err != nil {
@@ -319,11 +320,11 @@ func TestIngressPGReconciler_UpdateIngressHostname(t *testing.T) {
 	verifyTailscaleService(t, ft, "svc:updated-svc", []string{"tcp:443"})
 	verifyTailscaledConfig(t, fc, "test-pg", []string{"svc:updated-svc"})
 
-	_, err := ft.GetVIPService(context.Background(), "svc:my-svc")
+	_, err := ft.VIPServices().Get(context.Background(), "svc:my-svc")
 	if err == nil {
 		t.Fatalf("svc:my-svc not cleaned up")
 	}
-	if !isErrorTailscaleServiceNotFound(err) {
+	if !tailscale.IsNotFound(err) {
 		t.Fatalf("unexpected error: %v", err)
 	}
 }
@@ -877,20 +878,18 @@ func TestIngressPGReconciler_MultiCluster(t *testing.T) {
 	mustCreate(t, fc, ing)
 
 	// Simulate existing Tailscale Service from another cluster
-	existingVIPSvc := &tailscale.VIPService{
+	existingVIPSvc := tailscale.VIPService{
 		Name: "svc:my-svc",
 		Annotations: map[string]string{
 			ownerAnnotation: `{"ownerrefs":[{"operatorID":"operator-2"}]}`,
 		},
 	}
-	ft.vipServices = map[tailcfg.ServiceName]*tailscale.VIPService{
+	ft.VIPServices().CreateOrUpdate(t.Context(), existingVIPSvc)
-		"svc:my-svc": existingVIPSvc,
-	}
 
 	// Verify reconciliation adds our operator reference
 	expectReconciled(t, ingPGR, "default", "test-ingress")
 
-	tsSvc, err := ft.GetVIPService(context.Background(), "svc:my-svc")
+	tsSvc, err := ft.VIPServices().Get(context.Background(), "svc:my-svc")
 	if err != nil {
 		t.Fatalf("getting Tailscale Service: %v", err)
 	}
@@ -917,7 +916,7 @@ func TestIngressPGReconciler_MultiCluster(t *testing.T) {
 	}
 	expectRequeue(t, ingPGR, "default", "test-ingress")
 
-	tsSvc, err = ft.GetVIPService(context.Background(), "svc:my-svc")
+	tsSvc, err = ft.VIPServices().Get(context.Background(), "svc:my-svc")
 	if err != nil {
 		t.Fatalf("getting Tailscale Service after deletion: %v", err)
 	}
@@ -1024,7 +1023,7 @@ func populateTLSSecret(t *testing.T, c client.Client, pgName, domain string) {
 
 func verifyTailscaleService(t *testing.T, ft *fakeTSClient, serviceName string, wantPorts []string) {
 	t.Helper()
-	tsSvc, err := ft.GetVIPService(context.Background(), tailcfg.ServiceName(serviceName))
+	tsSvc, err := ft.VIPServices().Get(context.Background(), serviceName)
 	if err != nil {
 		t.Fatalf("getting Tailscale Service %q: %v", serviceName, err)
 	}
@@ -1203,7 +1202,9 @@ func setupIngressTest(t *testing.T) (*HAIngressReconciler, client.Client, *fakeT
 
 	fakeTsnetServer := &fakeTSNetServer{certDomains: []string{"foo.com"}}
 
-	ft := &fakeTSClient{}
+	ft := &fakeTSClient{
+		vipServices: make(map[string]tailscale.VIPService),
+	}
 	zl, err := zap.NewDevelopment()
 	if err != nil {
 		t.Fatal(err)
@@ -1211,7 +1212,7 @@ func setupIngressTest(t *testing.T) (*HAIngressReconciler, client.Client, *fakeT
 
 	ingPGR := &HAIngressReconciler{
 		Client:           fc,
-		tsClient:         ft,
+		clients:          tsclient.NewProvider(ft),
 		defaultTags:      []string{"tag:k8s"},
 		tsNamespace:      "operator-ns",
 		tsnetServer:      fakeTsnetServer,

cmd/k8s-operator/ingress_test.go

@@ -21,8 +21,11 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"tailscale.com/client/tailscale/v2"
+
 	"tailscale.com/ipn"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tstest"
 	"tailscale.com/util/mak"
@@ -30,7 +33,9 @@ import (
 
 func TestTailscaleIngress(t *testing.T) {
 	fc := fake.NewFakeClient(ingressClass())
-	ft := &fakeTSClient{}
+	ft := &fakeTSClient{
+		vipServices: make(map[string]tailscale.VIPService),
+	}
 	fakeTsnetServer := &fakeTSNetServer{certDomains: []string{"foo.com"}}
 	zl, err := zap.NewDevelopment()
 	if err != nil {
@@ -41,7 +46,7 @@ func TestTailscaleIngress(t *testing.T) {
 		ingressClassName: "tailscale",
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			tsnetServer:       fakeTsnetServer,
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
@@ -130,7 +135,7 @@ func TestTailscaleIngressHostname(t *testing.T) {
 		ingressClassName: "tailscale",
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			tsnetServer:       fakeTsnetServer,
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
@@ -269,7 +274,7 @@ func TestTailscaleIngressWithProxyClass(t *testing.T) {
 		ingressClassName: "tailscale",
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			tsnetServer:       fakeTsnetServer,
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
@@ -378,7 +383,7 @@ func TestTailscaleIngressWithServiceMonitor(t *testing.T) {
 		ingressClassName: "tailscale",
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			tsnetServer:       fakeTsnetServer,
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
@@ -530,7 +535,7 @@ func TestIngressProxyClassAnnotation(t *testing.T) {
 				ingressClassName: "tailscale",
 				ssr: &tailscaleSTSReconciler{
 					Client:            fc,
-					tsClient:          &fakeTSClient{},
+					clients:           tsclient.NewProvider(&fakeTSClient{}),
 					tsnetServer:       &fakeTSNetServer{certDomains: []string{"test-host"}},
 					defaultTags:       []string{"tag:test"},
 					operatorNamespace: "operator-ns",
@@ -601,7 +606,7 @@ func TestIngressLetsEncryptStaging(t *testing.T) {
 				ingressClassName: "tailscale",
 				ssr: &tailscaleSTSReconciler{
 					Client:            fc,
-					tsClient:          &fakeTSClient{},
+					clients:           tsclient.NewProvider(&fakeTSClient{}),
 					tsnetServer:       &fakeTSNetServer{certDomains: []string{"test-host"}},
 					defaultTags:       []string{"tag:test"},
 					operatorNamespace: "operator-ns",
@@ -710,7 +715,7 @@ func TestEmptyPath(t *testing.T) {
 				ingressClassName: "tailscale",
 				ssr: &tailscaleSTSReconciler{
 					Client:            fc,
-					tsClient:          ft,
+					clients:           tsclient.NewProvider(ft),
 					tsnetServer:       fakeTsnetServer,
 					defaultTags:       []string{"tag:k8s"},
 					operatorNamespace: "operator-ns",
@@ -853,7 +858,7 @@ func TestTailscaleIngressWithHTTPRedirect(t *testing.T) {
 		ingressClassName: "tailscale",
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			tsnetServer:       fakeTsnetServer,
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",

cmd/k8s-operator/metrics_resources.go

@@ -19,6 +19,7 @@ import (
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+	kube "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
 	"tailscale.com/kube/kubetypes"
 )
@@ -227,13 +228,13 @@ func metricsResourceLabels(opts *metricsOpts) map[string]string {
 		kubetypes.LabelManaged:   "true",
 		labelMetricsTarget:       opts.proxyStsName,
 		labelPromProxyType:       opts.proxyType,
-		labelPromProxyParentName: opts.proxyLabels[LabelParentName],
+		labelPromProxyParentName: kube.TruncateLabelValue(opts.proxyLabels[LabelParentName]),
 	}
 	// Include namespace label for proxies created for a namespaced type.
 	if isNamespacedProxyType(opts.proxyType) {
-		lbls[labelPromProxyParentNamespace] = opts.proxyLabels[LabelParentNamespace]
+		lbls[labelPromProxyParentNamespace] = kube.TruncateLabelValue(opts.proxyLabels[LabelParentNamespace])
 	}
-	lbls[labelPromJob] = promJobName(opts)
+	lbls[labelPromJob] = kube.TruncateLabelValue(promJobName(opts))
 	return lbls
 }
 
@@ -250,11 +251,11 @@ func promJobName(opts *metricsOpts) string {
 func metricsSvcSelector(proxyLabels map[string]string, proxyType string) map[string]string {
 	sel := map[string]string{
 		labelPromProxyType:       proxyType,
-		labelPromProxyParentName: proxyLabels[LabelParentName],
+		labelPromProxyParentName: kube.TruncateLabelValue(proxyLabels[LabelParentName]),
 	}
 	// Include namespace label for proxies created for a namespaced type.
 	if isNamespacedProxyType(proxyType) {
-		sel[labelPromProxyParentNamespace] = proxyLabels[LabelParentNamespace]
+		sel[labelPromProxyParentNamespace] = kube.TruncateLabelValue(proxyLabels[LabelParentNamespace])
 	}
 	return sel
 }

cmd/k8s-operator/nameserver.go

@@ -190,6 +190,8 @@ func (a *NameserverReconciler) maybeProvision(ctx context.Context, tsDNSCfg *tsa
 	}
 	if tsDNSCfg.Spec.Nameserver.Pod != nil {
 		dCfg.tolerations = tsDNSCfg.Spec.Nameserver.Pod.Tolerations
+		dCfg.affinity = tsDNSCfg.Spec.Nameserver.Pod.Affinity
+		dCfg.nodeSelector = tsDNSCfg.Spec.Nameserver.Pod.NodeSelector
 	}
 
 	for _, deployable := range []deployable{saDeployable, deployDeployable, svcDeployable, cmDeployable} {
@@ -225,6 +227,8 @@ type deployConfig struct {
 	namespace    string
 	clusterIP    string
 	tolerations  []corev1.Toleration
+	affinity     *corev1.Affinity
+	nodeSelector map[string]string
 }
 
 var (
@@ -250,6 +254,8 @@ var (
 			d.ObjectMeta.Labels = cfg.labels
 			d.ObjectMeta.OwnerReferences = cfg.ownerRefs
 			d.Spec.Template.Spec.Tolerations = cfg.tolerations
+			d.Spec.Template.Spec.Affinity = cfg.affinity
+			d.Spec.Template.Spec.NodeSelector = cfg.nodeSelector
 			updateF := func(oldD *appsv1.Deployment) {
 				oldD.Spec = d.Spec
 			}

cmd/k8s-operator/nameserver_test.go

@@ -43,6 +43,9 @@ func TestNameserverReconciler(t *testing.T) {
 					ClusterIP: "5.4.3.2",
 				},
 				Pod: &tsapi.NameserverPod{
+					NodeSelector: map[string]string{
+						"foo": "bar",
+					},
 					Tolerations: []corev1.Toleration{
 						{
 							Key:      "some-key",
@@ -51,6 +54,23 @@ func TestNameserverReconciler(t *testing.T) {
 							Effect:   corev1.TaintEffectNoSchedule,
 						},
 					},
+					Affinity: &corev1.Affinity{
+						NodeAffinity: &corev1.NodeAffinity{
+							RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
+								NodeSelectorTerms: []corev1.NodeSelectorTerm{
+									{
+										MatchExpressions: []corev1.NodeSelectorRequirement{
+											{
+												Key:      "some-key",
+												Operator: corev1.NodeSelectorOpIn,
+												Values:   []string{"some-value"},
+											},
+										},
+									},
+								},
+							},
+						},
+					},
 				},
 			},
 		},
@@ -97,6 +117,26 @@ func TestNameserverReconciler(t *testing.T) {
 				Effect:   corev1.TaintEffectNoSchedule,
 			},
 		}
+		wantsDeploy.Spec.Template.Spec.Affinity = &corev1.Affinity{
+			NodeAffinity: &corev1.NodeAffinity{
+				RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
+					NodeSelectorTerms: []corev1.NodeSelectorTerm{
+						{
+							MatchExpressions: []corev1.NodeSelectorRequirement{
+								{
+									Key:      "some-key",
+									Operator: corev1.NodeSelectorOpIn,
+									Values:   []string{"some-value"},
+								},
+							},
+						},
+					},
+				},
+			},
+		}
+		wantsDeploy.Spec.Template.Spec.NodeSelector = map[string]string{
+			"foo": "bar",
+		}
 
 		expectEqual(t, fc, wantsDeploy)
 	})

cmd/k8s-operator/operator.go

@@ -46,9 +46,9 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
 	"tailscale.com/client/local"
-	"tailscale.com/client/tailscale"
 	"tailscale.com/envknob"
 	"tailscale.com/hostinfo"
 	"tailscale.com/ipn"
@@ -57,6 +57,7 @@ import (
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
 	"tailscale.com/k8s-operator/reconciler/proxygrouppolicy"
 	"tailscale.com/k8s-operator/reconciler/tailnet"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tsnet"
 	"tailscale.com/tstime"
@@ -84,10 +85,6 @@ const (
 )
 
 func main() {
-	// Required to use our client API. We're fine with the instability since the
-	// client lives in the same repo as this code.
-	tailscale.I_Acknowledge_This_API_Is_Unstable = true
-
 	var (
 		tsNamespace           = defaultEnv("OPERATOR_NAMESPACE", "")
 		tslogging             = defaultEnv("OPERATOR_LOGGING", "info")
@@ -155,7 +152,7 @@ func main() {
 		}))
 	}
 
-	rOpts := reconcilerOpts{
+	runReconcilers(reconcilerOpts{
 		log:                           zlog,
 		tsServer:                      s,
 		tsClient:                      tsc,
@@ -170,15 +167,14 @@ func main() {
 		defaultProxyClass:             defaultProxyClass,
 		loginServer:                   loginServer,
 		ingressClassName:              ingressClassName,
-	}
+	})
-	runReconcilers(rOpts)
 }
 
 // initTSNet initializes the tsnet.Server and logs in to Tailscale. If CLIENT_ID
 // is set, it authenticates to the Tailscale API using the federated OIDC workload
 // identity flow. Otherwise, it uses the CLIENT_ID_FILE and CLIENT_SECRET_FILE
 // environment variables to authenticate with static credentials.
-func initTSNet(zlog *zap.SugaredLogger, loginServer string) (*tsnet.Server, tsClient) {
+func initTSNet(zlog *zap.SugaredLogger, loginServer string) (*tsnet.Server, *tailscale.Client) {
 	var (
 		clientID         = defaultEnv("CLIENT_ID", "")          // Used for workload identity federation.
 		clientIDPath     = defaultEnv("CLIENT_ID_FILE", "")     // Used for static client credentials.
@@ -187,19 +183,23 @@ func initTSNet(zlog *zap.SugaredLogger, loginServer string) (*tsnet.Server, tsCl
 		kubeSecret       = defaultEnv("OPERATOR_SECRET", "")
 		operatorTags     = defaultEnv("OPERATOR_INITIAL_TAGS", "tag:k8s-operator")
 	)
+
 	startlog := zlog.Named("startup")
 	if clientID == "" && (clientIDPath == "" || clientSecretPath == "") {
 		startlog.Fatalf("CLIENT_ID_FILE and CLIENT_SECRET_FILE must be set") // TODO(tomhjp): error message can mention WIF once it's publicly available.
 	}
+
 	tsc, err := newTSClient(zlog.Named("ts-api-client"), clientID, clientIDPath, clientSecretPath, loginServer)
 	if err != nil {
 		startlog.Fatalf("error creating Tailscale client: %v", err)
 	}
+
 	s := &tsnet.Server{
 		Hostname:   hostname,
 		Logf:       zlog.Named("tailscaled").Debugf,
 		ControlURL: loginServer,
 	}
+
 	if p := os.Getenv("TS_PORT"); p != "" {
 		port, err := strconv.ParseUint(p, 10, 16)
 		if err != nil {
@@ -207,6 +207,7 @@ func initTSNet(zlog *zap.SugaredLogger, loginServer string) (*tsnet.Server, tsCl
 		}
 		s.Port = uint16(port)
 	}
+
 	if kubeSecret != "" {
 		st, err := kubestore.New(logger.Discard, kubeSecret)
 		if err != nil {
@@ -214,6 +215,7 @@ func initTSNet(zlog *zap.SugaredLogger, loginServer string) (*tsnet.Server, tsCl
 		}
 		s.Store = st
 	}
+
 	if err := s.Start(); err != nil {
 		startlog.Fatalf("starting tailscale server: %v", err)
 	}
@@ -239,27 +241,29 @@ waitOnline:
 			if loginDone {
 				break
 			}
-			caps := tailscale.KeyCapabilities{
+
-				Devices: tailscale.KeyDeviceCapabilities{
+			var caps tailscale.KeyCapabilities
-					Create: tailscale.KeyDeviceCreateCapabilities{
+			caps.Devices.Create.Reusable = false
-						Reusable:      false,
+			caps.Devices.Create.Preauthorized = true
-						Preauthorized: true,
+			caps.Devices.Create.Tags = strings.Split(operatorTags, ",")
-						Tags:          strings.Split(operatorTags, ","),
+
-					},
+			authKey, err := tsc.Keys().CreateAuthKey(ctx, tailscale.CreateKeyRequest{Capabilities: caps})
-				},
-			}
-			authkey, _, err := tsc.CreateKey(ctx, caps)
 			if err != nil {
 				startlog.Fatalf("creating operator authkey: %v", err)
 			}
-			if err := lc.Start(ctx, ipn.Options{
+
-				AuthKey: authkey,
+			opts := ipn.Options{
-			}); err != nil {
+				AuthKey: authKey.Key,
+			}
+
+			if err = lc.Start(ctx, opts); err != nil {
 				startlog.Fatalf("starting tailscale: %v", err)
 			}
-			if err := lc.StartLoginInteractive(ctx); err != nil {
+
+			if err = lc.StartLoginInteractive(ctx); err != nil {
 				startlog.Fatalf("starting login: %v", err)
 			}
+
 			startlog.Debugf("requested login by authkey")
 			loginDone = true
 		case "NeedsMachineAuth":
@@ -286,6 +290,12 @@ func serviceManagedResourceFilterPredicate() predicate.Predicate {
 	})
 }
 
+type (
+	ClientProvider interface {
+		For(tailnet string) (tsclient.Client, error)
+	}
+)
+
 // runReconcilers starts the controller-runtime manager and registers the
 // ServiceReconciler. It blocks forever.
 func runReconcilers(opts reconcilerOpts) {
@@ -334,11 +344,14 @@ func runReconcilers(opts reconcilerOpts) {
 		startlog.Fatalf("could not create manager: %v", err)
 	}
 
+	clients := tsclient.NewProvider(tsclient.Wrap(opts.tsClient))
+
 	tailnetOptions := tailnet.ReconcilerOptions{
 		Client:             mgr.GetClient(),
 		TailscaleNamespace: opts.tailscaleNamespace,
 		Clock:              tstime.DefaultClock{},
 		Logger:             opts.log,
+		Registry:           clients,
 	}
 
 	if err = tailnet.NewReconciler(tailnetOptions).Register(mgr); err != nil {
@@ -368,7 +381,7 @@ func runReconcilers(opts reconcilerOpts) {
 	ssr := &tailscaleSTSReconciler{
 		Client:                 mgr.GetClient(),
 		tsnetServer:            opts.tsServer,
-		tsClient:               opts.tsClient,
+		clients:                clients,
 		defaultTags:            strings.Split(opts.proxyTags, ","),
 		operatorNamespace:      opts.tailscaleNamespace,
 		proxyImage:             opts.proxyImage,
@@ -460,7 +473,7 @@ func runReconcilers(opts reconcilerOpts) {
 		Watches(&tsapi.ProxyGroup{}, ingressProxyGroupFilter).
 		Complete(&HAIngressReconciler{
 			recorder:         eventRecorder,
-			tsClient:         opts.tsClient,
+			clients:          clients,
 			tsnetServer:      opts.tsServer,
 			defaultTags:      strings.Split(opts.proxyTags, ","),
 			Client:           mgr.GetClient(),
@@ -486,7 +499,7 @@ func runReconcilers(opts reconcilerOpts) {
 		Watches(&discoveryv1.EndpointSlice{}, ingressSvcFromEpsFilter).
 		Complete(&HAServiceReconciler{
 			recorder:    eventRecorder,
-			tsClient:    opts.tsClient,
+			clients:     clients,
 			defaultTags: strings.Split(opts.proxyTags, ","),
 			Client:      mgr.GetClient(),
 			logger:      opts.log.Named("service-pg-reconciler"),
@@ -684,8 +697,9 @@ func runReconcilers(opts reconcilerOpts) {
 			Client:            mgr.GetClient(),
 			log:               opts.log.Named("recorder-reconciler"),
 			clock:             tstime.DefaultClock{},
-			tsClient:    opts.tsClient,
+			clients:           clients,
-			loginServer: opts.loginServer,
+			authKeyRateLimits: make(map[string]*rate.Limiter),
+			authKeyReissuing:  make(map[string]bool),
 		})
 	if err != nil {
 		startlog.Fatalf("could not create Recorder reconciler: %v", err)
@@ -706,7 +720,7 @@ func runReconcilers(opts reconcilerOpts) {
 			Client:      mgr.GetClient(),
 			recorder:    eventRecorder,
 			logger:      opts.log.Named("kube-apiserver-ts-service-reconciler"),
-			tsClient:    opts.tsClient,
+			clients:     clients,
 			tsNamespace: opts.tailscaleNamespace,
 			defaultTags: strings.Split(opts.proxyTags, ","),
 			operatorID:  id,
@@ -738,7 +752,7 @@ func runReconcilers(opts reconcilerOpts) {
 			Client:   mgr.GetClient(),
 			log:      opts.log.Named("proxygroup-reconciler"),
 			clock:    tstime.DefaultClock{},
-			tsClient: opts.tsClient,
+			clients:  clients,
 
 			tsNamespace:       opts.tailscaleNamespace,
 			tsProxyImage:      opts.proxyImage,
@@ -763,7 +777,7 @@ func runReconcilers(opts reconcilerOpts) {
 type reconcilerOpts struct {
 	log                *zap.SugaredLogger
 	tsServer           *tsnet.Server
-	tsClient           tsClient
+	tsClient           *tailscale.Client
 	tailscaleNamespace string       // namespace in which operator resources will be deployed
 	restConfig         *rest.Config // config for connecting to the kube API server
 	proxyImage         string       // <proxy-image-repo>:<proxy-image-tag>

cmd/k8s-operator/operator_test.go

@@ -24,8 +24,10 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
 	"tailscale.com/k8s-operator/apis/v1alpha1"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/net/dns/resolvconffile"
 	"tailscale.com/tstest"
@@ -43,7 +45,7 @@ func TestLoadBalancerClass(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -62,7 +64,7 @@ func TestLoadBalancerClass(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationTailnetTargetFQDN: "invalid.example.com",
 			},
@@ -203,7 +205,7 @@ func TestLoadBalancerClass(t *testing.T) {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "test",
 			Namespace: "default",
-			UID:       types.UID("1234-UID"),
+			UID:       "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP: "10.20.30.40",
@@ -223,7 +225,7 @@ func TestTailnetTargetFQDNAnnotation(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -241,7 +243,7 @@ func TestTailnetTargetFQDNAnnotation(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationTailnetTargetFQDN: tailnetTargetFQDN,
 			},
@@ -333,7 +335,7 @@ func TestTailnetTargetIPAnnotation(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -351,7 +353,7 @@ func TestTailnetTargetIPAnnotation(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationTailnetTargetIP: tailnetTargetIP,
 			},
@@ -442,7 +444,7 @@ func TestTailnetTargetIPAnnotation_IPCouldNotBeParsed(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -457,7 +459,7 @@ func TestTailnetTargetIPAnnotation_IPCouldNotBeParsed(t *testing.T) {
 			Name:      "test",
 			Namespace: "default",
 
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationTailnetTargetIP: tailnetTargetIP,
 			},
@@ -510,7 +512,7 @@ func TestTailnetTargetIPAnnotation_InvalidIP(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -525,7 +527,7 @@ func TestTailnetTargetIPAnnotation_InvalidIP(t *testing.T) {
 			Name:      "test",
 			Namespace: "default",
 
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationTailnetTargetIP: tailnetTargetIP,
 			},
@@ -578,7 +580,7 @@ func TestAnnotations(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -596,7 +598,7 @@ func TestAnnotations(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				"tailscale.com/expose": "true",
 			},
@@ -663,7 +665,7 @@ func TestAnnotations(t *testing.T) {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "test",
 			Namespace: "default",
-			UID:       types.UID("1234-UID"),
+			UID:       "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP: "10.20.30.40",
@@ -682,7 +684,7 @@ func TestAnnotationIntoLB(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -700,7 +702,7 @@ func TestAnnotationIntoLB(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				"tailscale.com/expose": "true",
 			},
@@ -779,7 +781,7 @@ func TestAnnotationIntoLB(t *testing.T) {
 			Name:       "test",
 			Namespace:  "default",
 			Finalizers: []string{"tailscale.com/finalizer"},
-			UID:        types.UID("1234-UID"),
+			UID:        "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP:         "10.20.30.40",
@@ -812,7 +814,7 @@ func TestLBIntoAnnotation(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -830,7 +832,7 @@ func TestLBIntoAnnotation(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP:         "10.20.30.40",
@@ -925,7 +927,7 @@ func TestLBIntoAnnotation(t *testing.T) {
 			Annotations: map[string]string{
 				"tailscale.com/expose": "true",
 			},
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP: "10.20.30.40",
@@ -947,7 +949,7 @@ func TestCustomHostname(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -965,7 +967,7 @@ func TestCustomHostname(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				"tailscale.com/expose":   "true",
 				"tailscale.com/hostname": "reindeer-flotilla",
@@ -1034,7 +1036,7 @@ func TestCustomHostname(t *testing.T) {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "test",
 			Namespace: "default",
-			UID:       types.UID("1234-UID"),
+			UID:       "1234-UID",
 			Annotations: map[string]string{
 				"tailscale.com/hostname": "reindeer-flotilla",
 			},
@@ -1056,7 +1058,7 @@ func TestCustomPriorityClassName(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:                 fc,
-			tsClient:               ft,
+			clients:                tsclient.NewProvider(ft),
 			defaultTags:            []string{"tag:k8s"},
 			operatorNamespace:      "operator-ns",
 			proxyImage:             "tailscale/tailscale",
@@ -1075,7 +1077,7 @@ func TestCustomPriorityClassName(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				"tailscale.com/expose":   "true",
 				"tailscale.com/hostname": "tailscale-critical",
@@ -1212,7 +1214,7 @@ func TestServiceProxyClassAnnotation(t *testing.T) {
 				Client: fc,
 				ssr: &tailscaleSTSReconciler{
 					Client:            fc,
-					tsClient:          ft,
+					clients:           tsclient.NewProvider(ft),
 					defaultTags:       []string{"tag:k8s"},
 					operatorNamespace: "operator-ns",
 					proxyImage:        "tailscale/tailscale",
@@ -1308,7 +1310,7 @@ func TestProxyClassForService(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -1326,7 +1328,7 @@ func TestProxyClassForService(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP:         "10.20.30.40",
@@ -1397,7 +1399,7 @@ func TestDefaultLoadBalancer(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -1416,7 +1418,7 @@ func TestDefaultLoadBalancer(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP: "10.20.30.40",
@@ -1451,7 +1453,7 @@ func TestProxyFirewallMode(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -1471,7 +1473,7 @@ func TestProxyFirewallMode(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP: "10.20.30.40",
@@ -1545,7 +1547,7 @@ func Test_HeadlessService(t *testing.T) {
 			Name:      "test",
 			Namespace: "default",
 
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationExpose: "true",
 			},
@@ -1829,7 +1831,7 @@ func Test_authKeyRemoval(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -1842,7 +1844,7 @@ func Test_authKeyRemoval(t *testing.T) {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "test",
 			Namespace: "default",
-			UID:       types.UID("1234-UID"),
+			UID:       "1234-UID",
 		},
 		Spec: corev1.ServiceSpec{
 			ClusterIP:         "10.20.30.40",
@@ -1894,7 +1896,7 @@ func Test_externalNameService(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -1912,7 +1914,7 @@ func Test_externalNameService(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				AnnotationExpose: "true",
 			},
@@ -1988,7 +1990,7 @@ func Test_metricsResourceCreation(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			operatorNamespace: "operator-ns",
 		},
 		logger: zl.Sugar(),
@@ -2059,7 +2061,7 @@ func TestIgnorePGService(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",
@@ -2077,7 +2079,7 @@ func TestIgnorePGService(t *testing.T) {
 			// The apiserver is supposed to set the UID, but the fake client
 			// doesn't. So, set it explicitly because other code later depends
 			// on it being set.
-			UID: types.UID("1234-UID"),
+			UID: "1234-UID",
 			Annotations: map[string]string{
 				"tailscale.com/proxygroup": "test-pg",
 			},

cmd/k8s-operator/proxygroup.go

@@ -10,7 +10,6 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
-	"net/http"
 	"net/netip"
 	"slices"
 	"sort"
@@ -33,11 +32,12 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/client/tailscale"
 	"tailscale.com/ipn"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/egressservices"
 	"tailscale.com/kube/k8s-proxy/conf"
 	"tailscale.com/kube/kubetypes"
@@ -85,7 +85,7 @@ type ProxyGroupReconciler struct {
 	log      *zap.SugaredLogger
 	recorder record.EventRecorder
 	clock    tstime.Clock
-	tsClient tsClient
+	clients  ClientProvider
 
 	// User-specified defaults from the helm installation.
 	tsNamespace       string
@@ -122,7 +122,7 @@ func (r *ProxyGroupReconciler) Reconcile(ctx context.Context, req reconcile.Requ
 		return reconcile.Result{}, fmt.Errorf("failed to get tailscale.com ProxyGroup: %w", err)
 	}
 
-	tailscaleClient, loginUrl, err := r.getClientAndLoginURL(ctx, pg.Spec.Tailnet)
+	tsClient, err := r.clients.For(pg.Spec.Tailnet)
 	if err != nil {
 		oldPGStatus := pg.Status.DeepCopy()
 		nrr := &notReadyReason{
@@ -141,7 +141,7 @@ func (r *ProxyGroupReconciler) Reconcile(ctx context.Context, req reconcile.Requ
 			return reconcile.Result{}, nil
 		}
 
-		if done, err := r.maybeCleanup(ctx, tailscaleClient, pg); err != nil {
+		if done, err := r.maybeCleanup(ctx, tsClient, pg); err != nil {
 			if strings.Contains(err.Error(), optimisticLockErrorMsg) {
 				logger.Infof("optimistic lock error, retrying: %s", err)
 				return reconcile.Result{}, nil
@@ -160,7 +160,7 @@ func (r *ProxyGroupReconciler) Reconcile(ctx context.Context, req reconcile.Requ
 	}
 
 	oldPGStatus := pg.Status.DeepCopy()
-	staticEndpoints, nrr, err := r.reconcilePG(ctx, tailscaleClient, loginUrl, pg, logger)
+	staticEndpoints, nrr, err := r.reconcilePG(ctx, tsClient, pg, logger)
 	return reconcile.Result{}, errors.Join(err, r.maybeUpdateStatus(ctx, logger, pg, oldPGStatus, nrr, staticEndpoints))
 }
 
@@ -168,7 +168,7 @@ func (r *ProxyGroupReconciler) Reconcile(ctx context.Context, req reconcile.Requ
 // for deletion. It is separated out from Reconcile to make a clear separation
 // between reconciling the ProxyGroup, and posting the status of its created
 // resources onto the ProxyGroup status field.
-func (r *ProxyGroupReconciler) reconcilePG(ctx context.Context, tailscaleClient tsClient, loginUrl string, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger) (map[string][]netip.AddrPort, *notReadyReason, error) {
+func (r *ProxyGroupReconciler) reconcilePG(ctx context.Context, tsClient tsclient.Client, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger) (map[string][]netip.AddrPort, *notReadyReason, error) {
 	if !slices.Contains(pg.Finalizers, FinalizerName) {
 		// This log line is printed exactly once during initial provisioning,
 		// because once the finalizer is in place this block gets skipped. So,
@@ -209,7 +209,7 @@ func (r *ProxyGroupReconciler) reconcilePG(ctx context.Context, tailscaleClient
 		return notReady(reasonProxyGroupInvalid, fmt.Sprintf("invalid ProxyGroup spec: %v", err))
 	}
 
-	staticEndpoints, nrr, err := r.maybeProvision(ctx, tailscaleClient, loginUrl, pg, proxyClass)
+	staticEndpoints, nrr, err := r.maybeProvision(ctx, tsClient, pg, proxyClass)
 	if err != nil {
 		return nil, nrr, err
 	}
@@ -295,7 +295,7 @@ func (r *ProxyGroupReconciler) validate(ctx context.Context, pg *tsapi.ProxyGrou
 	return errors.Join(errs...)
 }
 
-func (r *ProxyGroupReconciler) maybeProvision(ctx context.Context, tailscaleClient tsClient, loginUrl string, pg *tsapi.ProxyGroup, proxyClass *tsapi.ProxyClass) (map[string][]netip.AddrPort, *notReadyReason, error) {
+func (r *ProxyGroupReconciler) maybeProvision(ctx context.Context, tsClient tsclient.Client, pg *tsapi.ProxyGroup, proxyClass *tsapi.ProxyClass) (map[string][]netip.AddrPort, *notReadyReason, error) {
 	logger := r.logger(pg.Name)
 	r.mu.Lock()
 	r.ensureStateAddedForProxyGroup(pg)
@@ -317,7 +317,7 @@ func (r *ProxyGroupReconciler) maybeProvision(ctx context.Context, tailscaleClie
 		}
 	}
 
-	staticEndpoints, err := r.ensureConfigSecretsCreated(ctx, tailscaleClient, loginUrl, pg, proxyClass, svcToNodePorts)
+	staticEndpoints, err := r.ensureConfigSecretsCreated(ctx, tsClient, pg, proxyClass, svcToNodePorts)
 	if err != nil {
 		if _, ok := errors.AsType[*FindStaticEndpointErr](err); ok {
 			reason := reasonProxyGroupCreationFailed
@@ -428,7 +428,7 @@ func (r *ProxyGroupReconciler) maybeProvision(ctx context.Context, tailscaleClie
 		return r.notReadyErrf(pg, logger, "error reconciling metrics resources: %w", err)
 	}
 
-	if err := r.cleanupDanglingResources(ctx, tailscaleClient, pg, proxyClass); err != nil {
+	if err := r.cleanupDanglingResources(ctx, tsClient, pg, proxyClass); err != nil {
 		return r.notReadyErrf(pg, logger, "error cleaning up dangling resources: %w", err)
 	}
 
@@ -625,7 +625,7 @@ func (r *ProxyGroupReconciler) ensureNodePortServiceCreated(ctx context.Context,
 
 // cleanupDanglingResources ensures we don't leak config secrets, state secrets, and
 // tailnet devices when the number of replicas specified is reduced.
-func (r *ProxyGroupReconciler) cleanupDanglingResources(ctx context.Context, tailscaleClient tsClient, pg *tsapi.ProxyGroup, pc *tsapi.ProxyClass) error {
+func (r *ProxyGroupReconciler) cleanupDanglingResources(ctx context.Context, tsClient tsclient.Client, pg *tsapi.ProxyGroup, pc *tsapi.ProxyClass) error {
 	logger := r.logger(pg.Name)
 	metadata, err := getNodeMetadata(ctx, pg, r.Client, r.tsNamespace)
 	if err != nil {
@@ -639,7 +639,7 @@ func (r *ProxyGroupReconciler) cleanupDanglingResources(ctx context.Context, tai
 
 		// Dangling resource, delete the config + state Secrets, as well as
 		// deleting the device from the tailnet.
-		if err := r.ensureDeviceDeleted(ctx, tailscaleClient, m.tsID, logger); err != nil {
+		if err := r.ensureDeviceDeleted(ctx, tsClient, m.tsID, logger); err != nil {
 			return err
 		}
 		if err := r.Delete(ctx, m.stateSecret); err != nil && !apierrors.IsNotFound(err) {
@@ -682,7 +682,7 @@ func (r *ProxyGroupReconciler) cleanupDanglingResources(ctx context.Context, tai
 // maybeCleanup just deletes the device from the tailnet. All the kubernetes
 // resources linked to a ProxyGroup will get cleaned up via owner references
 // (which we can use because they are all in the same namespace).
-func (r *ProxyGroupReconciler) maybeCleanup(ctx context.Context, tailscaleClient tsClient, pg *tsapi.ProxyGroup) (bool, error) {
+func (r *ProxyGroupReconciler) maybeCleanup(ctx context.Context, tsClient tsclient.Client, pg *tsapi.ProxyGroup) (bool, error) {
 	logger := r.logger(pg.Name)
 
 	metadata, err := getNodeMetadata(ctx, pg, r.Client, r.tsNamespace)
@@ -691,7 +691,7 @@ func (r *ProxyGroupReconciler) maybeCleanup(ctx context.Context, tailscaleClient
 	}
 
 	for _, m := range metadata {
-		if err := r.ensureDeviceDeleted(ctx, tailscaleClient, m.tsID, logger); err != nil {
+		if err := r.ensureDeviceDeleted(ctx, tsClient, m.tsID, logger); err != nil {
 			return false, err
 		}
 	}
@@ -712,25 +712,23 @@ func (r *ProxyGroupReconciler) maybeCleanup(ctx context.Context, tailscaleClient
 	return true, nil
 }
 
-func (r *ProxyGroupReconciler) ensureDeviceDeleted(ctx context.Context, tailscaleClient tsClient, id tailcfg.StableNodeID, logger *zap.SugaredLogger) error {
+func (r *ProxyGroupReconciler) ensureDeviceDeleted(ctx context.Context, tsClient tsclient.Client, id tailcfg.StableNodeID, logger *zap.SugaredLogger) error {
 	logger.Debugf("deleting device %s from control", string(id))
-	if err := tailscaleClient.DeleteDevice(ctx, string(id)); err != nil {
+	err := tsClient.Devices().Delete(ctx, string(id))
-		if errResp, ok := errors.AsType[tailscale.ErrResponse](err); ok && errResp.Status == http.StatusNotFound {
+	switch {
+	case tailscale.IsNotFound(err):
 		logger.Debugf("device %s not found, likely because it has already been deleted from control", string(id))
-		} else {
+	case err != nil:
 		return fmt.Errorf("error deleting device: %w", err)
 	}
-	} else {
-		logger.Debugf("device %s deleted from control", string(id))
-	}
 
+	logger.Debugf("device %s deleted from control", string(id))
 	return nil
 }
 
 func (r *ProxyGroupReconciler) ensureConfigSecretsCreated(
 	ctx context.Context,
-	tailscaleClient tsClient,
+	tsClient tsclient.Client,
-	loginUrl string,
 	pg *tsapi.ProxyGroup,
 	proxyClass *tsapi.ProxyClass,
 	svcToNodePorts map[string]uint16,
@@ -756,7 +754,7 @@ func (r *ProxyGroupReconciler) ensureConfigSecretsCreated(
 			return nil, err
 		}
 
-		authKey, err := r.getAuthKey(ctx, tailscaleClient, pg, existingCfgSecret, i, logger)
+		authKey, err := r.getAuthKey(ctx, tsClient, pg, existingCfgSecret, i, logger)
 		if err != nil {
 			return nil, err
 		}
@@ -838,8 +836,8 @@ func (r *ProxyGroupReconciler) ensureConfigSecretsCreated(
 				}
 			}
 
-			if loginUrl != "" {
+			if tsClient.LoginURL() != "" {
-				cfg.ServerURL = new(loginUrl)
+				cfg.ServerURL = new(tsClient.LoginURL())
 			}
 
 			if proxyClass != nil && proxyClass.Spec.TailscaleConfig != nil {
@@ -867,7 +865,7 @@ func (r *ProxyGroupReconciler) ensureConfigSecretsCreated(
 				return nil, err
 			}
 
-			configs, err := pgTailscaledConfig(pg, loginUrl, proxyClass, i, authKey, endpoints[nodePortSvcName], existingAdvertiseServices)
+			configs, err := pgTailscaledConfig(pg, tsClient.LoginURL(), proxyClass, i, authKey, endpoints[nodePortSvcName], existingAdvertiseServices)
 			if err != nil {
 				return nil, fmt.Errorf("error creating tailscaled config: %w", err)
 			}
@@ -904,7 +902,7 @@ func (r *ProxyGroupReconciler) ensureConfigSecretsCreated(
 // A new key is created if the config Secret doesn't exist yet, or if the
 // proxy has requested a reissue via its state Secret. An existing key is
 // retained while the device hasn't authed or a reissue is in progress.
-func (r *ProxyGroupReconciler) getAuthKey(ctx context.Context, tailscaleClient tsClient, pg *tsapi.ProxyGroup, existingCfgSecret *corev1.Secret, ordinal int32, logger *zap.SugaredLogger) (*string, error) {
+func (r *ProxyGroupReconciler) getAuthKey(ctx context.Context, tsClient tsclient.Client, pg *tsapi.ProxyGroup, existingCfgSecret *corev1.Secret, ordinal int32, logger *zap.SugaredLogger) (*string, error) {
 	// Get state Secret to check if it's already authed or has requested
 	// a fresh auth key.
 	stateSecret := &corev1.Secret{
@@ -931,7 +929,7 @@ func (r *ProxyGroupReconciler) getAuthKey(ctx context.Context, tailscaleClient t
 
 	if !createAuthKey {
 		var err error
-		createAuthKey, err = r.shouldReissueAuthKey(ctx, tailscaleClient, pg, stateSecret, cfgAuthKey)
+		createAuthKey, err = r.shouldReissueAuthKey(ctx, tsClient, pg, stateSecret, cfgAuthKey)
 		if err != nil {
 			return nil, err
 		}
@@ -945,7 +943,7 @@ func (r *ProxyGroupReconciler) getAuthKey(ctx context.Context, tailscaleClient t
 		if len(tags) == 0 {
 			tags = r.defaultTags
 		}
-		key, err := newAuthKey(ctx, tailscaleClient, tags)
+		key, err := newAuthKey(ctx, tsClient, tags)
 		if err != nil {
 			return nil, err
 		}
@@ -965,7 +963,7 @@ func (r *ProxyGroupReconciler) getAuthKey(ctx context.Context, tailscaleClient t
 // shouldReissueAuthKey returns true if the proxy needs a new auth key. It
 // tracks in-flight reissues via authKeyReissuing to avoid duplicate API calls
 // across reconciles.
-func (r *ProxyGroupReconciler) shouldReissueAuthKey(ctx context.Context, tailscaleClient tsClient, pg *tsapi.ProxyGroup, stateSecret *corev1.Secret, cfgAuthKey *string) (shouldReissue bool, err error) {
+func (r *ProxyGroupReconciler) shouldReissueAuthKey(ctx context.Context, tsClient tsclient.Client, pg *tsapi.ProxyGroup, stateSecret *corev1.Secret, cfgAuthKey *string) (shouldReissue bool, err error) {
 	r.mu.Lock()
 	reissuing := r.authKeyReissuing[stateSecret.Name]
 	r.mu.Unlock()
@@ -1017,7 +1015,7 @@ func (r *ProxyGroupReconciler) shouldReissueAuthKey(ctx context.Context, tailsca
 	r.log.Infof("Proxy failing to auth; attempting cleanup and new key")
 	if tsID := stateSecret.Data[kubetypes.KeyDeviceID]; len(tsID) > 0 {
 		id := tailcfg.StableNodeID(tsID)
-		if err := r.ensureDeviceDeleted(ctx, tailscaleClient, id, r.log); err != nil {
+		if err = r.ensureDeviceDeleted(ctx, tsClient, id, r.log); err != nil {
 			return false, err
 		}
 	}
@@ -1162,6 +1160,9 @@ func (r *ProxyGroupReconciler) ensureStateRemovedForProxyGroup(pg *tsapi.ProxyGr
 	gaugeIngressProxyGroupResources.Set(int64(r.ingressProxyGroups.Len()))
 	gaugeAPIServerProxyGroupResources.Set(int64(r.apiServerProxyGroups.Len()))
 	delete(r.authKeyRateLimits, pg.Name)
+	for i := range pgReplicas(pg) {
+		delete(r.authKeyReissuing, pgStateSecretName(pg.Name, i))
+	}
 }
 
 func pgTailscaledConfig(pg *tsapi.ProxyGroup, loginServer string, pc *tsapi.ProxyClass, idx int32, authKey *string, staticEndpoints []netip.AddrPort, oldAdvertiseServices []string) (tailscaledConfigs, error) {
@@ -1305,29 +1306,6 @@ func (r *ProxyGroupReconciler) getRunningProxies(ctx context.Context, pg *tsapi.
 	return devices, nil
 }
 
-// getClientAndLoginURL returns the appropriate Tailscale client and resolved login URL
-// for the given tailnet name. If no tailnet is specified, returns the default client
-// and login server. Applies fallback to the operator's login server if the tailnet
-// doesn't specify a custom login URL.
-func (r *ProxyGroupReconciler) getClientAndLoginURL(ctx context.Context, tailnetName string) (tsClient,
-	string, error) {
-	if tailnetName == "" {
-		return r.tsClient, r.loginServer, nil
-	}
-
-	tc, loginUrl, err := clientForTailnet(ctx, r.Client, r.tsNamespace, tailnetName)
-	if err != nil {
-		return nil, "", err
-	}
-
-	// Apply fallback if tailnet doesn't specify custom login URL
-	if loginUrl == "" {
-		loginUrl = r.loginServer
-	}
-
-	return tc, loginUrl, nil
-}
-
 type nodeMetadata struct {
 	ordinal     int32
 	stateSecret *corev1.Secret

cmd/k8s-operator/proxygroup_test.go

@@ -30,10 +30,12 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
-	"tailscale.com/client/tailscale"
+	"tailscale.com/client/tailscale/v2"
+
 	"tailscale.com/ipn"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/k8s-proxy/conf"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
@@ -43,7 +45,6 @@ import (
 
 const (
 	testProxyImage = "tailscale/tailscale:test"
-	initialCfgHash = "6632726be70cf224049580deb4d317bba065915b5fd415461d60ed621c91b196"
 )
 
 var (
@@ -641,7 +642,7 @@ func TestProxyGroupWithStaticEndpoints(t *testing.T) {
 				defaultProxyClass: "default-pc",
 
 				Client:            fc,
-				tsClient:          tsClient,
+				clients:           tsclient.NewProvider(tsClient),
 				recorder:          fr,
 				clock:             cl,
 				authKeyRateLimits: make(map[string]*rate.Limiter),
@@ -649,7 +650,7 @@ func TestProxyGroupWithStaticEndpoints(t *testing.T) {
 			}
 
 			for i, r := range tt.reconciles {
-				createdNodes := []corev1.Node{}
+				var createdNodes []corev1.Node
 				t.Run(tt.name, func(t *testing.T) {
 					for _, n := range r.nodes {
 						no := &corev1.Node{
@@ -786,7 +787,7 @@ func TestProxyGroupWithStaticEndpoints(t *testing.T) {
 					defaultProxyClass: "default-pc",
 
 					Client:            fc,
-					tsClient:          tsClient,
+					clients:           tsclient.NewProvider(tsClient),
 					recorder:          fr,
 					log:               zl.Sugar().With("TestName", tt.name).With("Reconcile", "cleanup"),
 					clock:             cl,
@@ -849,7 +850,7 @@ func TestProxyGroup(t *testing.T) {
 		defaultProxyClass: "default-pc",
 
 		Client:            fc,
-		tsClient:          tsClient,
+		clients:           tsclient.NewProvider(tsClient),
 		recorder:          fr,
 		log:               zl.Sugar(),
 		clock:             cl,
@@ -908,17 +909,13 @@ func TestProxyGroup(t *testing.T) {
 			t.Fatalf("expected %d egress ProxyGroups, got %d", expected, reconciler.egressProxyGroups.Len())
 		}
 		expectProxyGroupResources(t, fc, pg, true, pc)
-		keyReq := tailscale.KeyCapabilities{
+		var keyReq tailscale.KeyCapabilities
-			Devices: tailscale.KeyDeviceCapabilities{
+		keyReq.Devices.Create.Reusable = false
-				Create: tailscale.KeyDeviceCreateCapabilities{
+		keyReq.Devices.Create.Ephemeral = false
-					Reusable:      false,
+		keyReq.Devices.Create.Preauthorized = true
-					Ephemeral:     false,
+		keyReq.Devices.Create.Tags = []string{"tag:test-tag"}
-					Preauthorized: true,
+
-					Tags:          []string{"tag:test-tag"},
+		if diff := cmp.Diff(tsClient.keyRequests, []tailscale.KeyCapabilities{keyReq, keyReq}); diff != "" {
-				},
-			},
-		}
-		if diff := cmp.Diff(tsClient.KeyRequests(), []tailscale.KeyCapabilities{keyReq, keyReq}); diff != "" {
 			t.Fatalf("unexpected secrets (-got +want):\n%s", diff)
 		}
 	})
@@ -1059,7 +1056,7 @@ func TestProxyGroupTypes(t *testing.T) {
 		tsProxyImage:      testProxyImage,
 		Client:            fc,
 		log:               zl.Sugar(),
-		tsClient:          &fakeTSClient{},
+		clients:           tsclient.NewProvider(&fakeTSClient{}),
 		clock:             tstest.NewClock(tstest.ClockOpts{}),
 		authKeyRateLimits: make(map[string]*rate.Limiter),
 		authKeyReissuing:  make(map[string]bool),
@@ -1301,7 +1298,7 @@ func TestKubeAPIServerStatusConditionFlow(t *testing.T) {
 		tsProxyImage:      testProxyImage,
 		Client:            fc,
 		log:               zap.Must(zap.NewDevelopment()).Sugar(),
-		tsClient:          &fakeTSClient{},
+		clients:           tsclient.NewProvider(&fakeTSClient{}),
 		clock:             tstest.NewClock(tstest.ClockOpts{}),
 		authKeyRateLimits: make(map[string]*rate.Limiter),
 		authKeyReissuing:  make(map[string]bool),
@@ -1356,7 +1353,7 @@ func TestKubeAPIServerType_DoesNotOverwriteServicesConfig(t *testing.T) {
 		tsProxyImage:      testProxyImage,
 		Client:            fc,
 		log:               zap.Must(zap.NewDevelopment()).Sugar(),
-		tsClient:          &fakeTSClient{},
+		clients:           tsclient.NewProvider(&fakeTSClient{}),
 		clock:             tstest.NewClock(tstest.ClockOpts{}),
 		authKeyRateLimits: make(map[string]*rate.Limiter),
 		authKeyReissuing:  make(map[string]bool),
@@ -1443,7 +1440,7 @@ func TestIngressAdvertiseServicesConfigPreserved(t *testing.T) {
 		tsProxyImage:      testProxyImage,
 		Client:            fc,
 		log:               zap.Must(zap.NewDevelopment()).Sugar(),
-		tsClient:          &fakeTSClient{},
+		clients:           tsclient.NewProvider(&fakeTSClient{}),
 		clock:             tstest.NewClock(tstest.ClockOpts{}),
 		authKeyRateLimits: make(map[string]*rate.Limiter),
 		authKeyReissuing:  make(map[string]bool),
@@ -1713,7 +1710,7 @@ func TestProxyGroupGetAuthKey(t *testing.T) {
 			tsFirewallMode: "auto",
 
 			Client:            fc,
-			tsClient:          tsClient,
+			clients:           tsclient.NewProvider(tsClient),
 			recorder:          fr,
 			log:               zl.Sugar(),
 			clock:             cl,
@@ -2109,7 +2106,7 @@ func TestProxyGroupLetsEncryptStaging(t *testing.T) {
 				defaultTags:       []string{"tag:test"},
 				defaultProxyClass: tt.defaultProxyClass,
 				Client:            fc,
-				tsClient:          &fakeTSClient{},
+				clients:           tsclient.NewProvider(&fakeTSClient{}),
 				log:               zl.Sugar(),
 				clock:             cl,
 				authKeyRateLimits: make(map[string]*rate.Limiter),

cmd/k8s-operator/sts.go

@@ -12,7 +12,6 @@ import (
 	"errors"
 	"fmt"
 	"maps"
-	"net/http"
 	"os"
 	"path"
 	"slices"
@@ -30,11 +29,12 @@ import (
 	"k8s.io/apiserver/pkg/storage/names"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/yaml"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/client/tailscale"
 	"tailscale.com/ipn"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/net/netutil"
 	"tailscale.com/tailcfg"
@@ -174,7 +174,7 @@ type tsnetServer interface {
 type tailscaleSTSReconciler struct {
 	client.Client
 	tsnetServer            tsnetServer
-	tsClient               tsClient
+	clients                ClientProvider
 	defaultTags            []string
 	operatorNamespace      string
 	proxyImage             string
@@ -183,9 +183,9 @@ type tailscaleSTSReconciler struct {
 	loginServer            string
 }
 
-func (sts tailscaleSTSReconciler) validate() error {
+func (r *tailscaleSTSReconciler) validate() error {
-	if sts.tsFirewallMode != "" && !isValidFirewallMode(sts.tsFirewallMode) {
+	if r.tsFirewallMode != "" && !isValidFirewallMode(r.tsFirewallMode) {
-		return fmt.Errorf("invalid proxy firewall mode %s, valid modes are iptables, nftables or unset", sts.tsFirewallMode)
+		return fmt.Errorf("invalid proxy firewall mode %s, valid modes are iptables, nftables or unset", r.tsFirewallMode)
 	}
 	return nil
 }
@@ -197,22 +197,17 @@ func IsHTTPSEnabledOnTailnet(tsnetServer tsnetServer) bool {
 
 // Provision ensures that the StatefulSet for the given service is running and
 // up to date.
-func (a *tailscaleSTSReconciler) Provision(ctx context.Context, logger *zap.SugaredLogger, sts *tailscaleSTSConfig) (*corev1.Service, error) {
+func (r *tailscaleSTSReconciler) Provision(ctx context.Context, logger *zap.SugaredLogger, sts *tailscaleSTSConfig) (*corev1.Service, error) {
-	tailscaleClient, loginUrl, err := a.getClientAndLoginURL(ctx, sts.Tailnet)
-	if err != nil {
-		return nil, fmt.Errorf("failed to get tailscale client and loginUrl: %w", err)
-	}
-
 	// Do full reconcile.
 	// TODO (don't create Service for the Connector)
-	hsvc, err := a.reconcileHeadlessService(ctx, logger, sts)
+	hsvc, err := r.reconcileHeadlessService(ctx, logger, sts)
 	if err != nil {
 		return nil, fmt.Errorf("failed to reconcile headless service: %w", err)
 	}
 
 	proxyClass := new(tsapi.ProxyClass)
 	if sts.ProxyClassName != "" {
-		if err := a.Get(ctx, types.NamespacedName{Name: sts.ProxyClassName}, proxyClass); err != nil {
+		if err := r.Get(ctx, types.NamespacedName{Name: sts.ProxyClassName}, proxyClass); err != nil {
 			return nil, fmt.Errorf("failed to get ProxyClass: %w", err)
 		}
 		if !tsoperator.ProxyClassIsReady(proxyClass) {
@@ -222,12 +217,17 @@ func (a *tailscaleSTSReconciler) Provision(ctx context.Context, logger *zap.Suga
 	}
 	sts.ProxyClass = proxyClass
 
-	secretNames, err := a.provisionSecrets(ctx, tailscaleClient, loginUrl, sts, hsvc, logger)
+	tsClient, err := r.clients.For(sts.Tailnet)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get tailscale client: %w", err)
+	}
+
+	secretNames, err := r.provisionSecrets(ctx, tsClient, sts, hsvc, logger)
 	if err != nil {
 		return nil, fmt.Errorf("failed to create or get API key secret: %w", err)
 	}
 
-	_, err = a.reconcileSTS(ctx, logger, sts, hsvc, secretNames)
+	_, err = r.reconcileSTS(ctx, logger, sts, hsvc, secretNames)
 	if err != nil {
 		return nil, fmt.Errorf("failed to reconcile statefulset: %w", err)
 	}
@@ -237,57 +237,29 @@ func (a *tailscaleSTSReconciler) Provision(ctx context.Context, logger *zap.Suga
 		proxyLabels:  hsvc.Labels,
 		proxyType:    sts.proxyType,
 	}
-	if err = reconcileMetricsResources(ctx, logger, mo, sts.ProxyClass, a.Client); err != nil {
+	if err = reconcileMetricsResources(ctx, logger, mo, sts.ProxyClass, r.Client); err != nil {
 		return nil, fmt.Errorf("failed to ensure metrics resources: %w", err)
 	}
 	return hsvc, nil
 }
 
-// getClientAndLoginURL returns the appropriate Tailscale client and resolved login URL
-// for the given tailnet name. If no tailnet is specified, returns the default client
-// and login server. Applies fallback to the operator's login server if the tailnet
-// doesn't specify a custom login URL.
-func (a *tailscaleSTSReconciler) getClientAndLoginURL(ctx context.Context, tailnetName string) (tsClient,
-	string, error) {
-	if tailnetName == "" {
-		return a.tsClient, a.loginServer, nil
-	}
-
-	tc, loginUrl, err := clientForTailnet(ctx, a.Client, a.operatorNamespace, tailnetName)
-	if err != nil {
-		return nil, "", err
-	}
-
-	// Apply fallback if tailnet doesn't specify custom login URL
-	if loginUrl == "" {
-		loginUrl = a.loginServer
-	}
-
-	return tc, loginUrl, nil
-}
-
 // Cleanup removes all resources associated that were created by Provision with
 // the given labels. It returns true when all resources have been removed,
 // otherwise it returns false and the caller should retry later.
-func (a *tailscaleSTSReconciler) Cleanup(ctx context.Context, tailnet string, logger *zap.SugaredLogger, labels map[string]string, typ string) (done bool, _ error) {
+func (r *tailscaleSTSReconciler) Cleanup(ctx context.Context, tailnet string, logger *zap.SugaredLogger, labels map[string]string, typ string) (done bool, _ error) {
-	tailscaleClient := a.tsClient
+	tsClient, err := r.clients.For(tailnet)
-	if tailnet != "" {
-		tc, _, err := clientForTailnet(ctx, a.Client, a.operatorNamespace, tailnet)
 	if err != nil {
 		logger.Errorf("failed to get tailscale client: %v", err)
 		return false, nil
 	}
 
-		tailscaleClient = tc
-	}
-
 	// Need to delete the StatefulSet first, and delete it with foreground
 	// cascading deletion. That way, the pod that's writing to the Secret will
 	// stop running before we start looking at the Secret's contents, and
 	// assuming k8s ordering semantics don't mess with us, that should avoid
 	// tailscale device deletion races where we fail to notice a device that
 	// should be removed.
-	sts, err := getSingleObject[appsv1.StatefulSet](ctx, a.Client, a.operatorNamespace, labels)
+	sts, err := getSingleObject[appsv1.StatefulSet](ctx, r.Client, r.operatorNamespace, labels)
 	if err != nil {
 		return false, fmt.Errorf("getting statefulset: %w", err)
 	}
@@ -301,12 +273,12 @@ func (a *tailscaleSTSReconciler) Cleanup(ctx context.Context, tailnet string, lo
 		}
 
 		options := []client.DeleteAllOfOption{
-			client.InNamespace(a.operatorNamespace),
+			client.InNamespace(r.operatorNamespace),
 			client.MatchingLabels(labels),
 			client.PropagationPolicy(metav1.DeletePropagationForeground),
 		}
 
-		if err = a.DeleteAllOf(ctx, &appsv1.StatefulSet{}, options...); err != nil {
+		if err = r.DeleteAllOf(ctx, &appsv1.StatefulSet{}, options...); err != nil {
 			return false, fmt.Errorf("deleting statefulset: %w", err)
 		}
 
@@ -314,7 +286,7 @@ func (a *tailscaleSTSReconciler) Cleanup(ctx context.Context, tailnet string, lo
 		return false, nil
 	}
 
-	devices, err := a.DeviceInfo(ctx, labels, logger)
+	devices, err := r.DeviceInfo(ctx, labels, logger)
 	if err != nil {
 		return false, fmt.Errorf("getting device info: %w", err)
 	}
@@ -322,33 +294,36 @@ func (a *tailscaleSTSReconciler) Cleanup(ctx context.Context, tailnet string, lo
 	for _, dev := range devices {
 		if dev.id != "" {
 			logger.Debugf("deleting device %s from control", string(dev.id))
-			if err = tailscaleClient.DeleteDevice(ctx, string(dev.id)); err != nil {
+			err = tsClient.Devices().Delete(ctx, string(dev.id))
-				if errResp, ok := errors.AsType[tailscale.ErrResponse](err); ok && errResp.Status == http.StatusNotFound {
+			switch {
+			case tailscale.IsNotFound(err):
 				logger.Debugf("device %s not found, likely because it has already been deleted from control", string(dev.id))
-				} else {
+			case err != nil:
 				return false, fmt.Errorf("deleting device: %w", err)
 			}
-			} else {
+
 			logger.Debugf("device %s deleted from control", string(dev.id))
 		}
 	}
-	}
 
-	types := []client.Object{
+	resourceTypes := []client.Object{
 		&corev1.Service{},
 		&corev1.Secret{},
 	}
-	for _, typ := range types {
+
-		if err := a.DeleteAllOf(ctx, typ, client.InNamespace(a.operatorNamespace), client.MatchingLabels(labels)); err != nil {
+	for _, resourceType := range resourceTypes {
+		if err = r.DeleteAllOf(ctx, resourceType, client.InNamespace(r.operatorNamespace), client.MatchingLabels(labels)); err != nil {
 			return false, err
 		}
 	}
+
 	mo := &metricsOpts{
 		proxyLabels: labels,
-		tsNamespace: a.operatorNamespace,
+		tsNamespace: r.operatorNamespace,
 		proxyType:   typ,
 	}
-	if err = maybeCleanupMetricsResources(ctx, mo, a.Client); err != nil {
+
+	if err = maybeCleanupMetricsResources(ctx, mo, r.Client); err != nil {
 		return false, fmt.Errorf("error cleaning up metrics resources: %w", err)
 	}
 
@@ -382,12 +357,12 @@ func statefulSetNameBase(parent string) string {
 	}
 }
 
-func (a *tailscaleSTSReconciler) reconcileHeadlessService(ctx context.Context, logger *zap.SugaredLogger, sts *tailscaleSTSConfig) (*corev1.Service, error) {
+func (r *tailscaleSTSReconciler) reconcileHeadlessService(ctx context.Context, logger *zap.SugaredLogger, sts *tailscaleSTSConfig) (*corev1.Service, error) {
 	nameBase := statefulSetNameBase(sts.ParentResourceName)
 	hsvc := &corev1.Service{
 		ObjectMeta: metav1.ObjectMeta{
 			GenerateName: nameBase,
-			Namespace:    a.operatorNamespace,
+			Namespace:    r.operatorNamespace,
 			Labels:       sts.ChildResourceLabels,
 		},
 		Spec: corev1.ServiceSpec{
@@ -399,10 +374,10 @@ func (a *tailscaleSTSReconciler) reconcileHeadlessService(ctx context.Context, l
 		},
 	}
 	logger.Debugf("reconciling headless service for StatefulSet")
-	return createOrUpdate(ctx, a.Client, a.operatorNamespace, hsvc, func(svc *corev1.Service) { svc.Spec = hsvc.Spec })
+	return createOrUpdate(ctx, r.Client, r.operatorNamespace, hsvc, func(svc *corev1.Service) { svc.Spec = hsvc.Spec })
 }
 
-func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscaleClient tsClient, loginUrl string, stsC *tailscaleSTSConfig, hsvc *corev1.Service, logger *zap.SugaredLogger) ([]string, error) {
+func (r *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tsClient tsclient.Client, stsC *tailscaleSTSConfig, hsvc *corev1.Service, logger *zap.SugaredLogger) ([]string, error) {
 	secretNames := make([]string, stsC.Replicas)
 
 	// Start by ensuring we have Secrets for the desired number of replicas. This will handle both creating and scaling
@@ -411,7 +386,7 @@ func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscale
 		secret := &corev1.Secret{
 			ObjectMeta: metav1.ObjectMeta{
 				Name:      fmt.Sprintf("%s-%d", hsvc.Name, i),
-				Namespace: a.operatorNamespace,
+				Namespace: r.operatorNamespace,
 				Labels:    stsC.ChildResourceLabels,
 			},
 		}
@@ -426,7 +401,7 @@ func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscale
 		secretNames[i] = secret.Name
 
 		var orig *corev1.Secret // unmodified copy of secret
-		if err := a.Get(ctx, client.ObjectKeyFromObject(secret), secret); err == nil {
+		if err := r.Get(ctx, client.ObjectKeyFromObject(secret), secret); err == nil {
 			logger.Debugf("secret %s/%s already exists", secret.GetNamespace(), secret.GetName())
 			orig = secret.DeepCopy()
 		} else if !apierrors.IsNotFound(err) {
@@ -437,21 +412,23 @@ func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscale
 			authKey string
 			err     error
 		)
+
 		if orig == nil {
 			// Create API Key secret which is going to be used by the statefulset
 			// to authenticate with Tailscale.
 			logger.Debugf("creating authkey for new tailscale proxy")
 			tags := stsC.Tags
 			if len(tags) == 0 {
-				tags = a.defaultTags
+				tags = r.defaultTags
 			}
-			authKey, err = newAuthKey(ctx, tailscaleClient, tags)
+
+			authKey, err = newAuthKey(ctx, tsClient, tags)
 			if err != nil {
 				return nil, err
 			}
 		}
 
-		configs, err := tailscaledConfig(stsC, loginUrl, authKey, orig, hostname)
+		configs, err := tailscaledConfig(stsC, tsClient.LoginURL(), authKey, orig, hostname)
 		if err != nil {
 			return nil, fmt.Errorf("error creating tailscaled config: %w", err)
 		}
@@ -483,12 +460,12 @@ func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscale
 
 		if orig != nil && !apiequality.Semantic.DeepEqual(latest, orig) {
 			logger.With("config", sanitizeConfig(latestConfig)).Debugf("patching the existing proxy Secret")
-			if err = a.Patch(ctx, secret, client.MergeFrom(orig)); err != nil {
+			if err = r.Patch(ctx, secret, client.MergeFrom(orig)); err != nil {
 				return nil, err
 			}
 		} else {
 			logger.With("config", sanitizeConfig(latestConfig)).Debugf("creating a new Secret for the proxy")
-			if err = a.Create(ctx, secret); err != nil {
+			if err = r.Create(ctx, secret); err != nil {
 				return nil, err
 			}
 		}
@@ -497,7 +474,7 @@ func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscale
 	// Next, we check if we have additional secrets and remove them and their associated device. This happens when we
 	// scale an StatefulSet down.
 	var secrets corev1.SecretList
-	if err := a.List(ctx, &secrets, client.InNamespace(a.operatorNamespace), client.MatchingLabels(stsC.ChildResourceLabels)); err != nil {
+	if err := r.List(ctx, &secrets, client.InNamespace(r.operatorNamespace), client.MatchingLabels(stsC.ChildResourceLabels)); err != nil {
 		return nil, err
 	}
 
@@ -517,16 +494,14 @@ func (a *tailscaleSTSReconciler) provisionSecrets(ctx context.Context, tailscale
 		}
 
 		if dev != nil && dev.id != "" {
-			err = tailscaleClient.DeleteDevice(ctx, string(dev.id))
+			// If we get a not found error then this device has possibly already been deleted in the admin console.
-			if errResp, ok := errors.AsType[*tailscale.ErrResponse](err); ok && errResp.Status == http.StatusNotFound {
+			// So we can ignore this and move on to removing the secret.
-				// This device has possibly already been deleted in the admin console. So we can ignore this
+			if err = tsClient.Devices().Delete(ctx, string(dev.id)); err != nil && !tailscale.IsNotFound(err) {
-				// and move on to removing the secret.
-			} else if err != nil {
 				return nil, err
 			}
 		}
 
-		if err = a.Delete(ctx, &secret); err != nil {
+		if err = r.Delete(ctx, &secret); err != nil {
 			return nil, err
 		}
 	}
@@ -550,9 +525,9 @@ func sanitizeConfig(c ipn.ConfigVAlpha) ipn.ConfigVAlpha {
 // It retrieves info from a Kubernetes Secret labeled with the provided labels. Capver is cross-validated against the
 // Pod to ensure that it is the currently running Pod that set the capver. If the Pod or the Secret does not exist, the
 // returned capver is -1. Either of device ID, hostname and IPs can be empty string if not found in the Secret.
-func (a *tailscaleSTSReconciler) DeviceInfo(ctx context.Context, childLabels map[string]string, logger *zap.SugaredLogger) ([]*device, error) {
+func (r *tailscaleSTSReconciler) DeviceInfo(ctx context.Context, childLabels map[string]string, logger *zap.SugaredLogger) ([]*device, error) {
 	var secrets corev1.SecretList
-	if err := a.List(ctx, &secrets, client.InNamespace(a.operatorNamespace), client.MatchingLabels(childLabels)); err != nil {
+	if err := r.List(ctx, &secrets, client.InNamespace(r.operatorNamespace), client.MatchingLabels(childLabels)); err != nil {
 		return nil, err
 	}
 
@@ -560,7 +535,7 @@ func (a *tailscaleSTSReconciler) DeviceInfo(ctx context.Context, childLabels map
 	for _, sec := range secrets.Items {
 		podUID := ""
 		pod := new(corev1.Pod)
-		err := a.Get(ctx, types.NamespacedName{Namespace: sec.Namespace, Name: sec.Name}, pod)
+		err := r.Get(ctx, types.NamespacedName{Namespace: sec.Namespace, Name: sec.Name}, pod)
 		switch {
 		case apierrors.IsNotFound(err):
 			// If the Pod is not found, we won't have its UID. We can still get the device information but the
@@ -633,22 +608,18 @@ func deviceInfo(sec *corev1.Secret, podUID string, log *zap.SugaredLogger) (dev
 	return dev, nil
 }
 
-func newAuthKey(ctx context.Context, tsClient tsClient, tags []string) (string, error) {
+func newAuthKey(ctx context.Context, client tsclient.Client, tags []string) (string, error) {
-	caps := tailscale.KeyCapabilities{
+	var caps tailscale.KeyCapabilities
-		Devices: tailscale.KeyDeviceCapabilities{
+	caps.Devices.Create.Reusable = false
-			Create: tailscale.KeyDeviceCreateCapabilities{
+	caps.Devices.Create.Preauthorized = true
-				Reusable:      false,
+	caps.Devices.Create.Tags = tags
-				Preauthorized: true,
-				Tags:          tags,
-			},
-		},
-	}
 
-	key, _, err := tsClient.CreateKey(ctx, caps)
+	key, err := client.Keys().CreateAuthKey(ctx, tailscale.CreateKeyRequest{Capabilities: caps})
 	if err != nil {
 		return "", err
 	}
-	return key, nil
+
+	return key.Key, nil
 }
 
 //go:embed deploy/manifests/proxy.yaml
@@ -657,7 +628,7 @@ var proxyYaml []byte
 //go:embed deploy/manifests/userspace-proxy.yaml
 var userspaceProxyYaml []byte
 
-func (a *tailscaleSTSReconciler) reconcileSTS(ctx context.Context, logger *zap.SugaredLogger, sts *tailscaleSTSConfig, headlessSvc *corev1.Service, proxySecrets []string) (*appsv1.StatefulSet, error) {
+func (r *tailscaleSTSReconciler) reconcileSTS(ctx context.Context, logger *zap.SugaredLogger, sts *tailscaleSTSConfig, headlessSvc *corev1.Service, proxySecrets []string) (*appsv1.StatefulSet, error) {
 	ss := new(appsv1.StatefulSet)
 	if sts.ServeConfig != nil && sts.ForwardClusterTrafficViaL7IngressProxy != true { // If forwarding cluster traffic via is required we need non-userspace + NET_ADMIN + forwarding
 		if err := yaml.Unmarshal(userspaceProxyYaml, &ss); err != nil {
@@ -670,17 +641,17 @@ func (a *tailscaleSTSReconciler) reconcileSTS(ctx context.Context, logger *zap.S
 		for i := range ss.Spec.Template.Spec.InitContainers {
 			c := &ss.Spec.Template.Spec.InitContainers[i]
 			if c.Name == "sysctler" {
-				c.Image = a.proxyImage
+				c.Image = r.proxyImage
 				break
 			}
 		}
 	}
 	pod := &ss.Spec.Template
 	container := &pod.Spec.Containers[0]
-	container.Image = a.proxyImage
+	container.Image = r.proxyImage
 	ss.ObjectMeta = metav1.ObjectMeta{
 		Name:      headlessSvc.Name,
-		Namespace: a.operatorNamespace,
+		Namespace: r.operatorNamespace,
 	}
 	for key, val := range sts.ChildResourceLabels {
 		mak.Set(&ss.ObjectMeta.Labels, key, val)
@@ -748,13 +719,13 @@ func (a *tailscaleSTSReconciler) reconcileSTS(ctx context.Context, logger *zap.S
 		})
 	}
 
-	if a.tsFirewallMode != "" {
+	if r.tsFirewallMode != "" {
 		container.Env = append(container.Env, corev1.EnvVar{
 			Name:  "TS_DEBUG_FIREWALL_MODE",
-			Value: a.tsFirewallMode,
+			Value: r.tsFirewallMode,
 		})
 	}
-	pod.Spec.PriorityClassName = a.proxyPriorityClassName
+	pod.Spec.PriorityClassName = r.proxyPriorityClassName
 
 	// Ingress/egress proxy configuration options.
 	if sts.ClusterTargetIP != "" {
@@ -829,7 +800,7 @@ func (a *tailscaleSTSReconciler) reconcileSTS(ctx context.Context, logger *zap.S
 		s.ObjectMeta.Labels = ss.Labels
 		s.ObjectMeta.Annotations = ss.Annotations
 	}
-	return createOrUpdate(ctx, a.Client, a.operatorNamespace, ss, updateSS)
+	return createOrUpdate(ctx, r.Client, r.operatorNamespace, ss, updateSS)
 }
 
 func appInfoForProxy(cfg *tailscaleSTSConfig) (string, error) {

cmd/k8s-operator/svc-for-pg.go

@@ -10,7 +10,6 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
-	"net/http"
 	"net/netip"
 	"reflect"
 	"slices"
@@ -27,11 +26,12 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/internal/client/tailscale"
 	"tailscale.com/ipn"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/ingressservices"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
@@ -57,7 +57,7 @@ type HAServiceReconciler struct {
 	isDefaultLoadBalancer bool
 	recorder              record.EventRecorder
 	logger                *zap.SugaredLogger
-	tsClient              tsClient
+	clients               ClientProvider
 	tsNamespace           string
 	defaultTags           []string
 	operatorID            string // stableID of the operator's Tailscale device
@@ -121,7 +121,7 @@ func (r *HAServiceReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 		return res, nil
 	}
 
-	tailscaleClient, err := clientFromProxyGroup(ctx, r.Client, pg, r.tsNamespace, r.tsClient)
+	tsClient, err := r.clients.For(pg.Spec.Tailnet)
 	if err != nil {
 		return res, fmt.Errorf("failed to get tailscale client: %w", err)
 	}
@@ -131,7 +131,7 @@ func (r *HAServiceReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 
 	if !svc.DeletionTimestamp.IsZero() || !r.isTailscaleService(svc) {
 		logger.Debugf("Service is being deleted or is (no longer) referring to Tailscale ingress/egress, ensuring any created resources are cleaned up")
-		_, err = r.maybeCleanup(ctx, hostname, svc, logger, tailscaleClient)
+		_, err = r.maybeCleanup(ctx, hostname, svc, logger, tsClient)
 		return res, err
 	}
 
@@ -139,7 +139,7 @@ func (r *HAServiceReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 	// is the case, we reconcile the Ingress one more time to ensure that concurrent updates to the Tailscale Service in a
 	// multi-cluster Ingress setup have not resulted in another actor overwriting our Tailscale Service update.
 	needsRequeue := false
-	needsRequeue, err = r.maybeProvision(ctx, hostname, svc, pg, logger, tailscaleClient)
+	needsRequeue, err = r.maybeProvision(ctx, hostname, svc, pg, logger, tsClient)
 	if err != nil {
 		if strings.Contains(err.Error(), optimisticLockErrorMsg) {
 			logger.Infof("optimistic lock error, retrying: %s", err)
@@ -162,7 +162,7 @@ func (r *HAServiceReconciler) Reconcile(ctx context.Context, req reconcile.Reque
 // If a Tailscale Service exists, but does not have an owner reference from any operator, we error
 // out assuming that this is an owner reference created by an unknown actor.
 // Returns true if the operation resulted in a Tailscale Service update.
-func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname string, svc *corev1.Service, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsClient) (svcsChanged bool, err error) {
+func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname string, svc *corev1.Service, pg *tsapi.ProxyGroup, logger *zap.SugaredLogger, tsClient tsclient.Client) (svcsChanged bool, err error) {
 	oldSvcStatus := svc.Status.DeepCopy()
 	defer func() {
 		if !apiequality.Semantic.DeepEqual(oldSvcStatus, &svc.Status) {
@@ -209,8 +209,8 @@ func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname strin
 	// 2. Ensure that there isn't a Tailscale Service with the same hostname
 	// already created and not owned by this Service.
 	serviceName := tailcfg.ServiceName("svc:" + hostname)
-	existingTSSvc, err := tsClient.GetVIPService(ctx, serviceName)
+	existingTSSvc, err := tsClient.VIPServices().Get(ctx, serviceName.String())
-	if err != nil && !isErrorTailscaleServiceNotFound(err) {
+	if err != nil && !tailscale.IsNotFound(err) {
 		return false, fmt.Errorf("error getting Tailscale Service %q: %w", hostname, err)
 	}
 
@@ -233,8 +233,8 @@ func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname strin
 		tags = strings.Split(tstr, ",")
 	}
 
-	tsSvc := &tailscale.VIPService{
+	tsSvc := tailscale.VIPService{
-		Name:        serviceName,
+		Name:        serviceName.String(),
 		Tags:        tags,
 		Ports:       []string{"do-not-validate"}, // we don't want to validate ports
 		Comment:     managedTSServiceComment,
@@ -249,12 +249,13 @@ func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname strin
 	// with the same generation number has been reconciled ~more than N times and stop attempting to apply updates.
 	if existingTSSvc == nil ||
 		!reflect.DeepEqual(tsSvc.Tags, existingTSSvc.Tags) ||
-		!ownersAreSetAndEqual(tsSvc, existingTSSvc) {
+		!ownersAreSetAndEqual(tsSvc, *existingTSSvc) {
 		logger.Infof("Ensuring Tailscale Service exists and is up to date")
-		if err := tsClient.CreateOrUpdateVIPService(ctx, tsSvc); err != nil {
+		if err = tsClient.VIPServices().CreateOrUpdate(ctx, tsSvc); err != nil {
 			return false, fmt.Errorf("error creating Tailscale Service: %w", err)
 		}
-		existingTSSvc = tsSvc
+
+		existingTSSvc = &tsSvc
 	}
 
 	cm, cfgs, err := ingressSvcsConfigs(ctx, r.Client, pg.Name, r.tsNamespace)
@@ -266,12 +267,12 @@ func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname strin
 		return false, nil
 	}
 
-	if existingTSSvc.Addrs == nil {
+	if len(existingTSSvc.Addrs) == 0 {
-		existingTSSvc, err = tsClient.GetVIPService(ctx, tsSvc.Name)
+		existingTSSvc, err = tsClient.VIPServices().Get(ctx, tsSvc.Name)
-		if err != nil {
+		switch {
+		case err != nil:
 			return false, fmt.Errorf("error getting Tailscale Service: %w", err)
-		}
+		case len(existingTSSvc.Addrs) == 0:
-		if existingTSSvc.Addrs == nil {
 			// TODO(irbekrm): this should be a retry
 			return false, fmt.Errorf("unexpected: Tailscale Service addresses not populated")
 		}
@@ -374,7 +375,7 @@ func (r *HAServiceReconciler) maybeProvision(ctx context.Context, hostname strin
 // Service is being deleted or is unexposed. The cleanup is safe for a multi-cluster setup- the Tailscale Service is only
 // deleted if it does not contain any other owner references. If it does the cleanup only removes the owner reference
 // corresponding to this Service.
-func (r *HAServiceReconciler) maybeCleanup(ctx context.Context, hostname string, svc *corev1.Service, logger *zap.SugaredLogger, tsClient tsClient) (svcChanged bool, err error) {
+func (r *HAServiceReconciler) maybeCleanup(ctx context.Context, hostname string, svc *corev1.Service, logger *zap.SugaredLogger, tsClient tsclient.Client) (svcChanged bool, err error) {
 	logger.Debugf("Ensuring any resources for Service are cleaned up")
 	ix := slices.Index(svc.Finalizers, svcPGFinalizerName)
 	if ix < 0 {
@@ -392,7 +393,7 @@ func (r *HAServiceReconciler) maybeCleanup(ctx context.Context, hostname string,
 
 	serviceName := tailcfg.ServiceName("svc:" + hostname)
 	//  1. Clean up the Tailscale Service.
-	svcChanged, err = cleanupTailscaleService(ctx, tsClient, serviceName, r.operatorID, logger)
+	svcChanged, err = cleanupTailscaleService(ctx, tsClient, serviceName.String(), r.operatorID, logger)
 	if err != nil {
 		return false, fmt.Errorf("error deleting Tailscale Service: %w", err)
 	}
@@ -425,7 +426,7 @@ func (r *HAServiceReconciler) maybeCleanup(ctx context.Context, hostname string,
 
 // Tailscale Services that are associated with the provided ProxyGroup and no longer managed this operator's instance are deleted, if not owned by other operator instances, else the owner reference is cleaned up.
 // Returns true if the operation resulted in existing Tailscale Service updates (owner reference removal).
-func (r *HAServiceReconciler) maybeCleanupProxyGroup(ctx context.Context, proxyGroupName string, logger *zap.SugaredLogger, tsClient tsClient) (svcsChanged bool, err error) {
+func (r *HAServiceReconciler) maybeCleanupProxyGroup(ctx context.Context, proxyGroupName string, logger *zap.SugaredLogger, tsClient tsclient.Client) (svcsChanged bool, err error) {
 	cm, config, err := ingressSvcsConfigs(ctx, r.Client, proxyGroupName, r.tsNamespace)
 	if err != nil {
 		return false, fmt.Errorf("failed to get ingress service config: %s", err)
@@ -453,7 +454,7 @@ func (r *HAServiceReconciler) maybeCleanupProxyGroup(ctx context.Context, proxyG
 				return false, fmt.Errorf("failed to update tailscaled config services: %w", err)
 			}
 
-			svcsChanged, err = cleanupTailscaleService(ctx, tsClient, tailcfg.ServiceName(tsSvcName), r.operatorID, logger)
+			svcsChanged, err = cleanupTailscaleService(ctx, tsClient, tsSvcName, r.operatorID, logger)
 			if err != nil {
 				return false, fmt.Errorf("deleting Tailscale Service %q: %w", tsSvcName, err)
 			}
@@ -517,29 +518,28 @@ func (r *HAServiceReconciler) shouldExposeClusterIP(svc *corev1.Service) bool {
 // If a Tailscale Service is found, but contains other owner references, only removes this operator's owner reference.
 // If a Tailscale Service by the given name is not found or does not contain this operator's owner reference, do nothing.
 // It returns true if an existing Tailscale Service was updated to remove owner reference, as well as any error that occurred.
-func cleanupTailscaleService(ctx context.Context, tsClient tsClient, name tailcfg.ServiceName, operatorID string, logger *zap.SugaredLogger) (updated bool, err error) {
+func cleanupTailscaleService(ctx context.Context, tsClient tsclient.Client, name string, operatorID string, logger *zap.SugaredLogger) (updated bool, err error) {
-	svc, err := tsClient.GetVIPService(ctx, name)
+	svc, err := tsClient.VIPServices().Get(ctx, name)
-	if err != nil {
+	switch {
-		errResp, ok := errors.AsType[tailscale.ErrResponse](err)
+	case tailscale.IsNotFound(err):
-		if ok && errResp.Status == http.StatusNotFound {
 		return false, nil
-		}
+	case err != nil:
-		if !ok {
+		return false, fmt.Errorf("unexpected error getting Tailscale Service %q: %w", name, err)
-			return false, fmt.Errorf("unexpected error getting Tailscale Service %q: %w", name.String(), err)
 	}
 
-		return false, fmt.Errorf("error getting Tailscale Service: %w", err)
-	}
 	if svc == nil {
 		return false, nil
 	}
+
 	o, err := parseOwnerAnnotation(svc)
 	if err != nil {
 		return false, fmt.Errorf("error parsing Tailscale Service owner annotation: %w", err)
 	}
+
 	if o == nil || len(o.OwnerRefs) == 0 {
 		return false, nil
 	}
+
 	// Comparing with the operatorID only means that we will not be able to
 	// clean up Tailscale Services in cases where the operator was deleted from the
 	// cluster before deleting the Ingress. Perhaps the comparison could be
@@ -550,18 +550,22 @@ func cleanupTailscaleService(ctx context.Context, tsClient tsClient, name tailcf
 	if ix == -1 {
 		return false, nil
 	}
+
 	if len(o.OwnerRefs) == 1 {
 		logger.Infof("Deleting Tailscale Service %q", name)
-		return false, tsClient.DeleteVIPService(ctx, name)
+		return false, tsClient.VIPServices().Delete(ctx, name)
 	}
+
 	o.OwnerRefs = slices.Delete(o.OwnerRefs, ix, ix+1)
 	logger.Infof("Updating Tailscale Service %q", name)
-	json, err := json.Marshal(o)
+
+	data, err := json.Marshal(o)
 	if err != nil {
 		return false, fmt.Errorf("error marshalling updated Tailscale Service owner reference: %w", err)
 	}
-	svc.Annotations[ownerAnnotation] = string(json)
+
-	return true, tsClient.CreateOrUpdateVIPService(ctx, svc)
+	svc.Annotations[ownerAnnotation] = string(data)
+	return true, tsClient.VIPServices().CreateOrUpdate(ctx, *svc)
 }
 
 func (r *HAServiceReconciler) backendRoutesSetup(ctx context.Context, serviceName, replicaName string, wantsCfg *ingressservices.Config, logger *zap.SugaredLogger) (bool, error) {

cmd/k8s-operator/svc-for-pg_test.go

@@ -22,15 +22,15 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"tailscale.com/client/tailscale/v2"
 
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/ingressservices"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tstest"
 	"tailscale.com/util/mak"
-
-	"tailscale.com/tailcfg"
 )
 
 func TestServicePGReconciler(t *testing.T) {
@@ -102,11 +102,11 @@ func TestServicePGReconciler_UpdateHostname(t *testing.T) {
 	verifyTailscaleService(t, ft, fmt.Sprintf("svc:%s", hostname), []string{"do-not-validate"})
 	verifyTailscaledConfig(t, fc, "test-pg", []string{fmt.Sprintf("svc:%s", hostname)})
 
-	_, err := ft.GetVIPService(context.Background(), tailcfg.ServiceName(fmt.Sprintf("svc:default-%s", svc.Name)))
+	_, err := ft.VIPServices().Get(context.Background(), fmt.Sprintf("svc:default-%s", svc.Name))
 	if err == nil {
 		t.Fatalf("svc:default-%s not cleaned up", svc.Name)
 	}
-	if !isErrorTailscaleServiceNotFound(err) {
+	if !tailscale.IsNotFound(err) {
 		t.Fatalf("unexpected error: %v", err)
 	}
 }
@@ -188,7 +188,9 @@ func setupServiceTest(t *testing.T) (*HAServiceReconciler, *corev1.Secret, clien
 		t.Fatal(err)
 	}
 
-	ft := &fakeTSClient{}
+	ft := &fakeTSClient{
+		vipServices: make(map[string]tailscale.VIPService),
+	}
 	zl, err := zap.NewDevelopment()
 	if err != nil {
 		t.Fatal(err)
@@ -197,7 +199,7 @@ func setupServiceTest(t *testing.T) (*HAServiceReconciler, *corev1.Secret, clien
 	cl := tstest.NewClock(tstest.ClockOpts{})
 	svcPGR := &HAServiceReconciler{
 		Client:      fc,
-		tsClient:    ft,
+		clients:     tsclient.NewProvider(ft),
 		clock:       cl,
 		defaultTags: []string{"tag:k8s"},
 		tsNamespace: "operator-ns",
@@ -275,22 +277,22 @@ func TestServicePGReconciler_MultiCluster(t *testing.T) {
 		if i == 0 {
 			ft = fti
 		} else {
-			pgr.tsClient = ft
+			pgr.clients = tsclient.NewProvider(ft)
 		}
 
 		svc, _ := setupTestService(t, "test-multi-cluster", "", "4.3.2.1", fc, stateSecret)
 		expectReconciled(t, pgr, "default", svc.Name)
 
-		tsSvcs, err := ft.ListVIPServices(context.Background())
+		tsSvcs, err := ft.VIPServices().List(t.Context())
 		if err != nil {
 			t.Fatalf("getting Tailscale Service: %v", err)
 		}
 
-		if len(tsSvcs.VIPServices) != 1 {
+		if len(tsSvcs) != 1 {
-			t.Fatalf("unexpected number of Tailscale Services (%d)", len(tsSvcs.VIPServices))
+			t.Fatalf("unexpected number of Tailscale Services (%d)", len(tsSvcs))
 		}
 
-		for _, svc := range tsSvcs.VIPServices {
+		for _, svc := range tsSvcs {
 			t.Logf("found Tailscale Service with name %q", svc.Name)
 		}
 	}
@@ -322,9 +324,9 @@ func TestIgnoreRegularService(t *testing.T) {
 
 	verifyTailscaledConfig(t, fc, "test-pg", nil)
 
-	tsSvcs, err := ft.ListVIPServices(context.Background())
+	tsSvcs, err := ft.VIPServices().List(t.Context())
 	if err == nil {
-		if len(tsSvcs.VIPServices) > 0 {
+		if len(tsSvcs) > 0 {
 			t.Fatal("unexpected Tailscale Services found")
 		}
 	}

cmd/k8s-operator/svc_test.go

@@ -16,7 +16,9 @@ import (
 	"k8s.io/apimachinery/pkg/types"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tstest"
 )
@@ -47,7 +49,7 @@ func TestService_DefaultProxyClassInitiallyNotReady(t *testing.T) {
 		Client: fc,
 		ssr: &tailscaleSTSReconciler{
 			Client:            fc,
-			tsClient:          ft,
+			clients:           tsclient.NewProvider(ft),
 			defaultTags:       []string{"tag:k8s"},
 			operatorNamespace: "operator-ns",
 			proxyImage:        "tailscale/tailscale",

cmd/k8s-operator/tailnet.go

@@ -1,71 +0,0 @@
-// Copyright (c) Tailscale Inc & contributors
-// SPDX-License-Identifier: BSD-3-Clause
-
-//go:build !plan9
-
-package main
-
-import (
-	"context"
-	"fmt"
-
-	"golang.org/x/oauth2"
-	"golang.org/x/oauth2/clientcredentials"
-	corev1 "k8s.io/api/core/v1"
-	"sigs.k8s.io/controller-runtime/pkg/client"
-
-	"tailscale.com/internal/client/tailscale"
-	"tailscale.com/ipn"
-	operatorutils "tailscale.com/k8s-operator"
-	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
-)
-
-func clientForTailnet(ctx context.Context, cl client.Client, namespace, name string) (tsClient, string, error) {
-	var tn tsapi.Tailnet
-	if err := cl.Get(ctx, client.ObjectKey{Name: name}, &tn); err != nil {
-		return nil, "", fmt.Errorf("failed to get tailnet %q: %w", name, err)
-	}
-
-	if !operatorutils.TailnetIsReady(&tn) {
-		return nil, "", fmt.Errorf("tailnet %q is not ready", name)
-	}
-
-	var secret corev1.Secret
-	if err := cl.Get(ctx, client.ObjectKey{Name: tn.Spec.Credentials.SecretName, Namespace: namespace}, &secret); err != nil {
-		return nil, "", fmt.Errorf("failed to get Secret %q in namespace %q: %w", tn.Spec.Credentials.SecretName, namespace, err)
-	}
-
-	baseURL := ipn.DefaultControlURL
-	if tn.Spec.LoginURL != "" {
-		baseURL = tn.Spec.LoginURL
-	}
-
-	credentials := clientcredentials.Config{
-		ClientID:     string(secret.Data["client_id"]),
-		ClientSecret: string(secret.Data["client_secret"]),
-		TokenURL:     baseURL + "/api/v2/oauth/token",
-	}
-
-	source := credentials.TokenSource(ctx)
-	httpClient := oauth2.NewClient(ctx, source)
-
-	ts := tailscale.NewClient(defaultTailnet, nil)
-	ts.UserAgent = "tailscale-k8s-operator"
-	ts.HTTPClient = httpClient
-	ts.BaseURL = baseURL
-
-	return ts, baseURL, nil
-}
-
-func clientFromProxyGroup(ctx context.Context, cl client.Client, pg *tsapi.ProxyGroup, namespace string, def tsClient) (tsClient, error) {
-	if pg.Spec.Tailnet == "" {
-		return def, nil
-	}
-
-	tailscaleClient, _, err := clientForTailnet(ctx, cl, namespace, pg.Spec.Tailnet)
-	if err != nil {
-		return nil, err
-	}
-
-	return tailscaleClient, nil
-}

cmd/k8s-operator/testutils_test.go

@@ -9,6 +9,7 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"maps"
 	"net/http"
 	"net/netip"
 	"path"
@@ -31,12 +32,12 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/internal/client/tailscale"
 	"tailscale.com/ipn"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
-	"tailscale.com/tailcfg"
 	"tailscale.com/util/mak"
 )
 
@@ -836,12 +837,131 @@ func expectEvents(t *testing.T, rec *record.FakeRecorder, wantsEvents []string)
 	}
 }
 
-type fakeTSClient struct {
+type (
+	fakeTSClient struct {
 		sync.Mutex
+		loginURL    string
 		keyRequests []tailscale.KeyCapabilities
 		deleted     []string
-	vipServices map[tailcfg.ServiceName]*tailscale.VIPService
+		devices     []tailscale.Device
+		vipServices map[string]tailscale.VIPService
+	}
+
+	fakeVIPServices struct {
+		mu          sync.RWMutex
+		vipServices map[string]tailscale.VIPService
+	}
+
+	fakeKeys struct {
+		keyRequests *[]tailscale.KeyCapabilities
+	}
+
+	fakeDevices struct {
+		deleted *[]string
+		devices *[]tailscale.Device
+	}
+)
+
+func (c *fakeTSClient) VIPServices() tsclient.VIPServiceResource {
+	return &fakeVIPServices{
+		vipServices: c.vipServices,
+	}
 }
+
+func (m *fakeVIPServices) List(_ context.Context) ([]tailscale.VIPService, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	if len(m.vipServices) == 0 {
+		return nil, tailscale.APIError{Status: http.StatusNotFound}
+	}
+
+	return slices.Collect(maps.Values(m.vipServices)), nil
+}
+
+func (m *fakeVIPServices) Delete(_ context.Context, name string) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	if _, ok := m.vipServices[name]; !ok {
+		return tailscale.APIError{Status: http.StatusNotFound}
+	}
+
+	delete(m.vipServices, name)
+	return nil
+}
+
+func (m *fakeVIPServices) Get(_ context.Context, name string) (*tailscale.VIPService, error) {
+	if svc, ok := m.vipServices[name]; ok {
+		return &svc, nil
+	}
+
+	return nil, tailscale.APIError{Status: http.StatusNotFound}
+}
+
+func (m *fakeVIPServices) CreateOrUpdate(_ context.Context, svc tailscale.VIPService) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	if svc.Addrs == nil {
+		svc.Addrs = []string{vipTestIP}
+	}
+
+	m.vipServices[svc.Name] = svc
+	return nil
+}
+
+func (c *fakeTSClient) Devices() tsclient.DeviceResource {
+	return &fakeDevices{
+		deleted: &c.deleted,
+		devices: &c.devices,
+	}
+}
+
+func (m *fakeDevices) Delete(_ context.Context, id string) error {
+	*m.deleted = append(*m.deleted, id)
+
+	return tailscale.APIError{Status: http.StatusNotFound}
+}
+
+func (m *fakeDevices) List(_ context.Context, _ ...tailscale.ListDevicesOptions) ([]tailscale.Device, error) {
+	return *m.devices, nil
+}
+
+func (m *fakeDevices) Get(_ context.Context, id string) (*tailscale.Device, error) {
+	if m.devices == nil {
+		return nil, tailscale.APIError{Status: http.StatusNotFound}
+	}
+
+	for _, dev := range *m.devices {
+		if dev.ID == id {
+			return &dev, nil
+		}
+	}
+
+	return nil, tailscale.APIError{Status: http.StatusNotFound}
+}
+
+func (c *fakeTSClient) Keys() tsclient.KeyResource {
+	return &fakeKeys{
+		keyRequests: &c.keyRequests,
+	}
+}
+
+func (m *fakeKeys) CreateAuthKey(_ context.Context, ckr tailscale.CreateKeyRequest) (*tailscale.Key, error) {
+	*m.keyRequests = append(*m.keyRequests, ckr.Capabilities)
+
+	return &tailscale.Key{Key: "new-authkey"}, nil
+}
+
+func (m *fakeKeys) List(_ context.Context, _ bool) ([]tailscale.Key, error) {
+	return nil, nil
+}
+
+func (c *fakeTSClient) LoginURL() string {
+	return c.loginURL
+}
+
 type fakeTSNetServer struct {
 	certDomains []string
 }
@@ -850,48 +970,6 @@ func (f *fakeTSNetServer) CertDomains() []string {
 	return f.certDomains
 }
 
-func (c *fakeTSClient) CreateKey(ctx context.Context, caps tailscale.KeyCapabilities) (string, *tailscale.Key, error) {
-	c.Lock()
-	defer c.Unlock()
-	c.keyRequests = append(c.keyRequests, caps)
-	k := &tailscale.Key{
-		ID:           "key",
-		Created:      time.Now(),
-		Capabilities: caps,
-	}
-	return "new-authkey", k, nil
-}
-
-func (c *fakeTSClient) Device(ctx context.Context, deviceID string, fields *tailscale.DeviceFieldsOpts) (*tailscale.Device, error) {
-	return &tailscale.Device{
-		DeviceID: deviceID,
-		Hostname: "hostname-" + deviceID,
-		Addresses: []string{
-			"1.2.3.4",
-			"::1",
-		},
-	}, nil
-}
-
-func (c *fakeTSClient) DeleteDevice(ctx context.Context, deviceID string) error {
-	c.Lock()
-	defer c.Unlock()
-	c.deleted = append(c.deleted, deviceID)
-	return nil
-}
-
-func (c *fakeTSClient) KeyRequests() []tailscale.KeyCapabilities {
-	c.Lock()
-	defer c.Unlock()
-	return c.keyRequests
-}
-
-func (c *fakeTSClient) Deleted() []string {
-	c.Lock()
-	defer c.Unlock()
-	return c.deleted
-}
-
 func removeResourceReqs(sts *appsv1.StatefulSet) {
 	if sts != nil {
 		sts.Spec.Template.Spec.Resources = nil
@@ -935,53 +1013,3 @@ func removeAuthKeyIfExistsModifier(t *testing.T) func(s *corev1.Secret) {
 		}
 	}
 }
-
-func (c *fakeTSClient) GetVIPService(ctx context.Context, name tailcfg.ServiceName) (*tailscale.VIPService, error) {
-	c.Lock()
-	defer c.Unlock()
-	if c.vipServices == nil {
-		return nil, tailscale.ErrResponse{Status: http.StatusNotFound}
-	}
-	svc, ok := c.vipServices[name]
-	if !ok {
-		return nil, tailscale.ErrResponse{Status: http.StatusNotFound}
-	}
-	return svc, nil
-}
-
-func (c *fakeTSClient) ListVIPServices(ctx context.Context) (*tailscale.VIPServiceList, error) {
-	c.Lock()
-	defer c.Unlock()
-	if c.vipServices == nil {
-		return nil, &tailscale.ErrResponse{Status: http.StatusNotFound}
-	}
-	result := &tailscale.VIPServiceList{}
-	for _, svc := range c.vipServices {
-		result.VIPServices = append(result.VIPServices, *svc)
-	}
-	return result, nil
-}
-
-func (c *fakeTSClient) CreateOrUpdateVIPService(ctx context.Context, svc *tailscale.VIPService) error {
-	c.Lock()
-	defer c.Unlock()
-	if c.vipServices == nil {
-		c.vipServices = make(map[tailcfg.ServiceName]*tailscale.VIPService)
-	}
-
-	if svc.Addrs == nil {
-		svc.Addrs = []string{vipTestIP}
-	}
-
-	c.vipServices[svc.Name] = svc
-	return nil
-}
-
-func (c *fakeTSClient) DeleteVIPService(ctx context.Context, name tailcfg.ServiceName) error {
-	c.Lock()
-	defer c.Unlock()
-	if c.vipServices != nil {
-		delete(c.vipServices, name)
-	}
-	return nil
-}

cmd/k8s-operator/tsclient.go

@@ -8,7 +8,7 @@ package main
 import (
 	"context"
 	"fmt"
-	"net/http"
+	"net/url"
 	"os"
 	"sync"
 	"time"
@@ -16,15 +16,12 @@ import (
 	"go.uber.org/zap"
 	"golang.org/x/oauth2"
 	"golang.org/x/oauth2/clientcredentials"
-	"tailscale.com/internal/client/tailscale"
+	"tailscale.com/client/tailscale/v2"
+
 	"tailscale.com/ipn"
-	"tailscale.com/tailcfg"
 )
 
-// defaultTailnet is a value that can be used in Tailscale API calls instead of tailnet name to indicate that the API
-// call should be performed on the default tailnet for the provided credentials.
 const (
-	defaultTailnet = "-"
 	oidcJWTPath = "/var/run/secrets/tailscale/serviceaccount/token"
 )
 
@@ -34,24 +31,31 @@ func newTSClient(logger *zap.SugaredLogger, clientID, clientIDPath, clientSecret
 		baseURL = loginServer
 	}
 
-	var httpClient *http.Client
+	base, err := url.Parse(baseURL)
+	if err != nil {
+		return nil, err
+	}
+
+	client := &tailscale.Client{
+		UserAgent: "tailscale-k8s-operator",
+		BaseURL:   base,
+	}
+
 	if clientID == "" {
 		// Use static client credentials mounted to disk.
-		id, err := os.ReadFile(clientIDPath)
+		clientIDBytes, err := os.ReadFile(clientIDPath)
 		if err != nil {
 			return nil, fmt.Errorf("error reading client ID %q: %w", clientIDPath, err)
 		}
-		secret, err := os.ReadFile(clientSecretPath)
+		clientSecretBytes, err := os.ReadFile(clientSecretPath)
 		if err != nil {
 			return nil, fmt.Errorf("reading client secret %q: %w", clientSecretPath, err)
 		}
-		credentials := clientcredentials.Config{
+
-			ClientID:     string(id),
+		client.Auth = &tailscale.OAuth{
-			ClientSecret: string(secret),
+			ClientID:     string(clientIDBytes),
-			TokenURL:     fmt.Sprintf("%s%s", baseURL, "/api/v2/oauth/token"),
+			ClientSecret: string(clientSecretBytes),
 		}
-		tokenSrc := credentials.TokenSource(context.Background())
-		httpClient = oauth2.NewClient(context.Background(), tokenSrc)
 	} else {
 		// Use workload identity federation.
 		tokenSrc := &jwtTokenSource{
@@ -62,34 +66,21 @@ func newTSClient(logger *zap.SugaredLogger, clientID, clientIDPath, clientSecret
 				TokenURL: fmt.Sprintf("%s%s", baseURL, "/api/v2/oauth/token-exchange"),
 			},
 		}
-		httpClient = &http.Client{
+
-			Transport: &oauth2.Transport{
+		client.Auth = &tailscale.IdentityFederation{
-				Source: tokenSrc,
+			ClientID: clientID,
+			IDTokenFunc: func() (string, error) {
+				token, err := tokenSrc.Token()
+				if err != nil {
+					return "", err
+				}
+
+				return token.AccessToken, nil
 			},
 		}
 	}
 
-	c := tailscale.NewClient(defaultTailnet, nil)
+	return client, nil
-	c.UserAgent = "tailscale-k8s-operator"
-	c.HTTPClient = httpClient
-	if loginServer != "" {
-		c.BaseURL = loginServer
-	}
-	return c, nil
-}
-
-type tsClient interface {
-	CreateKey(ctx context.Context, caps tailscale.KeyCapabilities) (string, *tailscale.Key, error)
-	Device(ctx context.Context, deviceID string, fields *tailscale.DeviceFieldsOpts) (*tailscale.Device, error)
-	DeleteDevice(ctx context.Context, nodeStableID string) error
-	// GetVIPService is a method for getting a Tailscale Service. VIPService is the original name for Tailscale Service.
-	GetVIPService(ctx context.Context, name tailcfg.ServiceName) (*tailscale.VIPService, error)
-	// ListVIPServices is a method for listing all Tailscale Services. VIPService is the original name for Tailscale Service.
-	ListVIPServices(ctx context.Context) (*tailscale.VIPServiceList, error)
-	// CreateOrUpdateVIPService is a method for creating or updating a Tailscale Service.
-	CreateOrUpdateVIPService(ctx context.Context, svc *tailscale.VIPService) error
-	// DeleteVIPService is a method for deleting a Tailscale Service.
-	DeleteVIPService(ctx context.Context, name tailcfg.ServiceName) error
 }
 
 // jwtTokenSource implements the [oauth2.TokenSource] interface, but with the

cmd/k8s-operator/tsclient_test.go

@@ -1,135 +0,0 @@
-// Copyright (c) Tailscale Inc & contributors
-// SPDX-License-Identifier: BSD-3-Clause
-
-//go:build !plan9
-
-package main
-
-import (
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"net/http/httptest"
-	"os"
-	"path/filepath"
-	"testing"
-
-	"go.uber.org/zap"
-	"golang.org/x/oauth2"
-)
-
-func TestNewStaticClient(t *testing.T) {
-	const (
-		clientIDFile     = "client-id"
-		clientSecretFile = "client-secret"
-	)
-
-	tmp := t.TempDir()
-	clientIDPath := filepath.Join(tmp, clientIDFile)
-	if err := os.WriteFile(clientIDPath, []byte("test-client-id"), 0600); err != nil {
-		t.Fatalf("error writing test file %q: %v", clientIDPath, err)
-	}
-	clientSecretPath := filepath.Join(tmp, clientSecretFile)
-	if err := os.WriteFile(clientSecretPath, []byte("test-client-secret"), 0600); err != nil {
-		t.Fatalf("error writing test file %q: %v", clientSecretPath, err)
-	}
-
-	srv := testAPI(t, 3600)
-	cl, err := newTSClient(zap.NewNop().Sugar(), "", clientIDPath, clientSecretPath, srv.URL)
-	if err != nil {
-		t.Fatalf("error creating Tailscale client: %v", err)
-	}
-
-	resp, err := cl.HTTPClient.Get(srv.URL)
-	if err != nil {
-		t.Fatalf("error making test API call: %v", err)
-	}
-	defer resp.Body.Close()
-
-	got, err := io.ReadAll(resp.Body)
-	if err != nil {
-		t.Fatalf("error reading response body: %v", err)
-	}
-	want := "Bearer " + testToken("/api/v2/oauth/token", "test-client-id", "test-client-secret", "")
-	if string(got) != want {
-		t.Errorf("got %q; want %q", got, want)
-	}
-}
-
-func TestNewWorkloadIdentityClient(t *testing.T) {
-	// 5 seconds is within expiryDelta leeway, so the access token will
-	// immediately be considered expired and get refreshed on each access.
-	srv := testAPI(t, 5)
-	cl, err := newTSClient(zap.NewNop().Sugar(), "test-client-id", "", "", srv.URL)
-	if err != nil {
-		t.Fatalf("error creating Tailscale client: %v", err)
-	}
-
-	// Modify the path where the JWT will be read from.
-	oauth2Transport, ok := cl.HTTPClient.Transport.(*oauth2.Transport)
-	if !ok {
-		t.Fatalf("expected oauth2.Transport, got %T", cl.HTTPClient.Transport)
-	}
-	jwtTokenSource, ok := oauth2Transport.Source.(*jwtTokenSource)
-	if !ok {
-		t.Fatalf("expected jwtTokenSource, got %T", oauth2Transport.Source)
-	}
-	tmp := t.TempDir()
-	jwtPath := filepath.Join(tmp, "token")
-	jwtTokenSource.jwtPath = jwtPath
-
-	for _, jwt := range []string{"test-jwt", "updated-test-jwt"} {
-		if err := os.WriteFile(jwtPath, []byte(jwt), 0600); err != nil {
-			t.Fatalf("error writing test file %q: %v", jwtPath, err)
-		}
-		resp, err := cl.HTTPClient.Get(srv.URL)
-		if err != nil {
-			t.Fatalf("error making test API call: %v", err)
-		}
-		defer resp.Body.Close()
-
-		got, err := io.ReadAll(resp.Body)
-		if err != nil {
-			t.Fatalf("error reading response body: %v", err)
-		}
-		if want := "Bearer " + testToken("/api/v2/oauth/token-exchange", "test-client-id", "", jwt); string(got) != want {
-			t.Errorf("got %q; want %q", got, want)
-		}
-	}
-}
-
-func testAPI(t *testing.T, expirationSeconds int) *httptest.Server {
-	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		t.Logf("test server got request: %s %s", r.Method, r.URL.Path)
-		switch r.URL.Path {
-		case "/api/v2/oauth/token", "/api/v2/oauth/token-exchange":
-			id, secret, ok := r.BasicAuth()
-			if !ok {
-				t.Fatal("missing or invalid basic auth")
-			}
-			w.Header().Set("Content-Type", "application/json")
-			if err := json.NewEncoder(w).Encode(map[string]any{
-				"access_token": testToken(r.URL.Path, id, secret, r.FormValue("jwt")),
-				"token_type":   "Bearer",
-				"expires_in":   expirationSeconds,
-			}); err != nil {
-				t.Fatalf("error writing response: %v", err)
-			}
-		case "/":
-			// Echo back the authz header for test assertions.
-			_, err := w.Write([]byte(r.Header.Get("Authorization")))
-			if err != nil {
-				t.Fatalf("error writing response: %v", err)
-			}
-		default:
-			w.WriteHeader(http.StatusNotFound)
-		}
-	}))
-	t.Cleanup(srv.Close)
-	return srv
-}
-
-func testToken(path, id, secret, jwt string) string {
-	return fmt.Sprintf("%s|%s|%s|%s", path, id, secret, jwt)
-}

cmd/k8s-operator/tsrecorder.go

@@ -10,14 +10,15 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
-	"net/http"
 	"slices"
 	"strconv"
 	"strings"
 	"sync"
+	"time"
 
 	"go.uber.org/zap"
 	xslices "golang.org/x/exp/slices"
+	"golang.org/x/time/rate"
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
 	rbacv1 "k8s.io/api/rbac/v1"
@@ -30,10 +31,11 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	"tailscale.com/client/tailscale/v2"
 
-	"tailscale.com/client/tailscale"
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/kube/kubetypes"
 	"tailscale.com/tailcfg"
 	"tailscale.com/tstime"
@@ -60,10 +62,10 @@ type RecorderReconciler struct {
 	log               *zap.SugaredLogger
 	recorder          record.EventRecorder
 	clock             tstime.Clock
+	clients           ClientProvider
 	tsNamespace       string
-	tsClient    tsClient
+	authKeyRateLimits map[string]*rate.Limiter // per-Recorder rate limiters for auth key re-issuance.
-	loginServer string
+	authKeyReissuing  map[string]bool
-
 	mu                sync.Mutex           // protects following
 	recorders         set.Slice[types.UID] // for recorders gauge
 }
@@ -99,7 +101,7 @@ func (r *RecorderReconciler) Reconcile(ctx context.Context, req reconcile.Reques
 		return reconcile.Result{}, nil
 	}
 
-	tailscaleClient, loginUrl, err := r.getClientAndLoginURL(ctx, tsr.Spec.Tailnet)
+	tsClient, err := r.clients.For(tsr.Spec.Tailnet)
 	if err != nil {
 		return setStatusReady(tsr, metav1.ConditionFalse, reasonRecorderTailnetUnavailable, err.Error())
 	}
@@ -112,7 +114,7 @@ func (r *RecorderReconciler) Reconcile(ctx context.Context, req reconcile.Reques
 			return reconcile.Result{}, nil
 		}
 
-		if done, err := r.maybeCleanup(ctx, tsr, tailscaleClient); err != nil {
+		if done, err := r.maybeCleanup(ctx, tsr, tsClient); err != nil {
 			return reconcile.Result{}, err
 		} else if !done {
 			logger.Debugf("Recorder resource cleanup not yet finished, will retry...")
@@ -144,7 +146,7 @@ func (r *RecorderReconciler) Reconcile(ctx context.Context, req reconcile.Reques
 		return setStatusReady(tsr, metav1.ConditionFalse, reasonRecorderInvalid, message)
 	}
 
-	if err = r.maybeProvision(ctx, tailscaleClient, loginUrl, tsr); err != nil {
+	if err = r.maybeProvision(ctx, tsClient, tsr); err != nil {
 		reason := reasonRecorderCreationFailed
 		message := fmt.Sprintf("failed creating Recorder: %s", err)
 		if strings.Contains(err.Error(), optimisticLockErrorMsg) {
@@ -162,47 +164,33 @@ func (r *RecorderReconciler) Reconcile(ctx context.Context, req reconcile.Reques
 	return setStatusReady(tsr, metav1.ConditionTrue, reasonRecorderCreated, reasonRecorderCreated)
 }
 
-// getClientAndLoginURL returns the appropriate Tailscale client and resolved login URL
+func (r *RecorderReconciler) maybeProvision(ctx context.Context, tsClient tsclient.Client, tsr *tsapi.Recorder) error {
-// for the given tailnet name. If no tailnet is specified, returns the default client
-// and login server. Applies fallback to the operator's login server if the tailnet
-// doesn't specify a custom login URL.
-func (r *RecorderReconciler) getClientAndLoginURL(ctx context.Context, tailnetName string) (tsClient,
-	string, error) {
-	if tailnetName == "" {
-		return r.tsClient, r.loginServer, nil
-	}
-
-	tc, loginUrl, err := clientForTailnet(ctx, r.Client, r.tsNamespace, tailnetName)
-	if err != nil {
-		return nil, "", err
-	}
-
-	// Apply fallback if tailnet doesn't specify custom login URL
-	if loginUrl == "" {
-		loginUrl = r.loginServer
-	}
-
-	return tc, loginUrl, nil
-}
-
-func (r *RecorderReconciler) maybeProvision(ctx context.Context, tailscaleClient tsClient, loginUrl string, tsr *tsapi.Recorder) error {
 	logger := r.logger(tsr.Name)
 
-	r.mu.Lock()
-	r.recorders.Add(tsr.UID)
-	gaugeRecorderResources.Set(int64(r.recorders.Len()))
-	r.mu.Unlock()
-
-	if err := r.ensureAuthSecretsCreated(ctx, tailscaleClient, tsr); err != nil {
-		return fmt.Errorf("error creating secrets: %w", err)
-	}
-
-	// State Secrets are pre-created so we can use the Recorder CR as its owner ref.
 	var replicas int32 = 1
 	if tsr.Spec.Replicas != nil {
 		replicas = *tsr.Spec.Replicas
 	}
 
+	r.mu.Lock()
+	r.recorders.Add(tsr.UID)
+	gaugeRecorderResources.Set(int64(r.recorders.Len()))
+	if _, ok := r.authKeyRateLimits[tsr.Name]; !ok {
+		r.authKeyRateLimits[tsr.Name] = rate.NewLimiter(rate.Every(30*time.Second), int(replicas))
+	}
+	for replica := range replicas {
+		name := fmt.Sprintf("%s-%d", tsr.Name, replica)
+		if _, ok := r.authKeyReissuing[name]; !ok {
+			r.authKeyReissuing[name] = false
+		}
+	}
+	r.mu.Unlock()
+
+	if err := r.ensureAuthSecretsCreated(ctx, tsClient, tsr); err != nil {
+		return fmt.Errorf("error creating secrets: %w", err)
+	}
+
+	// State Secrets are pre-created so we can use the Recorder CR as its owner ref.
 	for replica := range replicas {
 		sec := tsrStateSecret(tsr, r.tsNamespace, replica)
 		_, err := createOrUpdate(ctx, r.Client, r.tsNamespace, sec, func(s *corev1.Secret) {
@@ -252,7 +240,7 @@ func (r *RecorderReconciler) maybeProvision(ctx context.Context, tailscaleClient
 		return fmt.Errorf("error creating RoleBinding: %w", err)
 	}
 
-	ss := tsrStatefulSet(tsr, r.tsNamespace, loginUrl)
+	ss := tsrStatefulSet(tsr, r.tsNamespace, tsClient.LoginURL())
 	_, err = createOrUpdate(ctx, r.Client, r.tsNamespace, ss, func(s *appsv1.StatefulSet) {
 		s.ObjectMeta.Labels = ss.ObjectMeta.Labels
 		s.ObjectMeta.Annotations = ss.ObjectMeta.Annotations
@@ -271,13 +259,13 @@ func (r *RecorderReconciler) maybeProvision(ctx context.Context, tailscaleClient
 
 	// If we have scaled the recorder down, we will have dangling state secrets
 	// that we need to clean up.
-	if err = r.maybeCleanupSecrets(ctx, tailscaleClient, tsr); err != nil {
+	if err = r.maybeCleanupSecrets(ctx, tsClient, tsr); err != nil {
 		return fmt.Errorf("error cleaning up Secrets: %w", err)
 	}
 
 	var devices []tsapi.RecorderTailnetDevice
 	for replica := range replicas {
-		dev, ok, err := r.getDeviceInfo(ctx, tailscaleClient, tsr.Name, replica)
+		dev, ok, err := r.getDeviceInfo(ctx, tsClient, tsr.Name, replica)
 		switch {
 		case err != nil:
 			return fmt.Errorf("failed to get device info: %w", err)
@@ -342,7 +330,7 @@ func (r *RecorderReconciler) maybeCleanupServiceAccounts(ctx context.Context, ts
 	return nil
 }
 
-func (r *RecorderReconciler) maybeCleanupSecrets(ctx context.Context, tailscaleClient tsClient, tsr *tsapi.Recorder) error {
+func (r *RecorderReconciler) maybeCleanupSecrets(ctx context.Context, tsClient tsclient.Client, tsr *tsapi.Recorder) error {
 	options := []client.ListOption{
 		client.InNamespace(r.tsNamespace),
 		client.MatchingLabels(tsrLabels("recorder", tsr.Name, nil)),
@@ -382,11 +370,12 @@ func (r *RecorderReconciler) maybeCleanupSecrets(ctx context.Context, tailscaleC
 
 		if ok {
 			r.log.Debugf("deleting device %s", devicePrefs.Config.NodeID)
-			err = tailscaleClient.DeleteDevice(ctx, string(devicePrefs.Config.NodeID))
+			err = tsClient.Devices().Delete(ctx, string(devicePrefs.Config.NodeID))
-			if errResp, ok := errors.AsType[*tailscale.ErrResponse](err); ok && errResp.Status == http.StatusNotFound {
+			switch {
+			case tailscale.IsNotFound(err):
 			// This device has possibly already been deleted in the admin console. So we can ignore this
 			// and move on to removing the secret.
-			} else if err != nil {
+			case err != nil:
 				return err
 			}
 		}
@@ -402,7 +391,7 @@ func (r *RecorderReconciler) maybeCleanupSecrets(ctx context.Context, tailscaleC
 // maybeCleanup just deletes the device from the tailnet. All the kubernetes
 // resources linked to a Recorder will get cleaned up via owner references
 // (which we can use because they are all in the same namespace).
-func (r *RecorderReconciler) maybeCleanup(ctx context.Context, tsr *tsapi.Recorder, tailscaleClient tsClient) (bool, error) {
+func (r *RecorderReconciler) maybeCleanup(ctx context.Context, tsr *tsapi.Recorder, tsClient tsclient.Client) (bool, error) {
 	logger := r.logger(tsr.Name)
 
 	var replicas int32 = 1
@@ -426,12 +415,12 @@ func (r *RecorderReconciler) maybeCleanup(ctx context.Context, tsr *tsapi.Record
 
 		nodeID := string(devicePrefs.Config.NodeID)
 		logger.Debugf("deleting device %s from control", nodeID)
-		if err = tailscaleClient.DeleteDevice(ctx, nodeID); err != nil {
+		err = tsClient.Devices().Delete(ctx, nodeID)
-			if errResp, ok := errors.AsType[tailscale.ErrResponse](err); ok && errResp.Status == http.StatusNotFound {
+		switch {
+		case tailscale.IsNotFound(err):
 			logger.Debugf("device %s not found, likely because it has already been deleted from control", nodeID)
 			continue
-			}
+		case err != nil:
-
 			return false, fmt.Errorf("error deleting device: %w", err)
 		}
 
@@ -446,12 +435,16 @@ func (r *RecorderReconciler) maybeCleanup(ctx context.Context, tsr *tsapi.Record
 	r.mu.Lock()
 	r.recorders.Remove(tsr.UID)
 	gaugeRecorderResources.Set(int64(r.recorders.Len()))
+	delete(r.authKeyRateLimits, tsr.Name)
+	for replica := range replicas {
+		delete(r.authKeyReissuing, fmt.Sprintf("%s-%d", tsr.Name, replica))
+	}
 	r.mu.Unlock()
 
 	return true, nil
 }
 
-func (r *RecorderReconciler) ensureAuthSecretsCreated(ctx context.Context, tailscaleClient tsClient, tsr *tsapi.Recorder) error {
+func (r *RecorderReconciler) ensureAuthSecretsCreated(ctx context.Context, tsClient tsclient.Client, tsr *tsapi.Recorder) error {
 	var replicas int32 = 1
 	if tsr.Spec.Replicas != nil {
 		replicas = *tsr.Spec.Replicas
@@ -470,28 +463,122 @@ func (r *RecorderReconciler) ensureAuthSecretsCreated(ctx context.Context, tails
 			Name:      fmt.Sprintf("%s-auth-%d", tsr.Name, replica),
 		}
 
-		err := r.Get(ctx, key, &corev1.Secret{})
+		existingSecret := &corev1.Secret{}
+		err := r.Get(ctx, key, existingSecret)
 		switch {
 		case err == nil:
-			logger.Debugf("auth Secret %q already exists", key.Name)
+			reissue, err := r.shouldReissueAuthKey(ctx, tsClient, tsr, replica, existingSecret)
-			continue
+			if err != nil {
-		case !apierrors.IsNotFound(err):
+				return fmt.Errorf("error checking auth key reissue for replica %d: %w", replica, err)
-			return fmt.Errorf("failed to get Secret %q: %w", key.Name, err)
 			}
-
+			if !reissue {
-		authKey, err := newAuthKey(ctx, tailscaleClient, tags.Stringify())
+				logger.Debugf("auth Secret %q already exists, no reissue needed", key.Name)
+				continue
+			}
+			authKey, err := newAuthKey(ctx, tsClient, tags.Stringify())
 			if err != nil {
 				return err
 			}
-
+			existingSecret.Data["authkey"] = []byte(authKey)
-		if err = r.Create(ctx, tsrAuthSecret(tsr, r.tsNamespace, authKey, replica)); err != nil {
+			if err = r.Update(ctx, existingSecret); err != nil {
 				return err
 			}
+			continue
+		case apierrors.IsNotFound(err):
+			authKey, err := newAuthKey(ctx, tsClient, tags.Stringify())
+			if err != nil {
+				return err
+			}
+			if err := r.Create(ctx, tsrAuthSecret(tsr, r.tsNamespace, authKey, replica)); err != nil {
+				return err
+			}
+		default:
+			return fmt.Errorf("failed to get Secret %q: %w", key.Name, err)
+		}
 	}
 
 	return nil
 }
 
+// shouldReissueAuthKey returns true if the proxy needs a new auth key. It
+// tracks in-flight reissues via authKeyReissuing to avoid duplicate API calls
+// across reconciles.
+func (r *RecorderReconciler) shouldReissueAuthKey(ctx context.Context, tsClient tsclient.Client, tsr *tsapi.Recorder, replica int32, authSecret *corev1.Secret) (shouldReissue bool, err error) {
+	stateSecret, err := r.getStateSecret(ctx, tsr.Name, replica)
+	if err != nil || stateSecret == nil {
+		return false, err
+	}
+
+	stateSecretName := fmt.Sprintf("%s-%d", tsr.Name, replica)
+
+	r.mu.Lock()
+	reissuing := r.authKeyReissuing[stateSecretName]
+	r.mu.Unlock()
+
+	if reissuing {
+		_, requestStillPresent := stateSecret.Data[kubetypes.KeyReissueAuthkey]
+		if !requestStillPresent {
+			r.mu.Lock()
+			r.authKeyReissuing[stateSecretName] = false
+			r.mu.Unlock()
+			r.log.Debugf("auth key reissue completed for %q", stateSecretName)
+			return false, nil
+		}
+		r.log.Debugf("auth key already in process of re-issuance for %q, waiting", stateSecretName)
+		return false, nil
+	}
+
+	defer func() {
+		r.mu.Lock()
+		r.authKeyReissuing[stateSecretName] = shouldReissue
+		r.mu.Unlock()
+	}()
+
+	brokenAuthkey, ok := stateSecret.Data[kubetypes.KeyReissueAuthkey]
+	if !ok {
+		return false, nil
+	}
+
+	cfgAuthKey := string(authSecret.Data["authkey"])
+	empty := cfgAuthKey == ""
+	broken := cfgAuthKey == string(brokenAuthkey)
+
+	if !empty && !broken {
+		return false, nil
+	}
+
+	lim := r.authKeyRateLimits[tsr.Name]
+	if !lim.Allow() {
+		r.log.Debugf("auth key re-issuance rate limit exceeded, limit: %.2f, burst: %d, tokens: %.2f",
+			lim.Limit(), lim.Burst(), lim.Tokens())
+		return false, fmt.Errorf("auth key re-issuance rate limit exceeded for Recorder %q, will retry with backoff", tsr.Name)
+	}
+
+	r.log.Infof("Recorder replica %s failing to auth; attempting cleanup and new key", stateSecretName)
+	if tsID := stateSecret.Data[kubetypes.KeyDeviceID]; len(tsID) > 0 {
+		id := tailcfg.StableNodeID(tsID)
+		if err := r.ensureDeviceDeleted(ctx, tsClient, id, r.log); err != nil {
+			return false, err
+		}
+	}
+
+	return true, nil
+}
+
+func (r *RecorderReconciler) ensureDeviceDeleted(ctx context.Context, tsClient tsclient.Client, id tailcfg.StableNodeID, logger *zap.SugaredLogger) error {
+	logger.Debugf("deleting device %s from control", string(id))
+	err := tsClient.Devices().Delete(ctx, string(id))
+	switch {
+	case tailscale.IsNotFound(err):
+		logger.Debugf("device %s not found, likely because it has already been deleted from control", string(id))
+	case err != nil:
+		return fmt.Errorf("error deleting device: %w", err)
+	default:
+		logger.Debugf("device %s deleted from control", string(id))
+	}
+	return nil
+}
+
 func (r *RecorderReconciler) validate(ctx context.Context, tsr *tsapi.Recorder) error {
 	if !tsr.Spec.EnableUI && tsr.Spec.Storage.S3 == nil {
 		return errors.New("must either enable UI or use S3 storage to ensure recordings are accessible")
@@ -581,7 +668,7 @@ func getDevicePrefs(secret *corev1.Secret) (prefs prefs, ok bool, err error) {
 	return prefs, ok, nil
 }
 
-func (r *RecorderReconciler) getDeviceInfo(ctx context.Context, tailscaleClient tsClient, tsrName string, replica int32) (d tsapi.RecorderTailnetDevice, ok bool, err error) {
+func (r *RecorderReconciler) getDeviceInfo(ctx context.Context, tsClient tsclient.Client, tsrName string, replica int32) (d tsapi.RecorderTailnetDevice, ok bool, err error) {
 	secret, err := r.getStateSecret(ctx, tsrName, replica)
 	if err != nil || secret == nil {
 		return tsapi.RecorderTailnetDevice{}, false, err
@@ -595,7 +682,7 @@ func (r *RecorderReconciler) getDeviceInfo(ctx context.Context, tailscaleClient
 	// TODO(tomhjp): The profile info doesn't include addresses, which is why we
 	// need the API. Should maybe update tsrecorder to write IPs to the state
 	// Secret like containerboot does.
-	device, err := tailscaleClient.Device(ctx, string(prefs.Config.NodeID), nil)
+	device, err := tsClient.Devices().Get(ctx, string(prefs.Config.NodeID))
 	if err != nil {
 		return tsapi.RecorderTailnetDevice{}, false, fmt.Errorf("failed to get device info from API: %w", err)
 	}

cmd/k8s-operator/tsrecorder_test.go

@@ -14,6 +14,7 @@ import (
 
 	"github.com/google/go-cmp/cmp"
 	"go.uber.org/zap"
+	"golang.org/x/time/rate"
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
 	rbacv1 "k8s.io/api/rbac/v1"
@@ -21,9 +22,11 @@ import (
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"tailscale.com/client/tailscale/v2"
 
 	tsoperator "tailscale.com/k8s-operator"
 	tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
+	"tailscale.com/k8s-operator/tsclient"
 	"tailscale.com/tstest"
 )
 
@@ -48,18 +51,19 @@ func TestRecorder(t *testing.T) {
 		WithObjects(tsr).
 		WithStatusSubresource(tsr).
 		Build()
-	tsClient := &fakeTSClient{}
+	tsClient := &fakeTSClient{loginURL: tsLoginServer}
 	zl, _ := zap.NewDevelopment()
 	fr := record.NewFakeRecorder(2)
 	cl := tstest.NewClock(tstest.ClockOpts{})
 	reconciler := &RecorderReconciler{
 		tsNamespace:       tsNamespace,
 		Client:            fc,
-		tsClient:    tsClient,
+		clients:           tsclient.NewProvider(tsClient),
 		recorder:          fr,
 		log:               zl.Sugar(),
 		clock:             cl,
-		loginServer: tsLoginServer,
+		authKeyRateLimits: make(map[string]*rate.Limiter),
+		authKeyReissuing:  make(map[string]bool),
 	}
 
 	t.Run("invalid_spec_gives_an_error_condition", func(t *testing.T) {
@@ -194,8 +198,8 @@ func TestRecorder(t *testing.T) {
 	})
 
 	t.Run("populate_node_info_in_state_secret_and_see_it_appear_in_status", func(t *testing.T) {
-
 		const key = "profile-abc"
+
 		for replica := range *tsr.Spec.Replicas {
 			bytes, err := json.Marshal(map[string]any{
 				"Config": map[string]any{
@@ -218,6 +222,24 @@ func TestRecorder(t *testing.T) {
 			})
 		}
 
+		tsClient.devices = []tailscale.Device{
+			{
+				ID:        "node-0",
+				Hostname:  "hostname-node-0",
+				Addresses: []string{"1.2.3.4", "::1"},
+			},
+			{
+				ID:        "node-1",
+				Hostname:  "hostname-node-1",
+				Addresses: []string{"1.2.3.4", "::1"},
+			},
+			{
+				ID:        "node-2",
+				Hostname:  "hostname-node-2",
+				Addresses: []string{"1.2.3.4", "::1"},
+			},
+		}
+
 		expectReconciled(t, reconciler, "", tsr.Name)
 		tsr.Status.Devices = []tsapi.RecorderTailnetDevice{
 			{

cmd/k8s-proxy/k8s-proxy.go

@@ -31,6 +31,7 @@ import (
 	"k8s.io/utils/strings/slices"
 	"tailscale.com/client/local"
 	"tailscale.com/cmd/k8s-proxy/internal/config"
+	"tailscale.com/health"
 	"tailscale.com/hostinfo"
 	"tailscale.com/ipn"
 	"tailscale.com/ipn/store"
@@ -41,6 +42,7 @@ import (
 	"tailscale.com/kube/certs"
 	healthz "tailscale.com/kube/health"
 	"tailscale.com/kube/k8s-proxy/conf"
+	"tailscale.com/kube/kubeclient"
 	"tailscale.com/kube/kubetypes"
 	klc "tailscale.com/kube/localclient"
 	"tailscale.com/kube/metrics"
@@ -171,10 +173,31 @@ func run(logger *zap.SugaredLogger) error {
 
 	// If Pod UID unset, assume we're running outside of a cluster/not managed
 	// by the operator, so no need to set additional state keys.
+	var kc kubeclient.Client
+	var stateSecretName string
 	if podUID != "" {
 		if err := state.SetInitialKeys(st, podUID); err != nil {
 			return fmt.Errorf("error setting initial state: %w", err)
 		}
+
+		if cfg.Parsed.State != nil {
+			if name, ok := strings.CutPrefix(*cfg.Parsed.State, "kube:"); ok {
+				stateSecretName = name
+
+				kc, err = kubeclient.New(k8sProxyFieldManager)
+				if err != nil {
+					return err
+				}
+
+				var configAuthKey string
+				if cfg.Parsed.AuthKey != nil {
+					configAuthKey = *cfg.Parsed.AuthKey
+				}
+				if err := resetState(ctx, kc, stateSecretName, podUID, configAuthKey); err != nil {
+					return fmt.Errorf("error resetting state: %w", err)
+				}
+			}
+		}
 	}
 
 	var authKey string
@@ -197,23 +220,69 @@ func run(logger *zap.SugaredLogger) error {
 		ts.Hostname = *cfg.Parsed.Hostname
 	}
 
-	// Make sure we crash loop if Up doesn't complete in reasonable time.
-	upCtx, upCancel := context.WithTimeout(ctx, time.Minute)
-	defer upCancel()
-	if _, err := ts.Up(upCtx); err != nil {
-		return fmt.Errorf("error starting tailscale server: %w", err)
-	}
-	defer ts.Close()
 	lc, err := ts.LocalClient()
 	if err != nil {
 		return fmt.Errorf("error getting local client: %w", err)
 	}
 
-	// Setup for updating state keys.
+	// Make sure we crash loop if Up doesn't complete in reasonable time.
+	upCtx, upCancel := context.WithTimeout(ctx, 30*time.Second)
+	defer upCancel()
+
+	// ts.Up() deliberately ignores NeedsLogin because it fires transiently
+	// during normal auth-key login. We can watch for the login-state health
+	// warning here though, which only fires on terminal auth failure, and
+	// cancel early.
+	go func() {
+		w, err := lc.WatchIPNBus(upCtx, ipn.NotifyInitialHealthState)
+		if err != nil {
+			return
+		}
+		defer w.Close()
+		for {
+			n, err := w.Next()
+			if err != nil {
+				logger.Debugf("failed to process message from ipn bus: %s", err.Error())
+				return
+			}
+			if n.Health != nil {
+				if _, ok := n.Health.Warnings[health.LoginStateWarnable.Code]; ok {
+					upCancel()
+					return
+				}
+			}
+		}
+	}()
+
+	if _, err := ts.Up(upCtx); err != nil {
+		if kc != nil && stateSecretName != "" {
+			return handleAuthKeyReissue(ctx, lc, kc, stateSecretName, authKey, cfgChan, logger)
+		}
+		return err
+	}
+
+	defer ts.Close()
+
+	reissueCh := make(chan struct{}, 1)
 	if podUID != "" {
 		group.Go(func() error {
 			return state.KeepKeysUpdated(ctx, st, klc.New(lc))
 		})
+
+		if kc != nil && stateSecretName != "" {
+			needsReissue, err := checkInitialAuthState(ctx, lc)
+			if err != nil {
+				return fmt.Errorf("error checking initial auth state: %w", err)
+			}
+			if needsReissue {
+				logger.Info("Auth key missing or invalid after startup, requesting new key from operator")
+				return handleAuthKeyReissue(ctx, lc, kc, stateSecretName, authKey, cfgChan, logger)
+			}
+
+			group.Go(func() error {
+				return monitorAuthHealth(ctx, lc, reissueCh, logger)
+			})
+		}
 	}
 
 	if cfg.Parsed.HealthCheckEnabled.EqualBool(true) || cfg.Parsed.MetricsEnabled.EqualBool(true) {
@@ -362,6 +431,8 @@ func run(logger *zap.SugaredLogger) error {
 			}
 
 			cfgLogger.Infof("Config reloaded")
+		case <-reissueCh:
+			return handleAuthKeyReissue(ctx, lc, kc, stateSecretName, authKey, cfgChan, logger)
 		}
 	}
 }

cmd/k8s-proxy/kube.go

@@ -0,0 +1,161 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build !plan9
+
+package main
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"go.uber.org/zap"
+	"tailscale.com/client/local"
+	"tailscale.com/health"
+	"tailscale.com/ipn"
+	"tailscale.com/kube/authkey"
+	"tailscale.com/kube/k8s-proxy/conf"
+	"tailscale.com/kube/kubeapi"
+	"tailscale.com/kube/kubeclient"
+	"tailscale.com/kube/kubetypes"
+	"tailscale.com/tailcfg"
+)
+
+const k8sProxyFieldManager = "tailscale-k8s-proxy"
+
+// resetState clears k8s-proxy state from previous runs and sets
+// initial values. This ensures the operator doesn't use stale state when a Pod
+// is first recreated.
+//
+// It also clears the reissue_authkey marker if the operator has actioned it
+// (i.e., the config now has a different auth key than what was marked for
+// reissue).
+func resetState(ctx context.Context, kc kubeclient.Client, stateSecretName string, podUID string, configAuthKey string) error {
+	existingSecret, err := kc.GetSecret(ctx, stateSecretName)
+	switch {
+	case kubeclient.IsNotFoundErr(err):
+		return nil
+	case err != nil:
+		return fmt.Errorf("failed to read state Secret %q to reset state: %w", stateSecretName, err)
+	}
+
+	s := &kubeapi.Secret{
+		Data: map[string][]byte{
+			kubetypes.KeyCapVer: fmt.Appendf(nil, "%d", tailcfg.CurrentCapabilityVersion),
+		},
+	}
+	if podUID != "" {
+		s.Data[kubetypes.KeyPodUID] = []byte(podUID)
+	}
+
+	// Only clear reissue_authkey if the operator has actioned it.
+	brokenAuthkey, ok := existingSecret.Data[kubetypes.KeyReissueAuthkey]
+	if ok && configAuthKey != "" && string(brokenAuthkey) != configAuthKey {
+		s.Data[kubetypes.KeyReissueAuthkey] = nil
+	}
+
+	return kc.StrategicMergePatchSecret(ctx, stateSecretName, s, k8sProxyFieldManager)
+}
+
+// needsAuthKeyReissue reports whether the given backend state and health
+// warnings indicate a terminal auth failure requiring a new key from the
+// operator.
+func needsAuthKeyReissue(backendState string, healthWarnings []string) bool {
+	if backendState == ipn.NeedsLogin.String() {
+		return true
+	}
+	loginWarnableCode := string(health.LoginStateWarnable.Code)
+	for _, h := range healthWarnings {
+		if strings.Contains(h, loginWarnableCode) {
+			return true
+		}
+	}
+	return false
+}
+
+// checkInitialAuthState checks if the tsnet server is in an auth failure state
+// immediately after coming up. Returns true if auth key reissue is needed.
+func checkInitialAuthState(ctx context.Context, lc *local.Client) (bool, error) {
+	status, err := lc.Status(ctx)
+	if err != nil {
+		return false, fmt.Errorf("error getting status: %w", err)
+	}
+	return needsAuthKeyReissue(status.BackendState, status.Health), nil
+}
+
+// monitorAuthHealth watches the IPN bus for auth failures and triggers reissue
+// when needed. Runs until context is cancelled or auth failure is detected.
+func monitorAuthHealth(ctx context.Context, lc *local.Client, reissueCh chan<- struct{}, logger *zap.SugaredLogger) error {
+	w, err := lc.WatchIPNBus(ctx, ipn.NotifyInitialHealthState)
+	if err != nil {
+		return fmt.Errorf("failed to watch IPN bus for auth health: %w", err)
+	}
+	defer w.Close()
+
+	for {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+		n, err := w.Next()
+		if err != nil {
+			return err
+		}
+		if n.Health != nil {
+			if _, ok := n.Health.Warnings[health.LoginStateWarnable.Code]; ok {
+				logger.Info("Auth key failed to authenticate (may be expired or single-use), requesting new key from operator")
+				select {
+				case reissueCh <- struct{}{}:
+				case <-ctx.Done():
+				}
+				return nil
+			}
+		}
+	}
+}
+
+// handleAuthKeyReissue orchestrates the auth key reissue flow:
+// 1. Disconnect from control
+// 2. Set reissue marker in state Secret
+// 3. Wait for operator to provide new key
+// 4. Exit cleanly (Kubernetes will restart the pod with the new key)
+func handleAuthKeyReissue(ctx context.Context, lc *local.Client, kc kubeclient.Client, stateSecretName string, currentAuthKey string, cfgChan <-chan *conf.Config, logger *zap.SugaredLogger) error {
+	if err := lc.DisconnectControl(ctx); err != nil {
+		return fmt.Errorf("error disconnecting from control: %w", err)
+	}
+	if err := authkey.SetReissueAuthKey(ctx, kc, stateSecretName, currentAuthKey, k8sProxyFieldManager); err != nil {
+		return fmt.Errorf("failed to set reissue_authkey in Kubernetes Secret: %w", err)
+	}
+
+	var mu sync.Mutex
+	var latestAuthKey string
+	notify := make(chan struct{}, 1)
+
+	// we use this go func to abstract away conf.Config from the shared function
+	go func() {
+		for cfg := range cfgChan {
+			if cfg.Parsed.AuthKey != nil {
+				mu.Lock()
+				latestAuthKey = *cfg.Parsed.AuthKey
+				mu.Unlock()
+				select {
+				case notify <- struct{}{}:
+				default:
+				}
+			}
+		}
+	}()
+
+	getAuthKey := func() string {
+		mu.Lock()
+		defer mu.Unlock()
+		return latestAuthKey
+	}
+	clearFn := func(ctx context.Context) error {
+		return authkey.ClearReissueAuthKey(ctx, kc, stateSecretName, k8sProxyFieldManager)
+	}
+
+	return authkey.WaitForAuthKeyReissue(ctx, currentAuthKey, 10*time.Minute, getAuthKey, clearFn, notify)
+}

cmd/k8s-proxy/kube_test.go

@@ -0,0 +1,141 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build !plan9
+
+package main
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"github.com/google/go-cmp/cmp"
+	"tailscale.com/health"
+	"tailscale.com/kube/kubeapi"
+	"tailscale.com/kube/kubeclient"
+	"tailscale.com/kube/kubetypes"
+	"tailscale.com/tailcfg"
+)
+
+func TestResetState(t *testing.T) {
+	tests := []struct {
+		name          string
+		existingData  map[string][]byte
+		podUID        string
+		configAuthKey string
+		wantPatched   map[string][]byte
+	}{
+		{
+			name: "sets_capver_and_pod_uid",
+			existingData: map[string][]byte{
+				kubetypes.KeyDeviceID:   []byte("device-123"),
+				kubetypes.KeyDeviceFQDN: []byte("node.tailnet"),
+				kubetypes.KeyDeviceIPs:  []byte(`["100.64.0.1"]`),
+			},
+			podUID:        "pod-123",
+			configAuthKey: "new-key",
+			wantPatched: map[string][]byte{
+				kubetypes.KeyPodUID: []byte("pod-123"),
+			},
+		},
+		{
+			name: "clears_reissue_marker_when_actioned",
+			existingData: map[string][]byte{
+				kubetypes.KeyReissueAuthkey: []byte("old-key"),
+			},
+			podUID:        "pod-123",
+			configAuthKey: "new-key",
+			wantPatched: map[string][]byte{
+				kubetypes.KeyPodUID:         []byte("pod-123"),
+				kubetypes.KeyReissueAuthkey: nil,
+			},
+		},
+		{
+			name: "keeps_reissue_marker_when_not_actioned",
+			existingData: map[string][]byte{
+				kubetypes.KeyReissueAuthkey: []byte("old-key"),
+			},
+			podUID:        "pod-123",
+			configAuthKey: "old-key",
+			wantPatched: map[string][]byte{
+				kubetypes.KeyPodUID: []byte("pod-123"),
+			},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			tt.wantPatched[kubetypes.KeyCapVer] = fmt.Appendf(nil, "%d", tailcfg.CurrentCapabilityVersion)
+
+			var patched map[string][]byte
+			kc := &kubeclient.FakeClient{
+				GetSecretImpl: func(ctx context.Context, name string) (*kubeapi.Secret, error) {
+					return &kubeapi.Secret{Data: tt.existingData}, nil
+				},
+				StrategicMergePatchSecretImpl: func(ctx context.Context, name string, s *kubeapi.Secret, fm string) error {
+					patched = s.Data
+					return nil
+				},
+			}
+
+			err := resetState(context.Background(), kc, "test-secret", tt.podUID, tt.configAuthKey)
+			if err != nil {
+				t.Fatalf("resetState() error = %v", err)
+			}
+
+			if diff := cmp.Diff(tt.wantPatched, patched); diff != "" {
+				t.Errorf("resetState() mismatch (-want +got):\n%s", diff)
+			}
+		})
+	}
+}
+
+func TestNeedsAuthKeyReissue(t *testing.T) {
+	loginWarnableCode := string(health.LoginStateWarnable.Code)
+
+	tests := []struct {
+		name         string
+		backendState string
+		health       []string
+		want         bool
+	}{
+		{
+			name:         "running_healthy",
+			backendState: "Running",
+			want:         false,
+		},
+		{
+			name:         "needs_login",
+			backendState: "NeedsLogin",
+			want:         true,
+		},
+		{
+			name:         "running_with_login_warning",
+			backendState: "Running",
+			health:       []string{"warning: " + loginWarnableCode + ": you are logged out"},
+			want:         true,
+		},
+		{
+			name:         "running_with_unrelated_warning",
+			backendState: "Running",
+			health:       []string{"dns-not-working"},
+			want:         false,
+		},
+		{
+			name:         "running_no_warnings",
+			backendState: "Running",
+			health:       nil,
+			want:         false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := needsAuthKeyReissue(tt.backendState, tt.health)
+			if got != tt.want {
+				t.Errorf("needsAuthKeyReissue() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}

cmd/nardump/nardump.go

@@ -9,22 +9,13 @@
 // git-pull-oss.sh having Nix available.
 package main
 
-// For the format, see:
-// See https://gist.github.com/jbeda/5c79d2b1434f0018d693
-
 import (
-	"bufio"
-	"crypto/sha256"
-	"encoding/base64"
-	"encoding/binary"
 	"flag"
 	"fmt"
-	"io"
-	"io/fs"
 	"log"
 	"os"
-	"path"
+
-	"sort"
+	"tailscale.com/cmd/nardump/nardump"
 )
 
 var sri = flag.Bool("sri", false, "print SRI")
@@ -34,167 +25,16 @@ func main() {
 	if flag.NArg() != 1 {
 		log.Fatal("usage: nardump <dir>")
 	}
-	arg := flag.Arg(0)
+	fsys := os.DirFS(flag.Arg(0))
-	if err := os.Chdir(arg); err != nil {
-		log.Fatal(err)
-	}
 	if *sri {
-		hash := sha256.New()
+		s, err := nardump.SRI(fsys)
-		if err := writeNAR(hash, os.DirFS(".")); err != nil {
+		if err != nil {
 			log.Fatal(err)
 		}
-		fmt.Printf("sha256-%s\n", base64.StdEncoding.EncodeToString(hash.Sum(nil)))
+		fmt.Println(s)
 		return
 	}
-	bw := bufio.NewWriter(os.Stdout)
+	if err := nardump.WriteNAR(os.Stdout, fsys); err != nil {
-	if err := writeNAR(bw, os.DirFS(".")); err != nil {
 		log.Fatal(err)
 	}
-	bw.Flush()
-}
-
-// writeNARError is a sentinel panic type that's recovered by writeNAR
-// and converted into the wrapped error.
-type writeNARError struct{ err error }
-
-// narWriter writes NAR files.
-type narWriter struct {
-	w  io.Writer
-	fs fs.FS
-}
-
-// writeNAR writes a NAR file to w from the root of fs.
-func writeNAR(w io.Writer, fs fs.FS) (err error) {
-	defer func() {
-		if e := recover(); e != nil {
-			if we, ok := e.(writeNARError); ok {
-				err = we.err
-				return
-			}
-			panic(e)
-		}
-	}()
-	nw := &narWriter{w: w, fs: fs}
-	nw.str("nix-archive-1")
-	return nw.writeDir(".")
-}
-
-func (nw *narWriter) writeDir(dirPath string) error {
-	ents, err := fs.ReadDir(nw.fs, dirPath)
-	if err != nil {
-		return err
-	}
-	sort.Slice(ents, func(i, j int) bool {
-		return ents[i].Name() < ents[j].Name()
-	})
-	nw.str("(")
-	nw.str("type")
-	nw.str("directory")
-	for _, ent := range ents {
-		nw.str("entry")
-		nw.str("(")
-		nw.str("name")
-		nw.str(ent.Name())
-		nw.str("node")
-		mode := ent.Type()
-		sub := path.Join(dirPath, ent.Name())
-		var err error
-		switch {
-		case mode.IsDir():
-			err = nw.writeDir(sub)
-		case mode.IsRegular():
-			err = nw.writeRegular(sub)
-		case mode&os.ModeSymlink != 0:
-			err = nw.writeSymlink(sub)
-		default:
-			return fmt.Errorf("unsupported file type %v at %q", sub, mode)
-		}
-		if err != nil {
-			return err
-		}
-		nw.str(")")
-	}
-	nw.str(")")
-	return nil
-}
-
-func (nw *narWriter) writeRegular(path string) error {
-	nw.str("(")
-	nw.str("type")
-	nw.str("regular")
-	fi, err := fs.Stat(nw.fs, path)
-	if err != nil {
-		return err
-	}
-	if fi.Mode()&0111 != 0 {
-		nw.str("executable")
-		nw.str("")
-	}
-	contents, err := fs.ReadFile(nw.fs, path)
-	if err != nil {
-		return err
-	}
-	nw.str("contents")
-	if err := writeBytes(nw.w, contents); err != nil {
-		return err
-	}
-	nw.str(")")
-	return nil
-}
-
-func (nw *narWriter) writeSymlink(path string) error {
-	nw.str("(")
-	nw.str("type")
-	nw.str("symlink")
-	nw.str("target")
-	// broken symlinks are valid in a nar
-	// given we do os.chdir(dir) and os.dirfs(".") above
-	// readlink now resolves relative links even if they are broken
-	link, err := os.Readlink(path)
-	if err != nil {
-		return err
-	}
-	nw.str(link)
-	nw.str(")")
-	return nil
-}
-
-func (nw *narWriter) str(s string) {
-	if err := writeString(nw.w, s); err != nil {
-		panic(writeNARError{err})
-	}
-}
-
-func writeString(w io.Writer, s string) error {
-	var buf [8]byte
-	binary.LittleEndian.PutUint64(buf[:], uint64(len(s)))
-	if _, err := w.Write(buf[:]); err != nil {
-		return err
-	}
-	if _, err := io.WriteString(w, s); err != nil {
-		return err
-	}
-	return writePad(w, len(s))
-}
-
-func writeBytes(w io.Writer, b []byte) error {
-	var buf [8]byte
-	binary.LittleEndian.PutUint64(buf[:], uint64(len(b)))
-	if _, err := w.Write(buf[:]); err != nil {
-		return err
-	}
-	if _, err := w.Write(b); err != nil {
-		return err
-	}
-	return writePad(w, len(b))
-}
-
-func writePad(w io.Writer, n int) error {
-	pad := n % 8
-	if pad == 0 {
-		return nil
-	}
-	var zeroes [8]byte
-	_, err := w.Write(zeroes[:8-pad])
-	return err
 }

cmd/nardump/nardump/nardump.go

@@ -0,0 +1,193 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+// Package nardump writes a NAR (Nix Archive) representation of an
+// fs.FS to an io.Writer, or summarizes it as a Subresource Integrity
+// hash, as used by Nix flake.nix vendor and toolchain hashes.
+//
+// For the format, see:
+// https://gist.github.com/jbeda/5c79d2b1434f0018d693
+package nardump
+
+import (
+	"bufio"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/binary"
+	"fmt"
+	"io"
+	"io/fs"
+	"path"
+	"sort"
+)
+
+// WriteNAR writes a NAR-encoded representation of fsys, rooted at
+// the FS root, to w.
+//
+// The encoder issues many small writes; if w is not already a
+// *bufio.Writer, WriteNAR wraps it in one and flushes on return so
+// the caller doesn't have to.
+//
+// fsys must implement fs.ReadLinkFS to encode any symlinks it
+// contains; os.DirFS satisfies this on Go 1.25+.
+func WriteNAR(w io.Writer, fsys fs.FS) (err error) {
+	defer func() {
+		if e := recover(); e != nil {
+			if we, ok := e.(writeNARError); ok {
+				err = we.err
+				return
+			}
+			panic(e)
+		}
+	}()
+	bw, ok := w.(*bufio.Writer)
+	if !ok {
+		bw = bufio.NewWriter(w)
+		defer func() {
+			if flushErr := bw.Flush(); err == nil {
+				err = flushErr
+			}
+		}()
+	}
+	nw := &narWriter{w: bw, fs: fsys}
+	nw.str("nix-archive-1")
+	return nw.writeDir(".")
+}
+
+// SRI returns the Subresource Integrity hash of the NAR encoding of
+// fsys, in the form "sha256-<base64>". This is the format Nix
+// expects for vendorHash and similar fields.
+func SRI(fsys fs.FS) (string, error) {
+	h := sha256.New()
+	if err := WriteNAR(h, fsys); err != nil {
+		return "", err
+	}
+	return "sha256-" + base64.StdEncoding.EncodeToString(h.Sum(nil)), nil
+}
+
+// writeNARError is a sentinel panic type that's recovered by
+// WriteNAR and converted into the wrapped error.
+type writeNARError struct{ err error }
+
+// narWriter writes NAR files.
+type narWriter struct {
+	w  io.Writer
+	fs fs.FS
+}
+
+func (nw *narWriter) writeDir(dirPath string) error {
+	ents, err := fs.ReadDir(nw.fs, dirPath)
+	if err != nil {
+		return err
+	}
+	sort.Slice(ents, func(i, j int) bool {
+		return ents[i].Name() < ents[j].Name()
+	})
+	nw.str("(")
+	nw.str("type")
+	nw.str("directory")
+	for _, ent := range ents {
+		nw.str("entry")
+		nw.str("(")
+		nw.str("name")
+		nw.str(ent.Name())
+		nw.str("node")
+		mode := ent.Type()
+		sub := path.Join(dirPath, ent.Name())
+		var err error
+		switch {
+		case mode.IsDir():
+			err = nw.writeDir(sub)
+		case mode.IsRegular():
+			err = nw.writeRegular(sub)
+		case mode&fs.ModeSymlink != 0:
+			err = nw.writeSymlink(sub)
+		default:
+			return fmt.Errorf("unsupported file type %v at %q", sub, mode)
+		}
+		if err != nil {
+			return err
+		}
+		nw.str(")")
+	}
+	nw.str(")")
+	return nil
+}
+
+func (nw *narWriter) writeRegular(p string) error {
+	nw.str("(")
+	nw.str("type")
+	nw.str("regular")
+	fi, err := fs.Stat(nw.fs, p)
+	if err != nil {
+		return err
+	}
+	if fi.Mode()&0111 != 0 {
+		nw.str("executable")
+		nw.str("")
+	}
+	contents, err := fs.ReadFile(nw.fs, p)
+	if err != nil {
+		return err
+	}
+	nw.str("contents")
+	if err := writeBytes(nw.w, contents); err != nil {
+		return err
+	}
+	nw.str(")")
+	return nil
+}
+
+func (nw *narWriter) writeSymlink(p string) error {
+	nw.str("(")
+	nw.str("type")
+	nw.str("symlink")
+	nw.str("target")
+	link, err := fs.ReadLink(nw.fs, p)
+	if err != nil {
+		return err
+	}
+	nw.str(link)
+	nw.str(")")
+	return nil
+}
+
+func (nw *narWriter) str(s string) {
+	if err := writeString(nw.w, s); err != nil {
+		panic(writeNARError{err})
+	}
+}
+
+func writeString(w io.Writer, s string) error {
+	var buf [8]byte
+	binary.LittleEndian.PutUint64(buf[:], uint64(len(s)))
+	if _, err := w.Write(buf[:]); err != nil {
+		return err
+	}
+	if _, err := io.WriteString(w, s); err != nil {
+		return err
+	}
+	return writePad(w, len(s))
+}
+
+func writeBytes(w io.Writer, b []byte) error {
+	var buf [8]byte
+	binary.LittleEndian.PutUint64(buf[:], uint64(len(b)))
+	if _, err := w.Write(buf[:]); err != nil {
+		return err
+	}
+	if _, err := w.Write(b); err != nil {
+		return err
+	}
+	return writePad(w, len(b))
+}
+
+func writePad(w io.Writer, n int) error {
+	pad := n % 8
+	if pad == 0 {
+		return nil
+	}
+	var zeroes [8]byte
+	_, err := w.Write(zeroes[:8-pad])
+	return err
+}

cmd/nardump/nardump/nardump_test.go

@@ -0,0 +1,55 @@
+// Copyright (c) Tailscale Inc & contributors
+// SPDX-License-Identifier: BSD-3-Clause
+
+package nardump
+
+import (
+	"crypto/sha256"
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"testing"
+)
+
+// setupTmpdir sets up a known golden layout, covering all allowed file/folder types in a nar.
+func setupTmpdir(t *testing.T) string {
+	t.Helper()
+	tmpdir := t.TempDir()
+	must := func(err error) {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	must(os.MkdirAll(filepath.Join(tmpdir, "sub/dir"), 0755))
+	must(os.Symlink("brokenfile", filepath.Join(tmpdir, "brokenlink")))
+	must(os.Symlink("sub/dir", filepath.Join(tmpdir, "dirl")))
+	must(os.Symlink("/abs/nonexistentdir", filepath.Join(tmpdir, "dirb")))
+	f, err := os.Create(filepath.Join(tmpdir, "sub/dir/file1"))
+	must(err)
+	f.Close()
+	f, err = os.Create(filepath.Join(tmpdir, "file2m"))
+	must(err)
+	must(f.Truncate(2 * 1024 * 1024))
+	f.Close()
+	must(os.Symlink("../file2m", filepath.Join(tmpdir, "sub/goodlink")))
+	return tmpdir
+}
+
+func TestWriteNAR(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		// Skip test on Windows as the Nix package manager is not supported on this platform
+		t.Skip("nix package manager is not available on Windows")
+	}
+	dir := setupTmpdir(t)
+	// obtained via `nix-store --dump /tmp/... | sha256sum` of the above test dir
+	const expected = "727613a36f41030e93a4abf2649c3ec64a2757ccff364e3f6f7d544eb976e442"
+	h := sha256.New()
+	if err := WriteNAR(h, os.DirFS(dir)); err != nil {
+		t.Fatal(err)
+	}
+	if got := fmt.Sprintf("%x", h.Sum(nil)); got != expected {
+		t.Fatalf("sha256sum of nar: got %s, want %s", got, expected)
+	}
+}

cmd/nardump/nardump_test.go

@@ -1,52 +0,0 @@
-// Copyright (c) Tailscale Inc & contributors
-// SPDX-License-Identifier: BSD-3-Clause
-
-package main
-
-import (
-	"crypto/sha256"
-	"fmt"
-	"os"
-	"runtime"
-	"testing"
-)
-
-// setupTmpdir sets up a known golden layout, covering all allowed file/folder types in a nar
-func setupTmpdir(t *testing.T) string {
-	tmpdir := t.TempDir()
-	pwd, _ := os.Getwd()
-	os.Chdir(tmpdir)
-	defer os.Chdir(pwd)
-	os.MkdirAll("sub/dir", 0755)
-	os.Symlink("brokenfile", "brokenlink")
-	os.Symlink("sub/dir", "dirl")
-	os.Symlink("/abs/nonexistentdir", "dirb")
-	os.Create("sub/dir/file1")
-	f, _ := os.Create("file2m")
-	_ = f.Truncate(2 * 1024 * 1024)
-	f.Close()
-	os.Symlink("../file2m", "sub/goodlink")
-	return tmpdir
-}
-
-func TestWriteNar(t *testing.T) {
-	if runtime.GOOS == "windows" {
-		// Skip test on Windows as the Nix package manager is not supported on this platform
-		t.Skip("nix package manager is not available on Windows")
-	}
-	dir := setupTmpdir(t)
-	t.Run("nar", func(t *testing.T) {
-		// obtained via `nix-store --dump /tmp/... | sha256sum` of the above test dir
-		expected := "727613a36f41030e93a4abf2649c3ec64a2757ccff364e3f6f7d544eb976e442"
-		h := sha256.New()
-		os.Chdir(dir)
-		err := writeNAR(h, os.DirFS("."))
-		if err != nil {
-			t.Fatal(err)
-		}
-		hash := fmt.Sprintf("%x", h.Sum(nil))
-		if expected != hash {
-			t.Fatal("sha256sum of nar not matched", hash, expected)
-		}
-	})
-}

cmd/pgproxy/pgproxy.go

@@ -291,7 +291,7 @@ func (p *proxy) serve(sessionID int64, c net.Conn) error {
 			Certificates: p.downstreamCert,
 			MinVersion:   tls.VersionTLS12,
 		})
-		if err = uptc.HandshakeContext(ctx); err != nil {
+		if err = s.HandshakeContext(ctx); err != nil {
 			p.errors.Add("client-tls", 1)
 			return fmt.Errorf("client TLS handshake: %v", err)
 		}

cmd/sniproxy/sniproxy.go

@@ -138,9 +138,9 @@ func run(ctx context.Context, ts *tsnet.Server, wgPort int, hostname string, pro
 	}
 
 	// Finally, start mainloop to configure app connector based on information
-	// in the netmap.
+	// in the self node's CapMap. We set NotifyInitialNetMap so the first
-	// We set the NotifyInitialNetMap flag so we will always get woken with the
+	// Notify carries the current self node (now via Notify.SelfChange);
-	// current netmap, before only being woken on changes.
+	// subsequent self changes wake us up too.
 	bus, err := lc.WatchIPNBus(ctx, ipn.NotifyWatchEngineUpdates|ipn.NotifyInitialNetMap)
 	if err != nil {
 		log.Fatalf("watching IPN bus: %v", err)
@@ -155,10 +155,13 @@ func run(ctx context.Context, ts *tsnet.Server, wgPort int, hostname string, pro
 			log.Fatalf("reading IPN bus: %v", err)
 		}
 
-		// NetMap contains app-connector configuration
+		self := msg.SelfChange
-		if nm := msg.NetMap; nm != nil && nm.SelfNode.Valid() {
+		if self == nil {
+			continue
+		}
 		var c appctype.AppConnectorConfig
-			nmConf, err := tailcfg.UnmarshalNodeCapViewJSON[appctype.AppConnectorConfig](nm.SelfNode.CapMap(), configCapKey)
+		// View() lets us reuse the existing CapView decoder.
+		nmConf, err := tailcfg.UnmarshalNodeCapViewJSON[appctype.AppConnectorConfig](self.View().CapMap(), configCapKey)
 		if err != nil {
 			log.Printf("failed to read app connector configuration from coordination server: %v", err)
 		} else if len(nmConf) > 0 {
@@ -177,7 +180,6 @@ func run(ctx context.Context, ts *tsnet.Server, wgPort int, hostname string, pro
 		s.mergeConfigFromFlags(&c, ports, forwards)
 		s.srv.Configure(&c)
 	}
-	}
 }
 
 type sniproxy struct {

cmd/systray/systray.go

@@ -15,9 +15,11 @@ import (
 )
 
 var socket = flag.String("socket", paths.DefaultTailscaledSocket(), "path to tailscaled socket")
+var theme = flag.String("theme", "dark", "color theme for Tailscale icon: dark, dark:nobg, light, light:nobg")
 
 func main() {
 	flag.Parse()
 	lc := &local.Client{Socket: *socket}
+	systray.SetTheme(*theme)
 	new(systray.Menu).Run(lc)
 }

cmd/tailscale/cli/cli.go

@@ -28,6 +28,7 @@ import (
 	"tailscale.com/feature"
 	"tailscale.com/paths"
 	"tailscale.com/util/slicesx"
+	"tailscale.com/util/testenv"
 	"tailscale.com/version/distro"
 )
 
@@ -92,8 +93,8 @@ var localClient = local.Client{
 	Socket: paths.DefaultTailscaledSocket(),
 }
 
-// Run runs the CLI. The args do not include the binary name.
+// RunWithContext runs the CLI. The args do not include the binary name.
-func Run(args []string) (err error) {
+func RunWithContext(ctx context.Context, args []string) (err error) {
 	if runtime.GOOS == "linux" && os.Getenv("GOKRAZY_FIRST_START") == "1" && distro.Get() == distro.Gokrazy && os.Getppid() == 1 && len(args) == 0 {
 		// We're running on gokrazy and the user did not specify 'up'.
 		// Don't run the tailscale CLI and spam logs with usage; just exit.
@@ -163,7 +164,7 @@ func Run(args []string) (err error) {
 		return
 	}
 
-	err = rootCmd.Run(context.Background())
+	err = rootCmd.Run(ctx)
 	if local.IsAccessDeniedError(err) && os.Getuid() != 0 && runtime.GOOS != "windows" {
 		return fmt.Errorf("%v\n\nUse 'sudo tailscale %s'.\nTo not require root, use 'sudo tailscale set --operator=$USER' once.", err, strings.Join(args, " "))
 	}
@@ -173,6 +174,11 @@ func Run(args []string) (err error) {
 	return err
 }
 
+// Run is equivalent to calling [RunWithContext] with the background context.
+func Run(args []string) (err error) {
+	return RunWithContext(context.Background(), args)
+}
+
 type onceFlagValue struct {
 	flag.Value
 	set bool
@@ -194,17 +200,39 @@ func (v *onceFlagValue) IsBoolFlag() bool {
 	return ok && bf.IsBoolFlag()
 }
 
-// noDupFlagify modifies c recursively to make all the
+// noDupFlagify modifies c recursively to make all the flag values be
-// flag values be wrappers that permit setting the value
+// wrappers that permit setting the value at most once. If tb is
-// at most once.
+// non-nil, the original values are restored when the test completes.
-func noDupFlagify(c *ffcli.Command) {
+func noDupFlagify(c *ffcli.Command, tb testenv.TB) {
+	if tb == nil && testenv.InTest() {
+		return
+	}
+	type restore struct {
+		f *flag.Flag
+		v flag.Value
+	}
+	var restores []restore
+	var walk func(*ffcli.Command)
+	walk = func(c *ffcli.Command) {
 		if c.FlagSet != nil {
 			c.FlagSet.VisitAll(func(f *flag.Flag) {
+				if tb != nil {
+					restores = append(restores, restore{f, f.Value})
+				}
 				f.Value = &onceFlagValue{Value: f.Value}
 			})
 		}
 		for _, sub := range c.Subcommands {
-		noDupFlagify(sub)
+			walk(sub)
+		}
+	}
+	walk(c)
+	if tb != nil {
+		tb.Cleanup(func() {
+			for _, r := range restores {
+				r.f.Value = r.v
+			}
+		})
 	}
 }
 
@@ -221,7 +249,7 @@ var (
 	_ func() *ffcli.Command
 )
 
-func newRootCmd() *ffcli.Command {
+func newRootCmd(tb ...testenv.TB) *ffcli.Command {
 	rootfs := newFlagSet("tailscale")
 	rootfs.Func("socket", "path to tailscaled socket", func(s string) error {
 		localClient.Socket = s
@@ -303,7 +331,11 @@ change in the future.
 	})
 
 	ffcomplete.Inject(rootCmd, func(c *ffcli.Command) { c.LongHelp = hidden + c.LongHelp }, usageFunc)
-	noDupFlagify(rootCmd)
+	var t testenv.TB
+	if len(tb) > 0 {
+		t = tb[0]
+	}
+	noDupFlagify(rootCmd, t)
 	return rootCmd
 }
 

cmd/tailscale/cli/cli_test.go

@@ -779,11 +779,43 @@ func TestPrefsFromUpArgs(t *testing.T) {
 			wantErr: `--exit-node-allow-lan-access can only be used with --exit-node`,
 		},
 		{
-			name: "error_tag_prefix",
+			name: "error_tag_bad_prefix",
 			args: upArgsT{
-				advertiseTags: "foo",
+				advertiseTags: "notatag:foo",
+			},
+			wantErr: `tag: "notatag:foo": tags must start with 'tag:'`,
+		},
+		{
+			name: "tag_auto_prefix",
+			args: upArgsFromOSArgs("linux", "--advertise-tags=foo,bar"),
+			want: &ipn.Prefs{
+				ControlURL:          ipn.DefaultControlURL,
+				WantRunning:         true,
+				CorpDNS:             true,
+				AdvertiseTags:       []string{"tag:foo", "tag:bar"},
+				NoSNAT:              false,
+				NoStatefulFiltering: "true",
+				NetfilterMode:       preftype.NetfilterOn,
+				AutoUpdate: ipn.AutoUpdatePrefs{
+					Check: true,
+				},
+			},
+		},
+		{
+			name: "tag_mixed_prefix",
+			args: upArgsFromOSArgs("linux", "--advertise-tags=tag:foo,bar"),
+			want: &ipn.Prefs{
+				ControlURL:          ipn.DefaultControlURL,
+				WantRunning:         true,
+				CorpDNS:             true,
+				AdvertiseTags:       []string{"tag:foo", "tag:bar"},
+				NoSNAT:              false,
+				NoStatefulFiltering: "true",
+				NetfilterMode:       preftype.NetfilterOn,
+				AutoUpdate: ipn.AutoUpdatePrefs{
+					Check: true,
+				},
 			},
-			wantErr: `tag: "foo": tags must start with 'tag:'`,
 		},
 		{
 			name: "error_long_hostname",
@@ -1618,7 +1650,7 @@ func TestNoDups(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			cmd := newRootCmd()
+			cmd := newRootCmd(t)
 			makeQuietContinueOnError(cmd)
 			err := cmd.Parse(tt.args)
 			if got := fmt.Sprint(err); got != tt.want {

cmd/tailscale/cli/configure-kube.go

@@ -20,10 +20,8 @@ import (
 	"github.com/peterbourgon/ff/v3/ffcli"
 	"k8s.io/client-go/util/homedir"
 	"sigs.k8s.io/yaml"
-	"tailscale.com/ipn"
 	"tailscale.com/ipn/ipnstate"
 	"tailscale.com/tailcfg"
-	"tailscale.com/types/netmap"
 	"tailscale.com/util/dnsname"
 	"tailscale.com/version"
 )
@@ -98,12 +96,12 @@ func runConfigureKubeconfig(ctx context.Context, args []string) error {
 	if st.BackendState != "Running" {
 		return errors.New("Tailscale is not running")
 	}
-	nm, err := getNetMap(ctx)
+	dnsCfg, err := getDNSConfig(ctx)
 	if err != nil {
 		return err
 	}
 
-	targetFQDN, err := nodeOrServiceDNSNameFromArg(st, nm, hostOrFQDNOrIP)
+	targetFQDN, err := nodeOrServiceDNSNameFromArg(st, dnsCfg, hostOrFQDNOrIP)
 	if err != nil {
 		return err
 	}
@@ -240,14 +238,14 @@ func setKubeconfigForPeer(scheme, fqdn, filePath string) error {
 // nodeOrServiceDNSNameFromArg returns the PeerStatus.DNSName value from a peer
 // in st that matches the input arg which can be a base name, full DNS name, or
 // an IP. If none is found, it looks for a Tailscale Service
-func nodeOrServiceDNSNameFromArg(st *ipnstate.Status, nm *netmap.NetworkMap, arg string) (string, error) {
+func nodeOrServiceDNSNameFromArg(st *ipnstate.Status, dns *tailcfg.DNSConfig, arg string) (string, error) {
 	// First check for a node DNS name.
 	if dnsName, ok := nodeDNSNameFromArg(st, arg); ok {
 		return dnsName, nil
 	}
 
 	// If not found, check for a Tailscale Service DNS name.
-	rec, ok := serviceDNSRecordFromNetMap(nm, arg)
+	rec, ok := serviceDNSRecordFromDNSConfig(dns, arg)
 	if !ok {
 		return "", fmt.Errorf("no peer found for %q", arg)
 	}
@@ -269,25 +267,13 @@ func nodeOrServiceDNSNameFromArg(st *ipnstate.Status, nm *netmap.NetworkMap, arg
 	return "", fmt.Errorf("%q is in MagicDNS, but is not currently reachable on any known peer", arg)
 }
 
-func getNetMap(ctx context.Context) (*netmap.NetworkMap, error) {
+func getDNSConfig(ctx context.Context) (*tailcfg.DNSConfig, error) {
 	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
 	defer cancel()
-
+	return localClient.DNSConfig(ctx)
-	watcher, err := localClient.WatchIPNBus(ctx, ipn.NotifyInitialNetMap)
-	if err != nil {
-		return nil, err
-	}
-	defer watcher.Close()
-
-	n, err := watcher.Next()
-	if err != nil {
-		return nil, err
-	}
-
-	return n.NetMap, nil
 }
 
-func serviceDNSRecordFromNetMap(nm *netmap.NetworkMap, arg string) (rec tailcfg.DNSRecord, ok bool) {
+func serviceDNSRecordFromDNSConfig(dns *tailcfg.DNSConfig, arg string) (rec tailcfg.DNSRecord, ok bool) {
 	argIP, _ := netip.ParseAddr(arg)
 	argFQDN, err := dnsname.ToFQDN(arg)
 	argFQDNValid := err == nil
@@ -295,7 +281,7 @@ func serviceDNSRecordFromNetMap(nm *netmap.NetworkMap, arg string) (rec tailcfg.
 		return rec, false
 	}
 
-	for _, rec := range nm.DNS.ExtraRecords {
+	for _, rec := range dns.ExtraRecords {
 		if argIP.IsValid() {
 			recIP, _ := netip.ParseAddr(rec.Value)
 			if recIP == argIP {

cmd/tailscale/cli/configure_linux.go

@@ -18,7 +18,7 @@ func init() {
 	maybeSystrayCmd = systrayConfigCmd
 }
 
-var systrayArgs struct {
+var configSystrayArgs struct {
 	initSystem     string
 	installStartup bool
 }
@@ -32,7 +32,7 @@ func systrayConfigCmd() *ffcli.Command {
 		Exec:       configureSystray,
 		FlagSet: (func() *flag.FlagSet {
 			fs := newFlagSet("systray")
-			fs.StringVar(&systrayArgs.initSystem, "enable-startup", "",
+			fs.StringVar(&configSystrayArgs.initSystem, "enable-startup", "",
 				"Install startup script for init system. Currently supported systems are [systemd, freedesktop].")
 			return fs
 		})(),
@@ -40,8 +40,8 @@ func systrayConfigCmd() *ffcli.Command {
 }
 
 func configureSystray(_ context.Context, _ []string) error {
-	if systrayArgs.initSystem != "" {
+	if configSystrayArgs.initSystem != "" {
-		if err := systray.InstallStartupScript(systrayArgs.initSystem); err != nil {
+		if err := systray.InstallStartupScript(configSystrayArgs.initSystem); err != nil {
 			fmt.Printf("%s\n\n", err.Error())
 			return flag.ErrHelp
 		}

cmd/tailscale/cli/debug.go

@@ -670,18 +670,11 @@ func runNetmap(ctx context.Context, args []string) error {
 	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
 	defer cancel()
 
-	var mask ipn.NotifyWatchOpt = ipn.NotifyInitialNetMap
+	raw, err := localClient.DebugResultJSON(ctx, "current-netmap")
-	watcher, err := localClient.WatchIPNBus(ctx, mask)
 	if err != nil {
 		return err
 	}
-	defer watcher.Close()
+	j, _ := json.MarshalIndent(raw, "", "\t")
-
-	n, err := watcher.Next()
-	if err != nil {
-		return err
-	}
-	j, _ := json.MarshalIndent(n.NetMap, "", "\t")
 	fmt.Printf("%s\n", j)
 	return nil
 }

cmd/tailscale/cli/dns-status.go

@@ -14,9 +14,7 @@ import (
 
 	"github.com/peterbourgon/ff/v3/ffcli"
 	"tailscale.com/cmd/tailscale/cli/jsonoutput"
-	"tailscale.com/ipn"
 	"tailscale.com/types/dnstype"
-	"tailscale.com/types/netmap"
 )
 
 var dnsStatusCmd = &ffcli.Command{
@@ -120,11 +118,10 @@ func runDNSStatus(ctx context.Context, args []string) error {
 			SelfDNSName:     s.Self.DNSName,
 		}
 
-		netMap, err := fetchNetMap()
+		dnsConfig, err := localClient.DNSConfig(ctx)
 		if err != nil {
-			return fmt.Errorf("failed to fetch network map: %w", err)
+			return fmt.Errorf("failed to fetch DNS config: %w", err)
 		}
-		dnsConfig := netMap.DNS
 
 		for _, r := range dnsConfig.Resolvers {
 			data.Resolvers = append(data.Resolvers, makeDNSResolverInfo(r))
@@ -357,19 +354,3 @@ func formatDNSStatusText(data *jsonoutput.DNSStatusResult, all bool) string {
 	fmt.Fprintf(&sb, "[this is a preliminary version of this command; the output format may change in the future]\n")
 	return sb.String()
 }
-
-func fetchNetMap() (netMap *netmap.NetworkMap, err error) {
-	w, err := localClient.WatchIPNBus(context.Background(), ipn.NotifyInitialNetMap)
-	if err != nil {
-		return nil, err
-	}
-	defer w.Close()
-	notify, err := w.Next()
-	if err != nil {
-		return nil, err
-	}
-	if notify.NetMap == nil {
-		return nil, fmt.Errorf("no network map yet available, please try again later")
-	}
-	return notify.NetMap, nil
-}

cmd/tailscale/cli/file.go

@@ -32,6 +32,7 @@ import (
 	"tailscale.com/client/tailscale/apitype"
 	"tailscale.com/cmd/tailscale/cli/ffcomplete"
 	"tailscale.com/envknob"
+	"tailscale.com/ipn"
 	"tailscale.com/ipn/ipnstate"
 	"tailscale.com/net/tsaddr"
 	"tailscale.com/tailcfg"
@@ -78,6 +79,7 @@ var fileCpCmd = &ffcli.Command{
 		fs.StringVar(&cpArgs.name, "name", "", "alternate filename to use, especially useful when <file> is \"-\" (stdin)")
 		fs.BoolVar(&cpArgs.verbose, "verbose", false, "verbose output")
 		fs.BoolVar(&cpArgs.targets, "targets", false, "list possible file cp targets")
+		fs.DurationVar(&cpArgs.updateInterval, "update-interval", 250*time.Millisecond, "how often to repaint the progress line; zero or negative disables progress display entirely")
 		return fs
 	})(),
 }
@@ -86,6 +88,7 @@ var cpArgs struct {
 	name           string
 	verbose        bool
 	targets        bool
+	updateInterval time.Duration
 }
 
 func runCp(ctx context.Context, args []string) error {
@@ -119,9 +122,6 @@ func runCp(ctx context.Context, args []string) error {
 	if err != nil {
 		return fmt.Errorf("can't send to %s: %v", target, err)
 	}
-	if isOffline {
-		fmt.Fprintf(Stderr, "# warning: %s is offline\n", target)
-	}
 
 	if len(files) > 1 {
 		if cpArgs.name != "" {
@@ -132,7 +132,51 @@ func runCp(ctx context.Context, args []string) error {
 		}
 	}
 
-	for _, fileArg := range files {
+	// outFiles tracks per-name push state, populated by a goroutine subscribed
+	// to the IPN bus. tailscaled's OutgoingFile.Sent is the bytes-pulled-toward-
+	// peerAPI signal; it stays at 0 until the peerAPI request body is actually
+	// being read, which is what we want both for the progress display and for
+	// disarming the offline warning. The CLI's local-side bytes counter would
+	// say "100% sent" the moment net/http buffers a small body into the local
+	// unix-socket conn to tailscaled, well before the peer has heard a thing.
+	type pushState struct {
+		sent      atomic.Int64
+		warnTimer *time.Timer // disarmed on first byte sent to peerAPI; nil after
+	}
+	var (
+		outMu    sync.Mutex
+		outFiles = map[string]*pushState{} // keyed by file name
+	)
+
+	busCtx, cancelBus := context.WithCancel(ctx)
+	defer cancelBus()
+	go watchOutgoingFiles(busCtx, stableID, func(name string, sent int64) {
+		outMu.Lock()
+		ps := outFiles[name]
+		outMu.Unlock()
+		if ps == nil {
+			return
+		}
+		// Only ever advance ps.sent forward. Bus updates can arrive late
+		// (after the success path below has already written contentLength
+		// to ps.sent for an instant final-100% paint), so we'd otherwise
+		// regress the count and the progress printer would compute a
+		// negative delta on its next tick.
+		for {
+			old := ps.sent.Load()
+			if sent <= old {
+				return
+			}
+			if ps.sent.CompareAndSwap(old, sent) {
+				if old == 0 && ps.warnTimer != nil {
+					ps.warnTimer.Stop()
+				}
+				return
+			}
+		}
+	})
+
+	for i, fileArg := range files {
 		var fileContents *countingReader
 		var name = cpArgs.name
 		var contentLength int64 = -1
@@ -175,16 +219,57 @@ func runCp(ctx context.Context, args []string) error {
 			log.Printf("sending %q to %v/%v/%v ...", name, target, ip, stableID)
 		}
 
+		// Register this file with the watcher and, for the first file only,
+		// arm a timer that warns the user if no bytes have flowed to peerAPI
+		// after a few seconds. The watcher disarms it on first byte; PushFile
+		// returning also disarms it (cleanup, below). We don't gate on the
+		// netmap's Online bit (which can lag reality), but we do use it to
+		// pick between two warning messages.
+		ps := &pushState{}
+		if i == 0 {
+			ps.warnTimer = time.AfterFunc(3*time.Second, func() {
+				// vtRestartLine clears whatever (possibly progress) was on
+				// the current line, then we print the warning + \n so the
+				// next progress redraw lands on a fresh line below.
+				const vtRestartLine = "\r\x1b[K"
+				if isOffline {
+					fmt.Fprintf(Stderr, "%s# warning: %s is reportedly offline; trying anyway\n", vtRestartLine, target)
+				} else {
+					fmt.Fprintf(Stderr, "%s# warning: %s is not replying; trying anyway\n", vtRestartLine, target)
+				}
+			})
+		}
+		outMu.Lock()
+		outFiles[name] = ps
+		outMu.Unlock()
+
 		var group sync.WaitGroup
 		ctxProgress, cancelProgress := context.WithCancel(ctx)
 		defer cancelProgress()
-		if isatty.IsTerminal(os.Stderr.Fd()) {
+		if cpArgs.updateInterval > 0 && isatty.IsTerminal(os.Stderr.Fd()) {
-			group.Go(func() { progressPrinter(ctxProgress, name, fileContents.n.Load, contentLength) })
+			group.Go(func() {
+				progressPrinter(ctxProgress, name, ps.sent.Load, contentLength, cpArgs.updateInterval)
+			})
 		}
 
 		err := localClient.PushFile(ctx, stableID, contentLength, name, fileContents)
+		if err == nil {
+			// PushFile can finish faster than the IPN bus delivers a final
+			// OutgoingFile update, leaving the progress display stuck at 0%.
+			// Synthesize a "fully done" count before stopping the printer so
+			// its final paint shows 100%. For stdin (contentLength == -1) we
+			// don't know the size, so fall back to the local read count.
+			if contentLength >= 0 {
+				ps.sent.Store(contentLength)
+			} else {
+				ps.sent.Store(fileContents.n.Load())
+			}
+		}
 		cancelProgress()
 		group.Wait() // wait for progress printer to stop before reporting the error
+		if ps.warnTimer != nil {
+			ps.warnTimer.Stop()
+		}
 		if err != nil {
 			return err
 		}
@@ -195,15 +280,71 @@ func runCp(ctx context.Context, args []string) error {
 	return nil
 }
 
-func progressPrinter(ctx context.Context, name string, contentCount func() int64, contentLength int64) {
+// watchOutgoingFiles subscribes to the IPN bus and invokes onUpdate once
+// per OutgoingFile event for files going to peer. It runs until ctx is
+// done (which runCp does on return) and is best-effort: if the bus
+// subscription fails for any reason, onUpdate simply isn't called and the
+// caller's progress display stays at 0 — exactly the right degradation,
+// since the warning timer will then fire on its normal 3-second deadline.
+func watchOutgoingFiles(ctx context.Context, peer tailcfg.StableNodeID, onUpdate func(name string, sent int64)) {
+	// NotifyPeerChanges opts in to per-peer add/remove notifications so the
+	// bus stays responsive without us also subscribing to the full NetMap,
+	// which we don't read here.
+	w, err := localClient.WatchIPNBus(ctx, ipn.NotifyInitialOutgoingFiles|ipn.NotifyPeerChanges)
+	if err != nil {
+		return
+	}
+	defer w.Close()
+	for {
+		n, err := w.Next()
+		if err != nil {
+			return
+		}
+		for _, of := range n.OutgoingFiles {
+			if of.PeerID != peer {
+				continue
+			}
+			// tailscaled keeps Finished entries in its OutgoingFiles map
+			// across PushFile calls (see feature/taildrop/ext.go), so a
+			// re-send of the same filename will see both the old completed
+			// (Sent == DeclaredSize) entry and the new in-progress one.
+			// Without this filter the watcher's monotonic CAS would latch
+			// onto the old entry's max value and the new transfer would
+			// appear stuck at 100% from the first bus tick.
+			if of.Finished {
+				continue
+			}
+			onUpdate(of.Name, of.Sent)
+		}
+	}
+}
+
+// progressPrinter repaints a single-line transfer progress display every
+// interval. interval must be > 0; runCp's caller gates on the
+// --update-interval flag and skips invoking us when it's <= 0.
+//
+// It returns when ctx is done OR when it detects the transfer is stuck —
+// "stuck" being: contentCount has equalled contentLength with a near-zero
+// rate for >2 seconds. The stuck case prints a final newline so subsequent
+// output (e.g. an error from PushFile) lands on a fresh line below the
+// frozen progress line, instead of being painted over by it.
+func progressPrinter(ctx context.Context, name string, contentCount func() int64, contentLength int64, interval time.Duration) {
 	var rateValueFast, rateValueSlow tsrate.Value
-	rateValueFast.HalfLife = 1 * time.Second  // fast response for rate measurement
+	// tailscaled emits OutgoingFile.Sent updates at ~1 Hz, so most printer
-	rateValueSlow.HalfLife = 10 * time.Second // slow response for ETA measurement
+	// ticks see no delta. With too short a half-life the displayed rate
+	// roughly halves between updates and doubles back when one arrives,
+	// looking jumpy. 5s keeps the swing under ~15% while still settling
+	// within a few seconds of a real change.
+	rateValueFast.HalfLife = 5 * time.Second  // smoothed rate for display
+	rateValueSlow.HalfLife = 10 * time.Second // even slower, for ETA measurement
 	var prevContentCount int64
 	print := func() {
 		currContentCount := contentCount()
-		rateValueFast.Add(float64(currContentCount - prevContentCount))
+		// Clamp so a regression (which shouldn't happen, but tsrate.Value.Add
-		rateValueSlow.Add(float64(currContentCount - prevContentCount))
+		// panics on a negative count) can't take down the CLI.
+		delta := max(currContentCount-prevContentCount, 0)
+		rateValueFast.Add(float64(delta))
+		rateValueSlow.Add(float64(delta))
 		prevContentCount = currContentCount
 
 		const vtRestartLine = "\r\x1b[K"
@@ -215,16 +356,23 @@ func progressPrinter(ctx context.Context, name string, contentCount func() int64
 		if contentLength >= 0 {
 			currContentCount = min(currContentCount, contentLength) // cap at 100%
 			ratioRemain := float64(currContentCount) / float64(contentLength)
+			etaStr := "ETA -"
+			if rate := rateValueSlow.Rate(); rate > 0 {
 				bytesRemain := float64(contentLength - currContentCount)
-			secsRemain := bytesRemain / rateValueSlow.Rate()
+				secsRemain := bytesRemain / rate
 				secs := int(min(max(0, secsRemain), 99*60*60+59+60+59))
+				etaStr = fmt.Sprintf("ETA %02d:%02d:%02d", secs/60/60, (secs/60)%60, secs%60)
+			}
 			fmt.Fprintf(os.Stderr, "    %s    %s",
 				leftPad(fmt.Sprintf("%0.2f%%", 100.0*ratioRemain), len("100.00%")),
-				fmt.Sprintf("ETA %02d:%02d:%02d", secs/60/60, (secs/60)%60, secs%60))
+				etaStr)
 		}
 	}
 
-	tc := time.NewTicker(250 * time.Millisecond)
+	const stuckAfter = 2 * time.Second
+	var fullStartedAt time.Time // when we first observed currCount==contentLength with ~zero rate
+
+	tc := time.NewTicker(interval)
 	defer tc.Stop()
 	print()
 	for {
@@ -235,6 +383,24 @@ func progressPrinter(ctx context.Context, name string, contentCount func() int64
 			return
 		case <-tc.C:
 			print()
+			if contentLength < 0 {
+				continue
+			}
+			currCount := contentCount()
+			rate := rateValueFast.Rate()
+			if currCount >= contentLength && rate < 1 {
+				if fullStartedAt.IsZero() {
+					fullStartedAt = time.Now()
+				} else if time.Since(fullStartedAt) >= stuckAfter {
+					// Transfer is stuck at 100% with no movement. Stop
+					// repainting so we don't keep clobbering anything the
+					// rest of runCp prints (warnings, errors).
+					fmt.Fprintln(os.Stderr)
+					return
+				}
+			} else {
+				fullStartedAt = time.Time{}
+			}
 		}
 	}
 }
@@ -328,7 +494,10 @@ peerLoop:
 		return "", isOffline, errors.New("cannot send files: missing required Taildrop capability")
 
 	case ipnstate.TaildropTargetOffline:
-		return "", isOffline, errors.New("cannot send files: peer is offline")
+		// Don't gate on the server-reported Online bit (which lags reality
+		// and isn't always accurate). runCp probes reachability itself with
+		// TSMP pings.
+		return foundPeer.ID, isOffline, nil
 
 	case ipnstate.TaildropTargetNoPeerInfo:
 		return "", isOffline, errors.New("cannot send files: invalid or unrecognized peer")

cmd/tailscale/cli/jsonoutput/network-lock-log.go

@@ -159,7 +159,7 @@ type expandedAUMV1 struct {
 }
 
 // tkaKeyV1 is the expanded version of a [tka.Key], which describes
-// the public components of a key known to network-lock.
+// the public components of a key known to tailnet-lock.
 type tkaKeyV1 struct {
 	Kind string `json:"Kind,omitzero"`
 

cmd/tailscale/cli/jsonoutput/network-lock-status.go

@@ -116,7 +116,7 @@ type tailnetLockStatusV1Base struct {
 	// Enabled is true if Tailnet Lock is enabled.
 	Enabled bool
 
-	// PublicKey describes the node's network-lock public key.
+	// PublicKey describes the node's tailnet-lock public key.
 	PublicKey string `json:"PublicKey,omitzero"`
 
 	// NodeKey describes the node's current node-key. This field is not
@@ -144,7 +144,7 @@ type tailnetLockEnabledStatusV1 struct {
 	NodeKeySignature *tkaNodeKeySignatureV1
 
 	// TrustedKeys describes the keys currently trusted to make changes
-	// to network-lock.
+	// to tailnet-lock.
 	TrustedKeys []tkaKeyV1
 
 	// VisiblePeers describes peers which are visible in the netmap that

cmd/tailscale/cli/serve_legacy.go

@@ -848,10 +848,10 @@ func (e *serveEnv) enableFeatureInteractive(ctx context.Context, feature string,
 			e.lc.IncrementCounter(ctx, fmt.Sprintf("%s_enablement_lost_connection", feature), 1)
 			return err
 		}
-		if nm := n.NetMap; nm != nil && nm.SelfNode.Valid() {
+		if self := n.SelfChange; self != nil {
 			gotAll := true
 			for _, c := range caps {
-				if !nm.SelfNode.HasCap(c) {
+				if _, has := self.CapMap[c]; !has {
 					// The feature is not yet enabled.
 					// Continue blocking until it is.
 					gotAll = false

cmd/tailscale/cli/systray.go

@@ -7,6 +7,7 @@ package cli
 
 import (
 	"context"
+	"flag"
 
 	"github.com/peterbourgon/ff/v3/ffcli"
 	"tailscale.com/client/systray"
@@ -17,10 +18,20 @@ var systrayCmd = &ffcli.Command{
 	ShortUsage: "tailscale systray",
 	ShortHelp:  "Run a systray application to manage Tailscale",
 	LongHelp:   "Run a systray application to manage Tailscale.",
+	FlagSet: (func() *flag.FlagSet {
+		fs := newFlagSet("systray")
+		fs.StringVar(&systrayArgs.theme, "theme", "dark", "color theme for Tailscale icon: dark, dark:nobg, light, light:nobg")
+		return fs
+	})(),
 	Exec: runSystray,
 }
 
+var systrayArgs struct {
+	theme string
+}
+
 func runSystray(ctx context.Context, _ []string) error {
+	systray.SetTheme(systrayArgs.theme)
 	new(systray.Menu).Run(&localClient)
 	return nil
 }

cmd/tailscale/cli/up.go

@@ -113,12 +113,12 @@ func newUpFlagSet(goos string, upArgs *upArgsT, cmd string) *flag.FlagSet {
 	upf.BoolVar(&upArgs.exitNodeAllowLANAccess, "exit-node-allow-lan-access", false, "Allow direct access to the local network when routing traffic via an exit node")
 	upf.BoolVar(&upArgs.shieldsUp, "shields-up", false, "don't allow incoming connections")
 	upf.BoolVar(&upArgs.runSSH, "ssh", false, "run an SSH server, permitting access per tailnet admin's declared policy")
-	upf.StringVar(&upArgs.advertiseTags, "advertise-tags", "", "comma-separated ACL tags to request; each must start with \"tag:\" (e.g. \"tag:eng,tag:montreal,tag:ssh\")")
+	upf.StringVar(&upArgs.advertiseTags, "advertise-tags", "", "comma-separated ACL tags to request (e.g. \"tag:eng,tag:montreal,tag:ssh\"); the \"tag:\" prefix is optional and added automatically when omitted (e.g. \"eng,montreal,ssh\")")
 	upf.StringVar(&upArgs.hostname, "hostname", "", "hostname to use instead of the one provided by the OS")
 	upf.StringVar(&upArgs.advertiseRoutes, "advertise-routes", "", "routes to advertise to other nodes (comma-separated, e.g. \"10.0.0.0/8,192.168.0.0/24\") or empty string to not advertise routes")
 	upf.BoolVar(&upArgs.advertiseConnector, "advertise-connector", false, "advertise this node as an app connector")
 	upf.BoolVar(&upArgs.advertiseDefaultRoute, "advertise-exit-node", false, "offer to be an exit node for internet traffic for the tailnet")
-	upf.BoolVar(&upArgs.postureChecking, "report-posture", false, hidden+"allow management plane to gather device posture information")
+	upf.BoolVar(&upArgs.postureChecking, "report-posture", false, "allow management plane to gather device posture information")
 
 	if safesocket.GOOSUsesPeerCreds(goos) {
 		upf.StringVar(&upArgs.opUser, "operator", "", "Unix username to allow to operate on tailscaled without sudo")
@@ -309,9 +309,15 @@ func prefsFromUpArgs(upArgs upArgsT, warnf logger.Logf, st *ipnstate.Status, goo
 	var tags []string
 	if upArgs.advertiseTags != "" {
 		tags = strings.Split(upArgs.advertiseTags, ",")
-		for _, tag := range tags {
+		for i, tag := range tags {
-			err := tailcfg.CheckTag(tag)
+			// Allow users to omit the "tag:" prefix; if the tag has no
-			if err != nil {
+			// colon at all, add it for them. Tags with a colon must be
+			// fully qualified ("tag:foo") and are validated as-is.
+			if !strings.Contains(tag, ":") {
+				tag = "tag:" + tag
+				tags[i] = tag
+			}
+			if err := tailcfg.CheckTag(tag); err != nil {
 				return nil, fmt.Errorf("tag: %q: %s", tag, err)
 			}
 		}
@@ -726,7 +732,7 @@ func runUp(ctx context.Context, cmd string, args []string, upArgs upArgsT) (retE
 			if s := n.State; s != nil {
 				ipnIsRunning = *s == ipn.Running
 			}
-			if n.NetMap != nil && n.NetMap.NodeKey != origNodeKey {
+			if n.SelfChange != nil && n.SelfChange.Key != origNodeKey {
 				waitingForKeyChange = false
 			}
 			if ipnIsRunning && !waitingForKeyChange {

cmd/tailscale/depaware.txt

@@ -239,7 +239,7 @@ tailscale.com/cmd/tailscale dependencies: (generated by github.com/tailscale/dep
         tailscale.com/tstime                                         from tailscale.com/control/controlhttp+
         tailscale.com/tstime/mono                                    from tailscale.com/tstime/rate
         tailscale.com/tstime/rate                                    from tailscale.com/cmd/tailscale/cli
-        tailscale.com/tsweb                                          from tailscale.com/util/eventbus
+        tailscale.com/tsweb                                          from tailscale.com/util/eventbus+
         tailscale.com/tsweb/varz                                     from tailscale.com/util/usermetric+
         tailscale.com/types/appctype                                 from tailscale.com/client/local+
         tailscale.com/types/dnstype                                  from tailscale.com/tailcfg+
@@ -331,7 +331,7 @@ tailscale.com/cmd/tailscale dependencies: (generated by github.com/tailscale/dep
         golang.org/x/net/icmp                                        from tailscale.com/net/ping
         golang.org/x/net/idna                                        from golang.org/x/net/http/httpproxy+
         golang.org/x/net/internal/iana                               from golang.org/x/net/icmp+
-        golang.org/x/net/internal/socket                             from golang.org/x/net/icmp+
+        golang.org/x/net/internal/socket                             from golang.org/x/net/ipv4+
         golang.org/x/net/internal/socks                              from golang.org/x/net/proxy
         golang.org/x/net/ipv4                                        from golang.org/x/net/icmp+
         golang.org/x/net/ipv6                                        from golang.org/x/net/icmp+