cmd/{containerboot,k8s-operator},kube/kubetypes: unadvertise ingress services on shutdown (#15451)

Ensure no services are advertised as part of shutting down tailscaled.
Prefs are only edited if services are currently advertised, and they're
edited we wait for control's ~15s (+ buffer) delay to failover.

Note that editing prefs will trigger a synchronous write to the state
Secret, so it may fail to persist state if the ProxyGroup is getting
scaled down and therefore has its RBAC deleted at the same time, but that
failure doesn't stop prefs being updated within the local backend,
doesn't  affect connectivity to control, and the state Secret is
about to get deleted anyway, so the only negative side effect is a harmless
error log during shutdown. Control still learns that the node is no
longer advertising the service and triggers the failover.

Note that the first version of this used a PreStop lifecycle hook, but
that only supports GET methods and we need the shutdown to trigger side
effects (updating prefs) so it didn't seem appropriate to expose that
functionality on a GET endpoint that's accessible on the k8s network.

Updates tailscale/corp#24795

Change-Id: I0a9a4fe7a5395ca76135ceead05cbc3ee32b3d3c
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>

cmd/containerboot/serve.go

@@ -9,6 +9,7 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
+	"fmt"
 	"log"
 	"os"
 	"path/filepath"
@@ -169,3 +170,46 @@ func readServeConfig(path, certDomain string) (*ipn.ServeConfig, error) {
 	}
 	return &sc, nil
 }
+
+func ensureServicesNotAdvertised(ctx context.Context, lc *local.Client) error {
+	prefs, err := lc.GetPrefs(ctx)
+	if err != nil {
+		return fmt.Errorf("error getting prefs: %w", err)
+	}
+	if len(prefs.AdvertiseServices) == 0 {
+		return nil
+	}
+
+	log.Printf("serve proxy: unadvertising services: %v", prefs.AdvertiseServices)
+	if _, err := lc.EditPrefs(ctx, &ipn.MaskedPrefs{
+		AdvertiseServicesSet: true,
+		Prefs: ipn.Prefs{
+			AdvertiseServices: nil,
+		},
+	}); err != nil {
+		// EditPrefs only returns an error if it fails _set_ its local prefs.
+		// If it fails to _persist_ the prefs in state, we don't get an error
+		// and we continue waiting below, as control will failover as usual.
+		return fmt.Errorf("error setting prefs AdvertiseServices: %w", err)
+	}
+
+	// Services use the same (failover XOR regional routing) mechanism that
+	// HA subnet routers use. Unfortunately we don't yet get a reliable signal
+	// from control that it's responded to our unadvertisement, so the best we
+	// can do is wait for 20 seconds, where 15s is the approximate maximum time
+	// it should take for control to choose a new primary, and 5s is for buffer.
+	//
+	// Note: There is no guarantee that clients have been _informed_ of the new
+	// primary no matter how long we wait. We would need a mechanism to await
+	// netmap updates for peers to know for sure.
+	//
+	// See https://tailscale.com/kb/1115/high-availability for more details.
+	// TODO(tomhjp): Wait for a netmap update instead of sleeping when control
+	// supports that.
+	select {
+	case <-ctx.Done():
+		return nil
+	case <-time.After(20 * time.Second):
+		return nil
+	}
+}