cmd/k8s-operator: add multi replica support for recorders (#17864)

This commit adds the `spec.replicas` field to the `Recorder` custom
resource that allows for a highly available deployment of `tsrecorder`
within a kubernetes cluster.

Many changes were required here as the code hard-coded the assumption
of a single replica. This has required a few loops, similar to what we
do for the `Connector` resource to create auth and state secrets. It
was also required to add a check to remove dangling state and auth
secrets should the recorder be scaled down.

Updates: https://github.com/tailscale/tailscale/issues/17965

Signed-off-by: David Bond <davidsbond93@gmail.com>

cmd/k8s-operator/deploy/crds/tailscale.com_recorders.yaml

@@ -68,6 +68,11 @@ spec:
                     Corresponds to --ui tsrecorder flag https://tailscale.com/kb/1246/tailscale-ssh-session-recording#deploy-a-recorder-node.
                     Required if S3 storage is not set up, to ensure that recordings are accessible.
                   type: boolean
+                replicas:
+                  description: Replicas specifies how many instances of tsrecorder to run. Defaults to 1.
+                  type: integer
+                  format: int32
+                  minimum: 0
                 statefulSet:
                   description: |-
                     Configuration parameters for the Recorder's StatefulSet. The operator
@@ -1683,6 +1688,9 @@ spec:
                   items:
                     type: string
                     pattern: ^tag:[a-zA-Z][a-zA-Z0-9-]*$
+              x-kubernetes-validations:
+                - rule: '!(self.replicas > 1 && (!has(self.storage) || !has(self.storage.s3)))'
+                  message: S3 storage must be used when deploying multiple Recorder replicas
             status:
               description: |-
                 RecorderStatus describes the status of the recorder. This is set

cmd/k8s-operator/deploy/manifests/operator.yaml

@@ -3348,6 +3348,11 @@ spec:
                                     Corresponds to --ui tsrecorder flag https://tailscale.com/kb/1246/tailscale-ssh-session-recording#deploy-a-recorder-node.
                                     Required if S3 storage is not set up, to ensure that recordings are accessible.
                                 type: boolean
+                            replicas:
+                                description: Replicas specifies how many instances of tsrecorder to run. Defaults to 1.
+                                format: int32
+                                minimum: 0
+                                type: integer
                             statefulSet:
                                 description: |-
                                     Configuration parameters for the Recorder's StatefulSet. The operator
@@ -4964,6 +4969,9 @@ spec:
                                     type: string
                                 type: array
                         type: object
+                        x-kubernetes-validations:
+                            - message: S3 storage must be used when deploying multiple Recorder replicas
+                              rule: '!(self.replicas > 1 && (!has(self.storage) || !has(self.storage.s3)))'
                     status:
                         description: |-
                             RecorderStatus describes the status of the recorder. This is set