cmd/k8s-operator,k8s-operator: Kubernetes Peer Relays (#20495)

This commit contains the Kubernetes implementation of peer relays via the new `PeerRelay` CRD. It's a mega branch consisting of the commits of other PRs gone into this work: 1. https://github.com/tailscale/tailscale/pull/20211 2. https://github.com/tailscale/tailscale/pull/20329 3. https://github.com/tailscale/tailscale/pull/20423 4. https://github.com/tailscale/tailscale/pull/20503 An instance of the `PeerRelay` CRD deploys a `StatefulSet` of containerboot instances configured to advertise themselves as peer relays using the IP addresses configured via `LoadBalancer` services on each cloud provider (with some AWS specifics as it's less automatic than its competing cloud providers). Per replica, a `LoadBalancer` type `Service` resource is provisioned and its IP address is used to configure the respective relay. This has been tested with success in AWS, GCP & Azure and provides additional modification to `Service` resources via the CRD for any other kinds of deployment environments. It also contains some work that may appear to be duplication of what already exists within `cmd/k8s-operator` so we can start building an appropriate migration path for `Connector`, `ProxyGroup` etc into respective `k8s-operator/reconciler/*` packages. Closes https://github.com/tailscale/corp/issues/34524
2026-07-20 16:37:15 +01:00
parent 2dd5d82f56
commit be0e460a20
22 changed files with 3823 additions and 1 deletions
@@ -12,6 +12,8 @@
 - [ConnectorList](#connectorlist)
 - [DNSConfig](#dnsconfig)
 - [DNSConfigList](#dnsconfiglist)
+- [PeerRelay](#peerrelay)
+- [PeerRelayList](#peerrelaylist)
 - [ProxyClass](#proxyclass)
 - [ProxyClassList](#proxyclasslist)
 - [ProxyGroup](#proxygroup)
@@ -349,6 +351,7 @@ _Validation:_

 _Appears in:_
 - [ConnectorSpec](#connectorspec)
+- [PeerRelaySpec](#peerrelayspec)
 - [ProxyGroupSpec](#proxygroupspec)


@@ -536,6 +539,154 @@ _Appears in:_
 | `selector` _object (keys:string, values:string)_ | A selector which will be used to select the node's that will have their `ExternalIP`'s advertised<br />by the ProxyGroup as Static Endpoints. |  |  |


+#### PeerRelay
+
+
+
+
+
+
+
+_Appears in:_
+- [PeerRelayList](#peerrelaylist)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `apiVersion` _string_ | `tailscale.com/v1alpha1` | | |
+| `kind` _string_ | `PeerRelay` | | |
+| `kind` _string_ | Kind is a string value representing the REST resource this object represents.<br />Servers may infer this from the endpoint the client submits requests to.<br />Cannot be updated.<br />In CamelCase.<br />More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds |  |  |
+| `apiVersion` _string_ | APIVersion defines the versioned schema of this representation of an object.<br />Servers should convert recognized schemas to the latest internal value, and<br />may reject unrecognized values.<br />More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources |  |  |
+| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.3/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. |  |  |
+| `spec` _[PeerRelaySpec](#peerrelayspec)_ | Spec describes the desired state of the PeerRelay.<br />More info:<br />https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status |  |  |
+| `status` _[PeerRelayStatus](#peerrelaystatus)_ | Status describes the status of the PeerRelay. This is set<br />and managed by the Tailscale operator. |  |  |
+
+
+#### PeerRelayAWS
+
+
+
+PeerRelayAWS contains AWS-specific configuration for a PeerRelay.
+
+
+
+_Appears in:_
+- [PeerRelaySpec](#peerrelayspec)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `elasticIPs` _[PeerRelayAWSElasticIP](#peerrelayawselasticip) array_ | ElasticIPs pins each replica to a specific AWS EIP allocation and subnet. Only meaningful when Network Load<br />Balancers are provisioned by the AWS Load Balancer Controller. ElasticIPs supplies one allocation-subnet pair<br />per replica: replica N uses ElasticIPs[N]. The list must be at least as long as spec.replicas so every replica<br />has a distinct EIP; extra entries are permitted so that scale-up doesn't immediately trip validation.<br />When set, the reconciler stamps<br />service.beta.kubernetes.io/aws-load-balancer-eip-allocations and<br />service.beta.kubernetes.io/aws-load-balancer-subnets on each per-replica Service, overriding any values in<br />spec.service.annotations. |  | MinItems: 1 <br /> |
+
+
+#### PeerRelayAWSElasticIP
+
+
+
+PeerRelayAWSElasticIP pairs an EIP allocation with the subnet in the same AZ.
+
+
+
+_Appears in:_
+- [PeerRelayAWS](#peerrelayaws)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `allocationID` _string_ | AllocationID is the AWS EIP allocation ID (e.g. eipalloc-0123abcd) whose public IP this replica is reachable<br />on. Stamped as service.beta.kubernetes.io/aws-load-balancer-eip-allocations on the replica's Service. |  | Pattern: `^eipalloc-[0-9a-f]+$` <br /> |
+| `subnetID` _string_ | SubnetID is the AWS subnet in the same availability zone as AllocationID (e.g. subnet-0123abcd). Stamped as<br />service.beta.kubernetes.io/aws-load-balancer-subnets on the replica's Service so the NLB is provisioned in<br />the same AZ as the EIP. |  | Pattern: `^subnet-[0-9a-f]+$` <br /> |
+
+
+#### PeerRelayEndpoint
+
+
+
+
+
+
+
+_Appears in:_
+- [PeerRelayStatus](#peerrelaystatus)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `replica` _integer_ | Replica is the zero-based index of the peer relay replica this endpoint targets. |  |  |
+| `address` _string_ | Address is the public IP or hostname the cloud has allocated for this replica's LoadBalancer Service.<br />Peers reach this relay by connecting to Address:Port over UDP. |  |  |
+| `port` _integer_ | Port is the UDP port the peer relay listens on. |  |  |
+
+
+#### PeerRelayList
+
+
+
+
+
+
+
+
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `apiVersion` _string_ | `tailscale.com/v1alpha1` | | |
+| `kind` _string_ | `PeerRelayList` | | |
+| `kind` _string_ | Kind is a string value representing the REST resource this object represents.<br />Servers may infer this from the endpoint the client submits requests to.<br />Cannot be updated.<br />In CamelCase.<br />More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds |  |  |
+| `apiVersion` _string_ | APIVersion defines the versioned schema of this representation of an object.<br />Servers should convert recognized schemas to the latest internal value, and<br />may reject unrecognized values.<br />More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources |  |  |
+| `metadata` _[ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.3/#listmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. |  |  |
+| `items` _[PeerRelay](#peerrelay) array_ |  |  |  |
+
+
+#### PeerRelayService
+
+
+
+
+
+
+
+_Appears in:_
+- [PeerRelaySpec](#peerrelayspec)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `annotations` _object (keys:string, values:string)_ | Annotations to apply to the LoadBalancer service. Any annotations that conflict with those used by known<br />cloud providers to ensure IP addresses rather than DNS names are ignored. |  |  |
+
+
+#### PeerRelaySpec
+
+
+
+
+
+
+
+_Appears in:_
+- [PeerRelay](#peerrelay)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `tags` _[Tags](#tags)_ | Tags that the Tailscale node will be tagged with.<br />Defaults to [tag:k8s].<br />To autoapprove the device defined by a PeerRelay,<br />you can configure Tailscale ACLs to give these tags the necessary<br />permissions.<br />See https://tailscale.com/kb/1337/acl-syntax#autoapprovers.<br />If you specify custom tags here, you must also make the operator an owner of these tags.<br />See  https://tailscale.com/kb/1236/kubernetes-operator/#setting-up-the-kubernetes-operator.<br />Tags cannot be changed once a PeerRelay node has been created.<br />Tag values must be in form ^tag:[a-zA-Z][a-zA-Z0-9-]*$. |  | Pattern: `^tag:[a-zA-Z][a-zA-Z0-9-]*$` <br />Type: string <br /> |
+| `hostnamePrefix` _[HostnamePrefix](#hostnameprefix)_ | HostnamePrefix specifies the hostname prefix for each<br />replica. Each device will have the integer number<br />from its StatefulSet pod appended to this prefix to form the full hostname.<br />HostnamePrefix can contain lower case letters, numbers and dashes, it<br />must not start with a dash and must be between 1 and 62 characters long. |  | Pattern: `^[a-z0-9][a-z0-9-]{0,61}$` <br />Type: string <br /> |
+| `proxyClass` _string_ | ProxyClass is the name of the ProxyClass custom resource that<br />contains configuration options that should be applied to the<br />resources created for this PeerRelay. If unset, the operator will<br />create resources with the default configuration. |  |  |
+| `replicas` _integer_ | Replicas specifies how many devices to create. Set this to enable<br />high availability for peer relays.<br />https://tailscale.com/kb/1115/high-availability. Defaults to 1. | 1 | Minimum: 0 <br /> |
+| `tailnet` _string_ | Tailnet specifies the tailnet this PeerRelay should join. If blank, the default tailnet is used. When set, this<br />name must match that of a valid Tailnet resource. This field is immutable and cannot be changed once set. |  |  |
+| `service` _[PeerRelayService](#peerrelayservice)_ | Service contains configuration values to modify the LoadBalancer service used to expose the peer relay. |  |  |
+| `aws` _[PeerRelayAWS](#peerrelayaws)_ | AWS contains configuration for pinning each replica to a specific AWS Elastic IP and subnet. Only meaningful<br />when running on EKS with the AWS Load Balancer Controller. When set, the per-replica values override any<br />aws-load-balancer-eip-allocations or aws-load-balancer-subnets values supplied via spec.service.annotations. |  |  |
+
+
+#### PeerRelayStatus
+
+
+
+
+
+
+
+_Appears in:_
+- [PeerRelay](#peerrelay)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.3/#condition-v1-meta) array_ |  |  |  |
+| `endpoints` _[PeerRelayEndpoint](#peerrelayendpoint) array_ | Endpoints lists the public address:port pairs each peer relay replica is reachable on. There is one entry<br />per replica whose LoadBalancer Service has been assigned a public address; entries appear as the underlying<br />cloud provisions each Service. |  |  |
+
+
 #### Pod


@@ -1233,6 +1384,7 @@ _Validation:_

 _Appears in:_
 - [ConnectorSpec](#connectorspec)
+- [PeerRelaySpec](#peerrelayspec)
 - [ProxyGroupSpec](#proxygroupspec)
 - [RecorderSpec](#recorderspec)