This commit modifies the kubernetes operator to use the `tailscale-client-go-v2`
package instead of the internal tailscale client it was previously using. This
now gives us the ability to expand out custom resources and features as they
become available via the API module.
The tailnet reconciler has also been modified to manage clients as tailnets
are created and removed, providing each subsequent reconciler with a single
`ClientProvider` that obtains a tailscale client for the respective tailnet
by name, or the operator's default when presented with a blank string.
Fixes: https://github.com/tailscale/corp/issues/38418
Signed-off-by: David Bond <davidsbond93@gmail.com>
* cmd/k8s-operator: use correct tailnet client for L7 & L3 ingresses
This commit fixes a bug when using multi-tailnet within the operator
to spin up L7 & L3 ingresses where the client used to create the
tailscale services was not switching depending on the tailnet used
by the proxygroup backing the service/ingress.
Updates: https://github.com/tailscale/corp/issues/34561
Signed-off-by: David Bond <davidsbond93@gmail.com>
* cmd/k8s-operator: adding server url to proxygroups when a custom tailnet has been specified
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
(cherry picked from commit 3b21ac5504e713e32dfcd43d9ee21e7e712ac200)
---------
Signed-off-by: David Bond <davidsbond93@gmail.com>
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Co-authored-by: chaosinthecrd <tom@tmlabs.co.uk>
This commit implements a reconciler for the new `ProxyGroupPolicy`
custom resource. When created, all `ProxyGroupPolicy` resources
within the same namespace are merged into two `ValidatingAdmissionPolicy`
resources, one for egress and one for ingress.
These policies use CEL expressions to limit the usage of the
"tailscale.com/proxy-group" annotation on `Service` and `Ingress`
resources on create & update.
Included here is also a new e2e test that ensures that resources that
violate the policy return an error on creation, and that once the
policy is changed to allow them they can be created.
Closes: https://github.com/tailscale/corp/issues/36830
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit adds a new custom resource definition to the kubernetes
operator named `ProxyGroupPolicy`. This resource is namespace scoped
and is used as an allow list for which `ProxyGroup` resources can be
used within its namespace.
The `spec` contains two fields, `ingress` and `egress`. These should
contain the names of `ProxyGroup` resources to denote which can be
used as values in the `tailscale.com/proxy-group` annotation within
`Service` and `Ingress` resources.
The intention is for these policies to be merged within a namespace and
produce a `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding`
for both ingress and egress that prevents users from using names of
`ProxyGroup` resources in those annotations.
Closes: https://github.com/tailscale/corp/issues/36829
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit fixes an issue within containerboot that arose from the
kubernetes operator. When users enable metrics on custom resources that
are running on dual stack or ipv6 only clusters, they end up with an error
as we pass the hostport combintation using $(POD_IP):PORT.
In go, `netip.ParseAddrPort` expects square brackets `[]` to wrap the host
portion of an ipv6 address and would naturally, crash.
When loading the containerboot configuration from the environment we now
check if the `TS_LOCAL_ADDR_PORT` value contains the pod's v6 ip address.
If it does & does not already contain brackets, we add the brackets in.
Closes: #15762Closes: #15467
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit contains the implementation of multi-tailnet support within the Kubernetes Operator
Each of our custom resources now expose the `spec.tailnet` field. This field is a string that must match the name of an existing `Tailnet` resource. A `Tailnet` resource looks like this:
```yaml
apiVersion: tailscale.com/v1alpha1
kind: Tailnet
metadata:
name: example # This is the name that must be referenced by other resources
spec:
credentials:
secretName: example-oauth
```
Each `Tailnet` references a `Secret` resource that contains a set of oauth credentials. This secret must be created in the same namespace as the operator:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: example-oauth # This is the name that's referenced by the Tailnet resource.
namespace: tailscale
stringData:
client_id: "client-id"
client_secret: "client-secret"
```
When created, the operator performs a basic check that the oauth client has access to all required scopes. This is done using read actions on devices, keys & services. While this doesn't capture a missing "write" permission, it catches completely missing permissions. Once this check passes, the `Tailnet` moves into a ready state and can be referenced. Attempting to use a `Tailnet` in a non-ready state will stall the deployment of `Connector`s, `ProxyGroup`s and `Recorder`s until the `Tailnet` becomes ready.
The `spec.tailnet` field informs the operator that a `Connector`, `ProxyGroup`, or `Recorder` must be given an auth key generated using the specified oauth client. For backwards compatibility, the set of credentials the operator is configured with are considered the default. That is, where `spec.tailnet` is not set, the resource will be deployed in the same tailnet as the operator.
Updates https://github.com/tailscale/corp/issues/34561
This commit fixes a bug in our HA ingress reconciler where ingress resources would
be stuck in a deleting state should their associated VIP service be deleted within
control.
The reconciliation loop would check for the existence of the VIP service and if not
found perform no additional cleanup steps. The code has been modified to continue
onwards even if the VIP service is not found.
Fixes: https://github.com/tailscale/tailscale/issues/18049
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit adds the `spec.replicas` field to the `Recorder` custom
resource that allows for a highly available deployment of `tsrecorder`
within a kubernetes cluster.
Many changes were required here as the code hard-coded the assumption
of a single replica. This has required a few loops, similar to what we
do for the `Connector` resource to create auth and state secrets. It
was also required to add a check to remove dangling state and auth
secrets should the recorder be scaled down.
Updates: https://github.com/tailscale/tailscale/issues/17965
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the helm/static manifest configuration for the
k8s-operator to prefer the stable image tag. This avoids making those
using static manifests seeing unstable behaviour by default if they
do not manually make the change.
This is managed for us when using helm but not when generating the
static manifests.
Updates https://github.com/tailscale/tailscale/issues/10655
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the k8s-operator's api proxy implementation to only
enable forwarding of api requests to tsrecorder when an environment
variable is set.
This new environment variable is named `TS_EXPERIMENTAL_KUBE_API_EVENTS`.
Updates https://github.com/tailscale/corp/issues/32448
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the `DNSConfig` custom resource to allow specifying
a replica count when deploying a nameserver. This allows deploying
nameservers in a HA configuration.
Updates https://github.com/tailscale/corp/issues/32589
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the k8s operator to wrap its logger using the logtail
logger provided via the tsnet server. This causes any logs written by
the operator to make their way to Tailscale in the same fashion as
wireguard logs to be used by support.
This functionality can also be opted-out of entirely using the
"TS_NO_LOGS_NO_SUPPORT" environment variable.
Updates https://github.com/tailscale/corp/issues/32037
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit adds a new method to the tsnet.Server type named `Logger`
that returns the underlying logtail instance's Logf method.
This is intended to be used within the Kubernetes operator to wrap its
existing logger in a way such that operator specific logs can also be
sent to control for support & debugging purposes.
Updates https://github.com/tailscale/corp/issues/32037
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies containerboot's state reset process to handle the
state secret not existing. During other parts of the boot process we
gracefully handle the state secret not being created yet, but missed
that check within `resetContainerbootState`
Fixes https://github.com/tailscale/tailscale/issues/16804
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit adds a `replicas` field to the `Connector` custom resource that
allows users to specify the number of desired replicas deployed for their
connectors.
This allows users to deploy exit nodes, subnet routers and app connectors
in a highly available fashion.
Fixes#14020
Signed-off-by: David Bond <davidsbond93@gmail.com>
* Modifies the k8s-proxy to expose health check and metrics
endpoints on the Pod's IP.
* Moves cmd/containerboot/healthz.go and cmd/containerboot/metrics.go to
/kube to be shared with /k8s-proxy.
Updates #13358
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the kubernetes operator's `DNSConfig` resource
with the addition of a new field at `nameserver.service.clusterIP`.
This field allows users to specify a static in-cluster IP address of
the nameserver when deployed.
Fixes#14305
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the k8s-operator and k8s-proxy to support passing down
the accept-routes configuration from the proxy class as a configuration value
read and used by the k8s-proxy when ran as a distinct container managed by
the operator.
Updates #13358
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the k8s proxy application configuration to include a
new field named `ServerURL` which, when set, modifies
the tailscale coordination server used by the proxy. This works in the same
way as the operator and the proxies it deploys.
If unset, the default coordination server is used.
Updates https://github.com/tailscale/tailscale/issues/13358
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the operator to detect the usage of k8s-apiserver
type proxy groups that wish to use the letsencrypt staging directory and
apply the appropriate environment variable to the statefulset it
produces.
Updates #13358
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the k8s operator to allow for customisation of the ingress class name
via a new `OPERATOR_INGRESS_CLASS_NAME` environment variable. For backwards compatibility,
this defaults to `tailscale`.
When using helm, a new `ingress.name` value is provided that will set this environment variable
and modify the name of the deployed `IngressClass` resource.
Fixes https://github.com/tailscale/tailscale/issues/16248
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the operator helm chart values to bring the newly
added `loginServer` field to the top level. We felt as though it was a bit
confusing to be at the `operatorConfig` level as this value modifies the
behaviour or the operator, api server & all resources that the operator
manages.
Updates https://github.com/tailscale/corp/issues/29847
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit modifies the kubernetes operator to allow for customisation of the tailscale
login url. This provides some data locality for people that want to configure it.
This value is set in the `loginServer` helm value and is propagated down to all resources
managed by the operator. The only exception to this is recorder nodes, where additional
changes are required to support modifying the url.
Updates https://github.com/tailscale/corp/issues/29847
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit adds a NOTES.txt to the operator helm chart that will be written to the
terminal upon successful installation of the operator.
It includes a small list of knowledgebase articles with possible next steps for
the actor that installed the operator to the cluster. It also provides possible
commands to use for explaining the custom resources.
Fixes#13427
Signed-off-by: David Bond <davidsbond93@gmail.com>
David BondandGitHub
