The kube-apiserver proxy in the operator would only run in
auth proxy mode but thats not always desirable. There are
situations where the proxy should just be a transparent
proxy and not inject auth headers, so do that using a new
env var APISERVER_PROXY and deprecate the AUTH_PROXY env.
THe new env var has three options `false`, `true` and `noauth`.
Updates #8317
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It was only waiting for 0.5s (5ms * 100), but our CI
is too slow so make it wait up to 3s (10ms * 300).
Updates tailscale/corp#14515
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We would only check if the client was paused, but not
if the client was closed. This meant that a call to
Shutdown may block forever/leak goroutines
Updates #cleanup
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It would acquire the lock, calculate `nextState`, relase
the lock, then call `enterState` which would acquire the lock
again. There were obvious races there which could lead to
nil panics as seen in a test in a different repo.
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x70 pc=0x1050f2c7c]
goroutine 42240 [running]:
tailscale.com/ipn/ipnlocal.(*LocalBackend).enterStateLockedOnEntry(0x14002154e00, 0x6)
tailscale.com/ipn/ipnlocal/local.go:3715 +0x30c
tailscale.com/ipn/ipnlocal.(*LocalBackend).enterState(0x14002154e00?, 0x14002e3a140?)
tailscale.com/ipn/ipnlocal/local.go:3663 +0x8c
tailscale.com/ipn/ipnlocal.(*LocalBackend).stateMachine(0x14001f5e280?)
tailscale.com/ipn/ipnlocal/local.go:3836 +0x2c
tailscale.com/ipn/ipnlocal.(*LocalBackend).setWgengineStatus(0x14002154e00, 0x14002e3a190, {0x0?, 0x0?})
tailscale.com/ipn/ipnlocal/local.go:1193 +0x4d0
tailscale.com/wgengine.(*userspaceEngine).RequestStatus(0x14005d90300)
tailscale.com/wgengine/userspace.go:1051 +0x80
tailscale.com/wgengine.NewUserspaceEngine.func2({0x14002e3a0a0, 0x2, 0x140025cce40?})
tailscale.com/wgengine/userspace.go:318 +0x1a0
tailscale.com/wgengine/magicsock.(*Conn).updateEndpoints(0x14002154700, {0x105c13eaf, 0xf})
tailscale.com/wgengine/magicsock/magicsock.go:531 +0x424
created by tailscale.com/wgengine/magicsock.(*Conn).ReSTUN in goroutine 42077
tailscale.com/wgengine/magicsock/magicsock.go:2142 +0x3a4
```
Updates tailscale/corp#14480
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It would just fail the entire pkg, but would not print any
logs. It was already tracking all the logs, so have it emit
them when the pkg fails/times out.
Updates #9231
Signed-off-by: Maisem Ali <maisem@tailscale.com>
If Start was called multiple times concurrently, it would
create a new client and shutdown the previous one. However
there was a race possible between shutting down the old one
and assigning a new one where the concurent goroutine may
have assigned another one already and it would leak.
Updates tailscale/corp#14471
Signed-off-by: Maisem Ali <maisem@tailscale.com>
resetControlClientLocked is called while b.mu was held and
would call cc.Shutdown which would wait for the observer queue
to drain.
However, there may be active callbacks from cc already waiting for
b.mu resulting in a deadlock.
This makes it so that resetControlClientLocked does not call
Shutdown, and instead just returns the value.
It also makes it so that any status received from previous cc
are ignored.
Updates tailscale/corp#12827
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We already removed the async API, make it more sync and remove
the FinishLogout state too.
This also makes the callback be synchronous again as the previous
attempt was trying to work around the logout callback resulting
in a client shutdown getting blocked forever.
Updates #3833
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We have cases where the SetControlClientStatus would result in
a Shutdown call back into the auto client that would block
forever. The right thing to do here is to fix the LocalBackend
state machine but thats a different dumpster fire that we
are slowly making progress towards.
This makes it so that the SetControlClientStatus happens in a
different goroutine so that calls back into the auto client
do not block.
Also add a few missing mu.Unlocks in LocalBackend.Start.
Updates #9181
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Instead of confusing users, emit an event that explicitly tells the
user that HTTPS is disabled on the tailnet and that ingress may not
work until they enable it.
Updates #9141
Signed-off-by: Maisem Ali <maisem@tailscale.com>
While investigating the fix in 7538f38671,
I was curious why the testwrapper didn't fail. Turns out if the test
times out and there was no explicit failure, the only message we get
is that the overall pkg failed and no failure information about the
individual test. This resulted in a 0 exit code.
This fixes that by failing the explicit case of the pkg failing when
there is nothing to retry for that pkg.
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We would call Update on the secret, but that was racey and would occasionaly
fail. Instead use patch whenever we can.
Fixes errors like
```
boot: 2023/08/29 01:03:53 failed to set serve config: sending serve config: updating config: writing ServeConfig to StateStore: Operation cannot be fulfilled on secrets "ts-webdav-kfrzv-0": the object has been modified; please apply your changes to the latest version and try again
{"level":"error","ts":"2023-08-29T01:03:48Z","msg":"Reconciler error","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","Ingress":{"name":"webdav","namespace":"default"},"namespace":"default","name":"webdav","reconcileID":"96f5cfed-7782-4834-9b75-b0950fd563ed","error":"failed to provision: failed to create or get API key secret: Operation cannot be fulfilled on secrets \"ts-webdav-kfrzv-0\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
```
Updates #502
Updates #7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This was added in 3451b89e5f, but
resulted in the v6 Tailscale address being added to status when
when the forwarding only happened on the v4 address.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The new ingress reconcile raises events on failure, but I forgot to
add the updated permission.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
On k8s the serve-config secret mount is symlinked so checking against
the Name makes us miss the events.
Updates #7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously, the operator would only monitor Services and create
a Tailscale StatefulSet which acted as a L3 proxy which proxied
traffic inbound to the Tailscale IP onto the services ClusterIP.
This extends that functionality to also monitor Ingress resources
where the `ingressClassName=tailscale` and similarly creates a
Tailscale StatefulSet, acting as a L7 proxy instead.
Users can override the desired hostname by setting:
```
- tls
hosts:
- "foo"
```
Hostnames specified under `rules` are ignored as we only create a single
host. This is emitted as an event for users to see.
Fixes#7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This watches the provided path for a JSON encoded ipn.ServeConfig.
Everytime the file changes, or the nodes FQDN changes it reapplies
the ServeConfig.
At boot time, it nils out any previous ServeConfig just like tsnet does.
As the ServeConfig requires pre-existing knowledge of the nodes FQDN to do
SNI matching, it introduces a special `${TS_CERT_DOMAIN}` value in the JSON
file which is replaced with the known CertDomain before it is applied.
Updates #502
Updates #7895
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously users would have to unexpose/expose the service in order to
change Hostname/TargetIP. This now applies those changes by causing a
StatefulSet rollout now that a61a9ab087 is in.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously we would not reapply changes to TS_HOSTNAME etc when
then the container restarted and TS_AUTH_ONCE was enabled.
This splits those into two steps login and set, allowing us to
only rerun the set step on restarts.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
I forgot to move the defer out of the func, so the tsnet.Server
immediately closed after starting.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It was jumbled doing a lot of things, this breaks it up into
the svc reconciliation and the tailscale sts reconciliation.
Prep for future commit.
Updates #502
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Theory is that our long lived http2 connection to control would
get tainted by _something_ (unclear what) and would get closed.
This picks up the fix for golang/go#60818.
Updates tailscale/corp#5761
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It had a custom Clone func with a TODO to replace with cloner, resolve
that todo. Had to pull out the embedded Auth struct into a named struct.
Updates #cleanup
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously we would use the Impersonate-Group header to pass through
tags to the k8s api server. However, we would do nothing for non-tagged
nodes. Now that we have a way to specify these via peerCaps respect those
and send down groups for non-tagged nodes as well.
For tagged nodes, it defaults to sending down the tags as groups to retain
legacy behavior if there are no caps set. Otherwise, the tags are omitted.
Updates #5055
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Instead of having updates replace the map polls, create
a third goroutine which is solely responsible for making
sure that control is aware of the latest client state.
This also makes it so that the streaming map polls are only
broken when there are auth changes, or the client is paused.
Updates tailscale/corp#5761
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We would only look for duplicate profiles when a new login
occurred but when using `--force-reauth` we could switch
users which would end up with duplicate profiles.
Updates #7726
Signed-off-by: Maisem Ali <maisem@tailscale.com>
There are a few situations where we end up with duplicate profiles.
Add tests to identify those situations, fix in followup.
Updates #7726
Signed-off-by: Maisem Ali <maisem@tailscale.com>
It was being modified in two places in Direct for the auth routine
and then in LocalBackend when a new NetMap was received. This was
confusing, so make Direct also own changes to Persist when a new
NetMap is received.
Updates #7726
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Define PeerCapabilty and PeerCapMap as the new way of sending down
inter-peer capability information.
Previously, this was unstructured and you could only send down strings
which got too limiting for certain usecases. Instead add the ability
to send down raw JSON messages that are opaque to Tailscale but provide
the applications to define them however they wish.
Also update accessors to use the new values.
Updates #4217
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We were never resetting the backoff in streaming mapResponses.
The call to `PollNetMap` always returns with an error. Changing that contract
is harder, so manually reset backoff when a netmap is received.
Updates tailscale/corp#12894
Signed-off-by: Maisem Ali <maisem@tailscale.com>
`go test -json` outputs invalid JSON when a build fails.
Handle that case by reseting the json.Decode and continuing to read.
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Without this, the client would just get stuck dialing even if the
context was canceled.
Updates tailscale/corp#12590
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously it would wait for all tests to run before printing anything,
instead stream the results over a channel so that they can be emitted
immediately.
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously it would only print the failures without providing
more information on which package the failures from.
This commit makes it so that it prints out the package information
as well as the attempt numbers.
```
➜ tailscale.com git:(main) ✗ go run ./cmd/testwrapper ./cmd/...
ok tailscale.com/cmd/derper
ok tailscale.com/cmd/k8s-operator
ok tailscale.com/cmd/tailscale/cli
ok tailscale.com/cmd/tailscaled
=== RUN TestFlakeRun
flakytest.go:38: flakytest: issue tracking this flaky test: https://github.com/tailscale/tailscale/issues/0
flakytest_test.go:41: First run in testwrapper, failing so that test is retried. This is expected.
--- FAIL: TestFlakeRun (0.00s)
FAIL tailscale.com/cmd/testwrapper/flakytest
Attempt #2: Retrying flaky tests:
ok tailscale.com/cmd/testwrapper/flakytest
```
Updates #8493
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The panicLogWriter is too strict, and any panics that occur
get wrapped up in quotes. This makes it so that it will allow
panics to continue writing to Stderr without going through
logger.Logf.
Updates #cleanup
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Redo the testwrapper to track and only retry flaky tests instead
of retrying the entire pkg. It also fails early if a non-flaky test fails.
This also makes it so that the go test caches are used.
Fixes#7975
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Trying to SSH when SELinux is enforced results in errors like:
```
➜ ~ ssh ec2-user@<ip>
Last login: Thu Jun 1 22:51:44 from <ip2>
ec2-user: no shell: Permission denied
Connection to <ip> closed.
```
while the `/var/log/audit/audit.log` has
```
type=AVC msg=audit(1685661291.067:465): avc: denied { transition } for pid=5296 comm="login" path="/usr/bin/bash" dev="nvme0n1p1" ino=2564 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0
```
The right fix here would be to somehow install the appropriate context when
tailscale is installed on host, but until we figure out a way to do that
stop using the `login` cmd in these situations.
Updates #4908
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The netstack code had a bunch of logic to figure out if the LocalBackend should handle an
incoming connection and then would call the function directly on LocalBackend. Move that
logic to LocalBackend and refactor the methods to return conn handlers.
Updates #cleanup
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This basically allows running services on the SSH client and reaching
them from the SSH server during the session.
Updates #6575
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We were only closing on side of the pty/tty pair.
Close the other side too.
Thanks to @fritterhoff for reporting and debugging the issue!
Fixes#8119
Signed-off-by: Maisem Ali <maisem@tailscale.com>
getSingleObject can return `nil, nil`, getDeviceInfo was not handling
that case which resulted in panics.
Fixes#7303
Signed-off-by: Maisem Ali <maisem@tailscale.com>
In the case where the exit node requires SNAT, we would SNAT all traffic not just the
traffic meant to go through the exit node. This was a result of the default route being
added to the routing table which would match basically everything.
In this case, we need to account for all peers in the routing table not just the ones
that require NAT.
Fix and add a test.
Updates tailscale/corp#8020
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We need to always specify tags when creating an AuthKey from an OAuth key.
Check for that, and reuse the `--advertise-tags` param.
Updates #7982
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Previously we would error out when the recording server disappeared after the in memory
buffer filled up for the io.Copy. This makes it so that we handle failing open correctly
in that path.
Updates tailscale/corp#9967
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This adds support to try dialing out to multiple recorders each
with a 5s timeout and an overall 30s timeout. It also starts respecting
the actions `OnRecordingFailure` field if set, if it is not set
it fails open.
Updates tailscale/corp#9967
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This allows control to specify how to handle situations where the recorder
isn't available.
Updates tailscale/corp#9967
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Without this, the peer fails to do anything over the PeerAPI if it
has a masquerade address.
```
Apr 19 13:58:15 hydrogen tailscaled[6696]: peerapi: invalid request from <ip>:58334: 100.64.0.1/32 not found in self addresses
```
Updates #8020
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Found this when adding a test that does a ping over PeerAPI.
Our integration tests set up a trafficTrap to ensure that tailscaled
does not call out to the internet, and it does so via a HTTP_PROXY.
When adding a test for pings over PeerAPI, it triggered the trap and investigation
lead to the realization that we were not removing the Proxy when trying to
dial out to the PeerAPI.
Updates tailscale/corp#8020
Signed-off-by: Maisem Ali <maisem@tailscale.com>
For stores like k8s secrets we need to dial out to the k8s API as though Tailscale
wasn't running. The issue currently only manifests when you try to use an exit node
while running inside a k8s cluster and are trying to use Kubernetes secrets as the
backing store.
This doesn't address cmd/containerboot, which I'll do in a follow up.
Updates #7695
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This adds support to make exit nodes and subnet routers work
when in scenarios where NAT is required.
It also updates the NATConfig to be generated from a `wgcfg.Config` as
that handles merging prefs with the netmap, so it has the required information
about whether an exit node is already configured and whether routes are accepted.
Updates tailscale/corp#8020
Signed-off-by: Maisem Ali <maisem@tailscale.com>
There were two code paths that could fail depending on how fast
the recorder responses. This fixes that by returning the correct
error from both paths.
Fixes#7707
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We were checking against the wrong directory, instead if we
have a custom store configured just use that.
Fixes#7588Fixes#7665
Signed-off-by: Maisem Ali <maisem@tailscale.com>
We were not storing the ACME keys in the state store, they would always
be stored on disk.
Updates #7588
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Maisem AliandMaisem Ali
