We now embed node information into network flow logs.
By default, netlogfmt still prints out using Tailscale IP addresses.
Support a "--resolve-addrs=TYPE" flag that can be used to specify
resolving IP addresses as node IDs, hostnames, users, or tags.
Updates tailscale/corp#33352
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The REST API does not return a node name
with a trailing dot, while the internal node name
reported in the netmap does have one.
In order to be consistent with the API,
strip the dot when recording node information.
Updates tailscale/corp#33352
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This starts running the jsontags vet checker on the module.
All existing findings are adding to an allowlist.
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The cmd/jsontags is non-idiomatic since it is not a main binary.
Move it to a vet directory, which will eventually contain a vettool binary.
Update tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This migrates an internal tool to open source
so that we can run it on the tailscale.com module as well.
This PR does not yet set up a CI to run this analyzer.
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This rewrites the netlog package to support embedding node information in network flow logs.
Some bit of complexity comes in trying to pre-compute the expected size of the log message
after JSON serialization to ensure that we can respect maximum body limits in log uploading.
We also fix a bug in tstun, where we were recording the IP address after SNAT,
which was resulting in non-sensible connection flows being logged.
Updates tailscale/corp#33352
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This migrates an internal tool to open source
so that we can run it on the tailscale.com module as well.
We add the "util/safediff" also as a dependency of the tool.
This PR does not yet set up a CI to run this analyzer.
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Merge the connstats package into the netlog package
and unexport all of its declarations.
Remove the buildfeatures.HasConnStats and use HasNetLog instead.
Updates tailscale/corp#33352
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The connstats package was an unnecessary layer of indirection.
It was seperated out of wgengine/netlog so that net/tstun and
wgengine/magicsock wouldn't need a depenedency on the concrete
implementation of network flow logging.
Instead, we simply register a callback for counting connections.
This PR does the bare minimum work to prepare tstun and magicsock
to only care about that callback.
A future PR will delete connstats and merge it into netlog.
Updates tailscale/corp#33352
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Remove CBOR representation since it was never used.
We should support CBOR in the future, but for remove it
for now so that it is less work to add more fields.
Also, rely on just omitzero for JSON now that it is supported in Go 1.24.
Updates tailscale/corp#33352
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The default representation of time.Duration has different
JSON representation between v1 and v2.
Apply an explicit format flag that uses the v1 representation
so that this behavior does not change if serialized with v2.
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Our own WaitGroup wrapper type was a prototype implementation
for the Go method on the standard sync.WaitGroup type.
Now that there is first-class support for Go,
we should migrate over to using it and delete syncs.WaitGroup.
Updates #cleanup
Updates tailscale/tailscale#16330
Change-Id: Ib52b10f9847341ce29b4ca0da927dc9321691235
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This adds support for having every viewer type implement
jsonv2.MarshalerTo and jsonv2.UnmarshalerFrom.
This provides a significant boost in performance
as the json package no longer needs to validate
the entirety of the JSON value outputted by MarshalJSON,
nor does it need to identify the boundaries of a JSON value
in order to call UnmarshalJSON.
For deeply nested and recursive MarshalJSON or UnmarshalJSON calls,
this can improve runtime from O(N²) to O(N).
This still references "github.com/go-json-experiment/json"
instead of the experimental "encoding/json/v2" package
now available in Go 1.25 under goexperiment.jsonv2
so that code still builds without the experiment tag.
Of note, the "github.com/go-json-experiment/json" package
aliases the standard library under the right build conditions.
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This package promises more performance, but was never used.
The intent of the package is somewhat moot as "encoding/json"
in Go 1.25 (while under GOEXPERIMENT=jsonv2) has been
completely re-implemented using "encoding/json/v2"
such that unmarshal is dramatically faster.
Updates #cleanup
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Fix CompareAndSwap in the edge-case where
the underlying sync.AtomicValue is uninitialized
(i.e., Store was never called) and
the oldV is the zero value,
then perform CompareAndSwap with any(nil).
Also, document that T must be comparable.
This is a pre-existing restriction.
Fixes#16135
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The typical way to implement union types in Go
is to use an interface where the set of types is limited.
However, there historically has been poor support
in v1 "encoding/json" with interface types where
you can marshal such values, but fail to unmarshal them
since type information about the concrete type is lost.
The MakeInterfaceCoders function constructs custom
marshal/unmarshal functions such that the type name
is encoded in the JSON representation.
The set of valid concrete types for an interface
must be statically specified for this to function.
Updates tailscale/corp#22024
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The encoding/json/v2 effort may end up changing
the default represention of time.Duration in JSON.
See https://go.dev/issue/71631
The GoDuration type allows us to explicitly use
the time.Duration.String representation regardless of
whether we serialize with v1 or v2 of encoding/json.
Updates tailscale/corp#27502
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The json/v2 prototype is still in flux and the API can/will change.
Statically enforce that types implementing the v2 methods
satisfy the correct interface so that changes to the signature
can be statically detected by the compiler.
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The upstream module has seen significant work making
the v1 emulation layer a high fidelity re-implementation
of v1 "encoding/json".
This addresses several upstream breaking changes:
* MarshalJSONV2 renamed as MarshalJSONTo
* UnmarshalJSONV2 renamed as UnmarshalJSONFrom
* Options argument removed from MarshalJSONV2
* Options argument removed from UnmarshalJSONV2
Updates tailscale/corp#791
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
MutexValue is simply a value guarded by a mutex.
For any type that is not pointer-sized,
MutexValue will perform much better than AtomicValue
since it will not incur an allocation boxing the value
into an interface value (which is how Go's atomic.Value
is implemented under-the-hood).
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The omitzero tag option has been backported to v1 "encoding/json"
from the "encoding/json/v2" prototype and will land in Go1.24.
Until we fully upgrade to Go1.24, adjust the test to be agnostic
to which version of Go someone is using.
Updates tailscale/corp#25406
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The IfElse function is equivalent to the ternary (c ? a : b) operator
in many other languages like C. Unfortunately, this function
cannot perform short-circuit evaluation like in many other languages,
but this is a restriction that's not much different
than the pre-existing cmp.Or function.
The argument against ternary operators in Go is that
nested ternary operators become unreadable
(e.g., (c1 ? (c2 ? a : b) : (c2 ? x : y))).
But a single layer of ternary expressions can sometimes
make code much more readable.
Having the bools.IfElse function gives code authors the
ability to decide whether use of this is more readable or not.
Obviously, code authors will need to be judicious about
their use of this helper function.
Readability is more of an art than a science.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Throughout our codebase we have types that only exist only
to implement an io.Reader or io.Writer, when it would have been
simpler, cleaner, and more readable to use an inlined function literal
that closes over the relevant types.
This is arguably more readable since it keeps the semantic logic
in place rather than have it be isolated elsewhere.
Note that a function literal that closes over some variables
is semantic equivalent to declaring a struct with fields and
having the Read or Write method mutate those fields.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This package grew organically over time and
is an awful mix of explicitly declared options and
globally set parameters via environment variables and
other subtle effects.
Add a new Options and TransportOptions type to
allow for the creation of a Policy or http.RoundTripper
with some set of options.
The options struct avoids the need to add yet more
NewXXX functions for every possible combination of
ordered arguments.
The goal of this refactor is to allow specifying the http.Client
to use with the Policy.
Updates tailscale/corp#18177
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Re-use a pre-allocated bytes.Buffer struct and
shallow the copy the result of bytes.NewBuffer into it
to avoid allocating the struct.
Note that we're only reusing the bytes.Buffer struct itself
and not the underling []byte temporarily stored within it.
Updates #cleanup
Updates tailscale/corp#18514
Updates golang/go#67004
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The bools.Compare function compares boolean values
by reporting -1, 0, +1 for ordering so that it can be easily
used with slices.SortFunc.
Updates #cleanup
Updates tailscale/corp#11038
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
One primary purpose of WithLock is to mutate the underlying map.
However, this can lead to a panic if it happens to be nil.
Thus, always allocate a map before passing it to f.
Updates tailscale/corp#11038
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Add Keys, Values, and All to iterate over
all keys, values, and entries, respectively.
Updates #11038
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Some operations cannot be implemented with the prior API:
* Iterating over the map and deleting keys
* Iterating over the map and replacing items
* Calling APIs that expect a native Go map
Add a Map.WithLock method that acquires a write-lock on the map
and then calls a user-provided closure with the underlying Go map.
This allows users to interact with the Map as a regular Go map,
but with the gaurantees that it is concurrent safe.
Updates tailscale/corp#9115
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The Add method derives a new ID by adding a signed integer
to the ID, treating it as an unsigned 256-bit big-endian integer.
We also add Less and Compare methods to PrivateID to provide
feature parity with existing methods on PublicID.
Updates tailscale/corp#11038
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
If AtomicValue[T] is used with a T that is an interface kind,
then Store may panic if different concret types are ever stored.
Fix this by always wrapping in a concrete type.
Technically, this is only needed if T is an interface kind,
but there is no harm in doing it also for non-interface kinds.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
There is an undocumented 16KiB limit for text log messages.
However, the limit for JSON messages is 256KiB.
Even worse, logging JSON as text results in significant overhead
since each double quote needs to be escaped.
Instead, use logger.Logf.JSON to explicitly log the info as JSON.
We also modify osdiag to return the information as structured data
rather than implicitly have the package log on our behalf.
This gives more control to the caller on how to log.
Updates #7802
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Changes made:
* Avoid "encoding/json" for JSON processing, and instead use
"github.com/go-json-experiment/json/jsontext".
Use jsontext.Value.IsValid for validation, which is much faster.
Use jsontext.AppendQuote instead of our own JSON escaping.
* In drainPending, use a different maxLen depending on lowMem.
In lowMem mode, it is better to perform multiple uploads
than it is to construct a large body that OOMs the process.
* In drainPending, if an error is encountered draining,
construct an error message in the logtail JSON format
rather than something that is invalid JSON.
* In appendTextOrJSONLocked, use jsontext.Decoder to check
whether the input is a valid JSON object. This is faster than
the previous approach of unmarshaling into map[string]any and
then re-marshaling that data structure.
This is especially beneficial for network flow logging,
which produces relatively large JSON objects.
* In appendTextOrJSONLocked, enforce maxSize on the input.
If too large, then we may end up in a situation where the logs
can never be uploaded because it exceeds the maximum body size
that the Tailscale logs service accepts.
* Use "tailscale.com/util/truncate" to properly truncate a string
on valid UTF-8 boundaries.
* In general, remove unnecessary spaces in JSON output.
Performance:
name old time/op new time/op delta
WriteText 776ns ± 2% 596ns ± 1% -23.24% (p=0.000 n=10+10)
WriteJSON 110µs ± 0% 9µs ± 0% -91.77% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
WriteText 448B ± 0% 0B -100.00% (p=0.000 n=10+10)
WriteJSON 37.9kB ± 0% 0.0kB ± 0% -99.87% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
WriteText 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
WriteJSON 1.08k ± 0% 0.00k ± 0% -99.91% (p=0.000 n=10+10)
For text payloads, this is 1.30x faster.
For JSON payloads, this is 12.2x faster.
Updates #cleanup
Updates tailscale/corp#18514
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Buffer.Write has the exact same signature of io.Writer.Write.
The latter requires that implementations to never retain
the provided input buffer, which is an expectation that most
users will have when they see a Write signature.
The current behavior of Buffer.Write where it does retain
the input buffer is a risky precedent to set.
Switch the behavior to match io.Writer.Write.
There are only two implementations of Buffer in existence:
* logtail.memBuffer
* filch.Filch
The former can be fixed by cloning the input to Write.
This will cause an extra allocation in every Write,
but we can fix that will pooling on the caller side
in a follow-up PR.
The latter only passes the input to os.File.Write,
which does respect the io.Writer.Write requirements.
Updates #cleanup
Updates tailscale/corp#18514
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
There are no mutations to the input,
so we can support both ~string and ~[]byte just fine.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Specifying a smaller window size during compression
provides a knob to tweak the tradeoff between memory usage
and the compression ratio.
Updates tailscale/corp#18514
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This is based on empirical testing using actual logs data.
FastestCompression only incurs a marginal <1% compression ratio hit
for a 2.25x reduction in memory use for small payloads
(which are common if log uploads happen at a decently high frequency).
The memory savings for large payloads is much lower
(less than 1.1x reduction).
LowMemory only incurs a marginal <5% hit on performance
for a 1.6-2.0x reduction in memory use for small or large payloads.
The memory gains for both settings justifies the loss of benefits,
which are arguably minimal.
tailscale/corp#18514
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Use the zstdframe package where sensible instead of plumbing
around our own zstd.Encoder just for stateless operations.
This causes logtail to have a dependency on zstd,
but that's arguably okay since zstd support is implicit
to the protocol between a client and the logging service.
Also, virtually every caller to logger.NewLogger was
manually setting up a zstd.Encoder anyways,
meaning that zstd was functionally always a dependency.
Updates #cleanup
Updates tailscale/corp#18514
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The Go zstd package is not friendly for stateless zstd compression.
Passing around multiple zstd.Encoder just for stateless compression
is a waste of memory since the memory is never freed and seldom
used if no compression operations are happening.
For performance, we pool the relevant Encoder/Decoder
with the specific options set.
Functionally, this package is a wrapper over the Go zstd package
with a more ergonomic API for stateless operations.
This package can be used to cleanup various pre-existing zstd.Encoder
pools or one-off handlers spread throughout our codebases.
Performance:
BenchmarkEncode/Best 1690 610926 ns/op 25.78 MB/s 1 B/op 0 allocs/op
zstd_test.go:137: memory: 50.336 MiB
zstd_test.go:138: ratio: 3.269x
BenchmarkEncode/Better 10000 100939 ns/op 156.04 MB/s 0 B/op 0 allocs/op
zstd_test.go:137: memory: 20.399 MiB
zstd_test.go:138: ratio: 3.131x
BenchmarkEncode/Default 15775 74976 ns/op 210.08 MB/s 105 B/op 0 allocs/op
zstd_test.go:137: memory: 1.586 MiB
zstd_test.go:138: ratio: 3.064x
BenchmarkEncode/Fastest 23222 53977 ns/op 291.81 MB/s 26 B/op 0 allocs/op
zstd_test.go:137: memory: 599.458 KiB
zstd_test.go:138: ratio: 2.898x
BenchmarkEncode/FastestLowMemory 23361 50789 ns/op 310.13 MB/s 15 B/op 0 allocs/op
zstd_test.go:137: memory: 334.458 KiB
zstd_test.go:138: ratio: 2.898x
BenchmarkEncode/FastestNoChecksum 23086 50253 ns/op 313.44 MB/s 26 B/op 0 allocs/op
zstd_test.go:137: memory: 599.458 KiB
zstd_test.go:138: ratio: 2.900x
BenchmarkDecode/Checksum 70794 17082 ns/op 300.96 MB/s 4 B/op 0 allocs/op
zstd_test.go:163: memory: 316.438 KiB
BenchmarkDecode/NoChecksum 74935 15990 ns/op 321.51 MB/s 4 B/op 0 allocs/op
zstd_test.go:163: memory: 316.438 KiB
BenchmarkDecode/LowMemory 71043 16739 ns/op 307.13 MB/s 0 B/op 0 allocs/op
zstd_test.go:163: memory: 79.347 KiB
We can see that the options are taking effect where compression ratio improves
with higher levels and compression speed diminishes.
We can also see that LowMemory takes effect where the pooled coder object
references less memory than other cases.
We can see that the pooling is taking effect as there are 0 amortized allocations.
Additional performance:
BenchmarkEncodeParallel/zstd-24 1857 619264 ns/op 1796 B/op 49 allocs/op
BenchmarkEncodeParallel/zstdframe-24 1954 532023 ns/op 4293 B/op 49 allocs/op
BenchmarkDecodeParallel/zstd-24 5288 197281 ns/op 2516 B/op 49 allocs/op
BenchmarkDecodeParallel/zstdframe-24 6441 196254 ns/op 2513 B/op 49 allocs/op
In concurrent usage, handling the pooling in this package
has a marginal benefit over the zstd package,
which relies on a Go channel as the pooling mechanism.
In particular, coders can be freed by the GC when not in use.
Coders can be shared throughout the program if they use this package
instead of multiple independent pools doing the same thing.
The allocations are unrelated to pooling as they're caused by the spawning of goroutines.
Updates #cleanup
Updates tailscale/corp#18514
Updates tailscale/corp#17653
Updates tailscale/corp#18005
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Providing a hash.Block512 is an implementation detail of how deephash
works today, but providing an opaque type with mostly equivalent API
(i.e., HashUint8, HashBytes, etc. methods) is still sensible.
Thus, define a public Hasher type that exposes exactly the API
that an implementation of SelfHasher would want to call.
This gives us freedom to change the hashing algorithm of deephash
at some point in the future.
Also, this type is likely going to be called by types that are
going to memoize their own hash results, we additionally add
a HashSum method to simplify this use case.
Add documentation to SelfHasher on how a type might implement it.
Updates: corp#16409
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Also perform minor cleanups on the ctxkey package itself.
Provide guidance on when to use ctxkey.Key[T] over ctxkey.New.
Also, allow for interface kinds because the value wrapping trick
also happens to fix edge cases with interfaces in Go.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The lack of type-safety in context.WithValue leads to the common pattern
of defining of package-scoped type to ensure global uniqueness:
type fooKey struct{}
func withFoo(ctx context, v Foo) context.Context {
return context.WithValue(ctx, fooKey{}, v)
}
func fooValue(ctx context) Foo {
v, _ := ctx.Value(fooKey{}).(Foo)
return v
}
where usage becomes:
ctx = withFoo(ctx, foo)
foo := fooValue(ctx)
With many different context keys, this can be quite tedious.
Using generics, we can simplify this as:
var fooKey = ctxkey.New("mypkg.fooKey", Foo{})
where usage becomes:
ctx = fooKey.WithValue(ctx, foo)
foo := fooKey.Value(ctx)
See https://go.dev/issue/49189
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Simply reading the taildrop directory can pop up security dialogs
on platforms like macOS. Avoid this by only performing garbage collection
of partial and deleted files after the first received taildrop file,
which would have prompted the security dialog window.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Go has no way to explicitly identify Go struct as effectively a tuple,
so staticcheck assumes any external use of unkeyed literals is wrong.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
It is possible that upon a cold-start, we enqueue a partial file
for deletion that is resumed shortly after startup.
If the file transfer happens to last longer than deleteDelay,
we will delete the partial file, which is unfortunate.
The client spent a long time uploading a file,
only for it to be accidentally deleted.
It's a very rare race, but also a frustrating one
if it happens to manifest.
Fix the code to only delete partial files that
do not have an active puts against it.
We also fix a minor bug in ResumeReader
where we read b[:blockSize] instead of b[:cs.Size].
The former is the fixed size of 64KiB,
while the latter is usually 64KiB,
but may be less for the last block.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The progress printer was buggy where it would not print correctly
and some of the truncation logic was faulty.
The progress printer now prints something like:
go1.21.3.linux-amd64.tar.gz 21.53MiB 13.83MiB/s 33.88% ETA 00:00:03
where it shows
* the number of bytes transferred so far
* the rate of bytes transferred
(using a 1-second half-life for an exponentially weighted average)
* the progress made as a percentage
* the estimated time
(as calculated from the rate of bytes transferred)
Other changes:
* It now correctly prints the progress for very small files
* It prints at a faster rate (4Hz instead of 1Hz)
* It uses IEC units for byte quantities
(to avoid ambiguities of "kb" being kilobits or kilobytes)
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Previously, the test simply relied on:
defer close()
to cleanup file handles.
This works fine on Unix-based systems,
but not on Windows, which dislikes deleting files
where an open file handle continues to exist.
Fix the test by explicitly closing the file handle
after we are done with the resource.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The LoadFunc loads a value and calls a user-provided function.
The utility of this method is to ensure that the map lock is held
while executing user-provided logic.
This allows us to solve TOCTOU bugs that would be nearly imposible
to the solve without this API.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
WaitGroup.Wait should not be concurrently called WaitGroup.Add.
In other words, we should not start new goroutines after shutodwn is called.
Thus, add a conditional to check that shutdown has not been called
before starting off a new waitAndDelete goroutine.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
While the previous logic was correct, it did not perform well.
Resuming is a dance between the client and server, where
1. the client requests hashes for a partial file,
2. the server then computes those hashes,
3. the client computes hashes locally and compares them.
4. goto 1 while the partial file still has data
While step 2 is running, the client is sitting idle.
While step 3 is running, the server is sitting idle.
By streaming over the block hash immediately after the server
computes it, the client can start checking the hash,
while the server works on the next hash (in a pipelined manner).
This performs dramatically better and also uses less memory
as we don't need to hold a list of hashes, but only need to
handle one hash at a time.
There are two detriments to this approach:
* The HTTP API relies on a JSON stream,
which is not a standard REST-like pattern.
However, since we implement both client and server,
this is fine.
* While the stream is on-going, we hold an open file handle
on the server side while the file is being hashed.
On really slow streams, this could hold a file open forever.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
Minor fixes:
* The branch for listing or hashing partial files was inverted.
* The host for peerapi call needs to be real (rather than bogus).
* Handle remote peers that don't support resuming.
* Make resume failures non-fatal (since we can still continue).
This was tested locally, end-to-end system test is future work.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
File resumption requires keeping partial files around for some time,
but we must still eventually delete them if never resumed.
Thus, we implement asynchronous file deletion, which could
spawn a background goroutine to delete the files.
We also use the same mechanism for deleting files on Windows,
where a file can't be deleted if there is still an open file handle.
We can enqueue those with the asynchronous file deleter as well.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This adds support for parsing Range and Content-Range headers
according to RFC 7230. The package could be extended in the future
to handle other headers.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Perform the same m==nil check in Manager.{PartialFiles,HashPartialFile}
as we do in the other methods.
Fix HashPartialFile is properly handle a length of -1.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
We add the following API:
* type FileChecksums
* type Checksum
* func Manager.PartialFiles
* func Manager.HashPartialFile
* func ResumeReader
The Manager methods provide the ability to query for partial files
and retrieve a list of checksums for a given partial file.
The ResumeReader function is a helper that wraps an io.Reader
to discard content that is identical locally and remotely.
The FileChecksums type represents the checksums of a file
and is safe to JSON marshal and send over the wire.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
In almost every single use of Clock, there is a default behavior
we want to use when the interface is nil,
which is to use the the standard time package.
The Clock interface exists only for testing,
and so tests that care about mocking time
can adequately plumb the the Clock down the stack
and through various data structures.
However, the problem with Clock is that there are many
situations where we really don't care about mocking time
(e.g., measuring execution time for a log message),
where making sure that Clock is non-nil is not worth the burden.
In fact, in a recent refactoring, the biggest pain point was
dealing with nil-interface panics when calling tstime.Clock methods
where mocking time wasn't even needed for the relevant tests.
This required wasted time carefully reviewing the code to
make sure that tstime.Clock was always populated,
and even then we're not statically guaranteed to avoid a nil panic.
Ideally, what we want are default methods on Go interfaces,
but such a language construct does not exist.
However, we can emulate that behavior by declaring
a concrete type that embeds the interface.
If the underlying interface value is nil,
it provides some default behavior (i.e., use StdClock).
This provides us a nice balance of two goals:
* We can plumb tstime.DefaultClock in all relevant places
for use with mocking time in the tests that care.
* For all other logic that don't care about,
we never need to worry about whether tstime.DefaultClock
is nil or not. This is especially relevant in production code
where we don't want to panic.
Longer-term, we may want to perform a large-scale change
where we rename Clock to ClockInterface
and rename DefaultClock to just Clock.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Changes made:
* Move all HTTP related functionality from taildrop to ipnlocal.
* Add two arguments to taildrop.Manager.PutFile to specify
an opaque client ID and a resume offset (both unused for now).
* Cleanup the logic of taildrop.Manager.PutFile
to be easier to follow.
* Implement file conflict handling where duplicate files are renamed
(e.g., "IMG_1234.jpg" -> "IMG_1234 (2).jpg").
* Implement file de-duplication where "renaming" a partial file
simply deletes it if it already exists with the same contents.
* Detect conflicting active puts where a second concurrent put
results in an error.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
Changes made:
* Unexport declarations specific to internal taildrop functionality.
* Document all exported functionality.
* Move TestRedactErr to the taildrop package.
* Rename and invert Handler.DirectFileDoFinalRename as AvoidFinalRename.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This is being moved to taildrop, so clean it up to stop depending
on so much unreleated functionality by removing a dependency
on peerAPIHandler.
Updates tailscale/corp#14772
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
Update github.com/go-json-experiment/json to the latest version
and fix the build in light of some breaking API changes.
Updates #cleanup
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
9538e9f970 broke LocalBackend.WhoIs
where you can no longer lookup yourself in WhoIs.
This occurs because the LocalBackend.peers map only contains peers.
If we fail to lookup a peer, double-check whether it is ourself.
Fixes#9470
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
This eventually allows encoding packages that may respect
the proposed encoding.TextAppender interface.
The performance gains from this is between 10-30%.
Updates tailscale/corp#14379
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This makes it more maintainable for other code to statically depend
on the exact value of this string. It also makes it easier to
identify what code might depend on this string by looking up
references to this constant.
Updates tailscale/corp#13777
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
If the connection provided to sftp.NewServer is closed,
Serve returns the io.EOF error verbatim from io.Reader.Read.
This is an odd error since this is an expected situation,
so we manually ignore io.EOF.
This is somewhat buggy since the sftp package itself
incorrectly reports io.EOF in cases where it should actually
be reporting io.ErrUnexpectedEOF.
See https://github.com/pkg/sftp/pull/554 which patches Serve to
return nil on clean closes and fixes buggy uses of io.ReadFull.
Fixes#8592
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Calling both mono.Now() and time.Now() is slow and
leads to unnecessary precision errors.
Instead, directly compute mono.Time relative to baseMono and baseWall.
This is the opposite calculation as mono.Time.WallTime.
Updates tailscale/corp#8427
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
There are two race conditions in output handling.
The first race condition is due to a misuse of exec.Cmd.StdoutPipe.
The documentation explicitly forbids concurrent use of StdoutPipe
with exec.Cmd.Wait (see golang/go#60908) because Wait will
close both sides of the pipe once the process ends without
any guarantees that all data has been read from the pipe.
To fix this, we allocate the os.Pipes ourselves and
manage cleanup ourselves when the process has ended.
The second race condition is because sshSession.run waits
upon exec.Cmd to finish and then immediately proceeds to call ss.Exit,
which will close all output streams going to the SSH client.
This may interrupt any asynchronous io.Copy still copying data.
To fix this, we close the write-side of the os.Pipes after
the process has finished (and before calling ss.Exit) and
synchronously wait for the io.Copy routines to finish.
Fixes#7601
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Maisem Ali <maisem@tailscale.com>
The retry logic was pathological in the following ways:
* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.
* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.
We modify the retry logic in the following ways:
* We now respect the "Retry-After" header sent by the logging service.
* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.
* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.
Updates #tailscale/corp#11213
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Exclude traffic with 100.100.100.100 (for IPv4) and
with fd7a:115c:a1e0::53 (for IPv6) since this traffic with the
Tailscale service running locally on the node.
This traffic never left the node.
It also happens to be a high volume amount of traffic since
DNS requests occur over UDP with each request coming from a
unique port, thus resulting in many discrete traffic flows.
Fixestailscale/corp#10554
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The addition of WaitGroup.Go in the standard library has been
repeatedly proposed and rejected.
See golang/go#18022, golang/go#23538, and golang/go#39863
In summary, the argument for WaitGroup.Go is that it avoids bugs like:
go func() {
wg.Add(1)
defer wg.Done()
...
}()
where the increment happens after execution (not before)
and also (to a lesser degree) because:
wg.Go(func() {
...
})
is shorter and more readble.
The argument against WaitGroup.Go is that the provided function
takes no arguments and so inputs and outputs must closed over
by the provided function. The most common race bug for goroutines
is that the caller forgot to capture the loop iteration variable,
so this pattern may make it easier to be accidentally racy.
However, that is changing with golang/go#57969.
In my experience the probability of race bugs due to the former
still outwighs the latter, but I have no concrete evidence to prove it.
The existence of errgroup.Group.Go and frequent utility of the method
at least proves that this is a workable pattern and
the possibility of accidental races do not appear to
manifest as frequently as feared.
A reason *not* to use errgroup.Group everywhere is that there are many
situations where it doesn't make sense for the goroutine to return an error
since the error is handled in a different mechanism
(e.g., logged and ignored, formatted and printed to the frontend, etc.).
While you can use errgroup.Group by always returning nil,
the fact that you *can* return nil makes it easy to accidentally return
an error when nothing is checking the return of group.Wait.
This is not a hypothetical problem, but something that has bitten us
in usages that was only using errgroup.Group without intending to use
the error reporting part of it.
Thus, add a (yet another) variant of WaitGroup here that
is identical to sync.WaitGroup, but with an extra method.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Add Value, which measures the rate at which an event occurs,
exponentially weighted towards recent activity.
It is guaranteed to occupy O(1) memory, operate in O(1) runtime,
and is safe for concurrent use.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The time.Parse function has been optimized to the point
where it is faster than our custom implementation.
See upstream changes in:
* https://go.dev/cl/429862
* https://go.dev/cl/425197
* https://go.dev/cl/425116
Performance:
BenchmarkGoParse3339/Z 38.75 ns/op 0 B/op 0 allocs/op
BenchmarkGoParse3339/TZ 54.02 ns/op 0 B/op 0 allocs/op
BenchmarkParse3339/Z 40.17 ns/op 0 B/op 0 allocs/op
BenchmarkParse3339/TZ 87.06 ns/op 0 B/op 0 allocs/op
We can see that the stdlib implementation is now faster.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Share the same underlying implementation for both PrivateID and PublicID.
For the shared methods, declare them in the same order.
Only keep documentation on methods without obvious meaning.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Ever since the introduction of the "must" package,
most MustXXX functions are no longer necessary.
Remove this as it is no longer depended upon
from within this repository and by the internal private repository.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The log ID types were moved to a separate package so that
code that only depend on log ID types do not need to link
in the logic for the logtail client itself.
Not all code need the logtail client.
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
The Tailscale logging service has a hard limit on the maximum
log message size that can be accepted.
We want to ensure that netlog messages never exceed
this limit otherwise a client cannot transmit logs.
Move the goroutine for periodically dumping netlog messages
from wgengine/netlog to net/connstats.
This allows net/connstats to manage when it dumps messages,
either based on time or by size.
Updates tailscale/corp#8427
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Joe TsaiandGitHub
