tstest/largetailnet, tstest/integration/testcontrol: add in-process large-tailnet benchmark
Add a Go benchmark that exercises a single tailnet client (a [tsnet.Server]
running in the test process) against a synthetic large initial netmap and
a stream of caller-driven peer add/remove deltas, all in-process.
The harness is split in two parts:
- tstest/largetailnet, a reusable package containing a [Streamer]
that hijacks the map long-poll on a [testcontrol.Server] via the new
AltMapStream hook, sends one initial MapResponse with N synthetic
peers, and forwards caller-supplied delta MapResponses on the same
stream. Helpers like MakePeer / AllocPeer build synthetic peers with
unique IDs and addresses derived from the Tailscale ULA range.
- tstest/largetailnet/largetailnet_test.go, BenchmarkGiantTailnet
(headless tailscaled workload, no IPN bus subscriber) and
BenchmarkGiantTailnetBusWatcher (GUI-client workload with one
Notify subscriber attached). Both are gated on
--actually-test-giant-tailnet (skipped by default), stand up an
in-process testcontrol + tsnet.Server, let Up block until the
initial N-peer netmap has been processed, then ResetTimer and run
add+remove pairs via b.Loop. Per-delta sync is via a test-only
[ipnlocal.LocalBackend.AwaitNodeKeyForTest] channel that closes
once the just-added peer key appears in the netmap (no-watcher
variant) or via bus-Notify drain (bus-watcher variant).
To support the hijack, [testcontrol.Server] grows an AltMapStream hook
and a small MapStreamWriter interface for benchmarks/stress tests that
need to drive a controlled MapResponse sequence; the normal serveMap
path is untouched when AltMapStream is nil. The streamer answers
non-streaming "lite" map polls (which controlclient issues before the
streaming long-poll to push HostInfo) with an empty MapResponse and
returns immediately, so the streaming poll that follows is the one
that gets the initial netmap.
The benchmark is intended for before/after comparisons of netmap- and
delta-handling changes targeted at large tailnets. CPU profiles on
unmodified main show the expected O(N) hotspots:
setControlClientStatusLocked / authReconfigLocked /
userspaceEngine.Reconfig / setNetMapLocked, plus JSON encoding of the
full Notify.NetMap to bus watchers (which dominates the BusWatcher
variant).
Median ms/op over 10 runs on unmodified main, by tailnet size N:
N no-watcher bus-watcher
10000 32 166
50000 222 865
100000 504 1765
250000 1551 4696
Recommended invocation:
go test ./tstest/largetailnet/ -run=^$ \
-bench='BenchmarkGiantTailnet(BusWatcher)?$' \
-benchtime=2000x -timeout=10m \
--actually-test-giant-tailnet \
--giant-tailnet-n=250000 \
-cpuprofile=/tmp/giant.cpu.pprof
Updates #12542
Change-Id: I4f5b2bb271a36ba853d5a0ffe82054ef2b15c585
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This commit is contained in:
committed by
Brad Fitzpatrick
parent
33342aec32
commit
ad5436af0d
@@ -89,6 +89,9 @@ type Server struct {
|
||||
// MapResponse stream to modify the first MapResponse sent in response to it.
|
||||
ModifyFirstMapResponse func(*tailcfg.MapResponse, *tailcfg.MapRequest)
|
||||
|
||||
// AltMapStream, if non-nil, takes over serveMap. See [AltMapStreamFunc].
|
||||
AltMapStream AltMapStreamFunc
|
||||
|
||||
initMuxOnce sync.Once
|
||||
mux *http.ServeMux
|
||||
|
||||
@@ -1144,6 +1147,15 @@ func (s *Server) serveMap(w http.ResponseWriter, r *http.Request, mkey key.Machi
|
||||
go panic(fmt.Sprintf("bad map request: %v", err))
|
||||
}
|
||||
|
||||
if s.AltMapStream != nil {
|
||||
// The caller takes over the stream entirely; it must handle
|
||||
// keeping the HTTP response alive until ctx is done.
|
||||
compress := req.Compress != ""
|
||||
w.WriteHeader(200)
|
||||
s.AltMapStream(ctx, &mapStreamSender{s: s, w: w, compress: compress}, req)
|
||||
return
|
||||
}
|
||||
|
||||
jitter := rand.N(8 * time.Second)
|
||||
keepAlive := 50*time.Second + jitter
|
||||
|
||||
@@ -1486,12 +1498,51 @@ func (s *Server) takeRawMapMessage(nk key.NodePublic) (mapResJSON []byte, ok boo
|
||||
return mapResJSON, true
|
||||
}
|
||||
|
||||
// AltMapStreamFunc is the type of [Server.AltMapStream]: a callback that
|
||||
// takes over the serveMap handler entirely. The callback hand-builds and
|
||||
// sends MapResponses via the provided [MapStreamWriter] and is responsible
|
||||
// for keeping the stream alive until ctx is done. When set, the normal
|
||||
// per-node map-stream state machine in serveMap is bypassed.
|
||||
//
|
||||
// The callback is invoked for every map long-poll, including the
|
||||
// non-streaming "lite" polls controlclient issues to push HostInfo updates
|
||||
// (req.Stream == false). Implementations that only care about the streaming
|
||||
// long-poll typically respond to non-streaming polls with an empty
|
||||
// MapResponse and return immediately.
|
||||
//
|
||||
// This hook is for benchmarks and stress tests that need to drive clients
|
||||
// with a controlled sequence of responses.
|
||||
type AltMapStreamFunc func(ctx context.Context, w MapStreamWriter, req *tailcfg.MapRequest)
|
||||
|
||||
// MapStreamWriter is the interface passed to an [AltMapStreamFunc],
|
||||
// letting the callback write framed MapResponse messages directly onto the
|
||||
// long-poll HTTP response.
|
||||
type MapStreamWriter interface {
|
||||
// SendMapMessage encodes and writes msg as a single framed
|
||||
// MapResponse on the stream. It respects the client's Compress flag
|
||||
// (captured when the stream started).
|
||||
SendMapMessage(msg *tailcfg.MapResponse) error
|
||||
}
|
||||
|
||||
// mapStreamSender implements [MapStreamWriter] for [Server.AltMapStream]
|
||||
// callbacks.
|
||||
type mapStreamSender struct {
|
||||
s *Server
|
||||
w http.ResponseWriter
|
||||
compress bool
|
||||
}
|
||||
|
||||
func (m *mapStreamSender) SendMapMessage(msg *tailcfg.MapResponse) error {
|
||||
return m.s.sendMapMsg(m.w, m.compress, msg)
|
||||
}
|
||||
|
||||
func (s *Server) sendMapMsg(w http.ResponseWriter, compress bool, msg any) error {
|
||||
resBytes, err := s.encode(compress, msg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(resBytes) > 16<<20 {
|
||||
const maxMapSize = 256 << 20 // 256MB
|
||||
if len(resBytes) > maxMapSize {
|
||||
return fmt.Errorf("map message too big: %d", len(resBytes))
|
||||
}
|
||||
var siz [4]byte
|
||||
|
||||
Reference in New Issue
Block a user