webnet/tailscale
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

ipn/ipnlocal, control/controlclient: process node adds/removes in constant time

Browse Source 
For large tailnets (~50k+ nodes) with frequent peer churn (ephemeral
GitHub Actions workers etc.), tailscaled used to rebuild the full
netmap and fan it out on the IPN bus on every MapResponse that
added or removed a peer. There were two O(N) costs per delta: the
full netmap rebuild + every Notify.NetMap encode to every bus watcher.

This change tackles both:

  1. Plumb O(1) peer add/remove through the delta path. PeersChanged
     and PeersRemoved no longer prevent the delta happy path; instead,
     they mutate the per-node-backend peer map in place.

  2. Restrict ipn.Notify.NetMap emission to the platforms whose host
     GUIs still depend on it (Windows, macOS, iOS) and migrate
     in-tree consumers off it everywhere else:

     - Migrate reactive consumers (containerboot, kube agents,
       sniproxy, tsconsensus, etc.) off Notify.NetMap to the
       previously-added Notify.SelfChange signal so they no longer
       have to subscribe to the full netmap.
     - Add ipn.NotifyNoNetMap so GUI clients on "legacy-emit" platforms
       that have already migrated can opt out of the per-watcher
       NetMap encode.
     - Gate Notify.NetMap emission on the producer side by a compile-
       time GOOS check, so the supporting code is dead-code-eliminated
       on Linux and other geese where no GUI consumer needs it.

Re-running BenchmarkGiantTailnet from tstest/largetailnet, which was
added along with baseline numbers on unmodified main in ad5436af0d,
the per-delta cost (one peer add+remove pair) is now ~O(1) regardless
of tailnet size N:

    N         no-watcher (ms/op)            bus-watcher (ms/op)
              before    now     factor      before    now     factor
     10000        32   0.11       300x         166   0.13      1300x
     50000       222   0.11      2000x         865   0.13      6700x
    100000       504   0.12      4100x        1765   0.13     13400x
    250000      1551   0.12     12500x        4696   0.15     32400x

Updates #12542

Change-Id: I94e34b37331d1a8ec74c299deffadf4d061fda9e
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This commit is contained in:
Brad Fitzpatrick
2026-05-21 09:26:19 -07:00
committed by Brad Fitzpatrick
parent 2703f91174
commit aa5da2e5f2
23 changed files with 1521 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ipn/backend.go
+149 -19
View File
@@ -88,10 +88,61 @@ const (
NotifyInitialClientVersion NotifyWatchOpt = 1 << 11 // if set, the first Notify message (sent immediately) will contain the current ClientVersion if available and if update checks are enabled
// NotifyPeerChanges, if set, causes netmap delta updates to be sent as [tailcfg.PeerChange] rather than a full NetMap.
// Full netmap responses from the control plane are still sent as a full NetMap. PeerChanges are only sent to sessions
// that have opted in to this mode.
// NotifyPeerChanges, if set, opts the watcher into peer-set delta
// notifications: [Notify.PeersChanged] (peer added or full-Node
// replaced) and [Notify.PeersRemoved] (peer removed by NodeID).
//
// Without this bit, peer adds/removes/replacements are not delivered
// over the bus at all (consumers fall back to fetching the netmap on
// demand or, on legacy-emit platforms, to watching [Notify.NetMap]).
//
// Watchers that want narrower per-field updates as well (Online,
// LastSeen, DERPHome, Endpoints) should additionally set
// [NotifyPeerPatches]. Without [NotifyPeerPatches], any per-field
// patch tailscaled would have emitted as a [tailcfg.PeerChange] is
// promoted into a full-Node entry in [Notify.PeersChanged] for this
// watcher, so a watcher that opts only into [NotifyPeerChanges] still
// observes every per-peer mutation; it just receives them as full
// Nodes rather than narrow patches. The cost is bus bandwidth.
//
// On platforms where the legacy [Notify.NetMap] is still emitted
// (Windows, macOS, iOS, Android), it is permitted to combine this
// with [NotifyInitialNetMap] for backwards compatibility. New code
// should pair this with [NotifyInitialStatus] instead.
NotifyPeerChanges NotifyWatchOpt = 1 << 12
// NotifyNoNetMap, if set, suppresses the legacy [Notify.NetMap] field on
// runtime (non-initial) Notify messages delivered to this watcher. It
// only matters on platforms where tailscaled still emits NetMap on the
// bus by default — Windows, macOS, and iOS — and is intended for GUI
// clients on those platforms that have migrated to read peers via
// [Notify.PeersChanged] / [LocalClient.NetMap]. The initial-state NetMap
// (sent when [NotifyInitialNetMap] is set) is unaffected.
NotifyNoNetMap NotifyWatchOpt = 1 << 13
// NotifyInitialStatus, if set, causes the first Notify message (sent
// immediately) to contain the current [ipnstate.Status] in
// [Notify.InitialStatus]. Together with [Notify.SelfChange] and
// [Notify.PeersChanged] on subsequent messages, it lets a watcher
// stitch together a continuous view of the local node's state without
// fetching the netmap directly. Prefer this over [LocalClient.NetMap]
// for new code that wants a stable, client-facing snapshot type.
NotifyInitialStatus NotifyWatchOpt = 1 << 14
// NotifyPeerPatches, if set, opts the watcher into narrow per-field
// peer patches via [Notify.PeerChangedPatch]. It implies
// [NotifyPeerChanges]: a watcher with [NotifyPeerPatches] also
// receives [Notify.PeersChanged] and [Notify.PeersRemoved].
//
// This is the lower-bandwidth mode: changes to fields that fit in a
// [tailcfg.PeerChange] (currently Online, LastSeen, DERPHome,
// Endpoints) ride as patches; only changes that don't fit ride as
// full Nodes in [Notify.PeersChanged].
//
// Without this bit but with [NotifyPeerChanges], the producer
// promotes any patch into a full-Node entry in [Notify.PeersChanged]
// for this session, at the cost of bandwidth.
NotifyPeerPatches NotifyWatchOpt = 1 << 15
)
// Notify is a communication from a backend (e.g. tailscaled) to a frontend
@@ -113,10 +164,9 @@ type Notify struct {
// For State InUseOtherUser, ErrMessage is not critical and just contains the details.
ErrMessage *string
LoginFinished *empty.Message // non-nil when/if the login process succeeded
State *State // if non-nil, the new or current IPN state
Prefs *PrefsView // if non-nil && Valid, the new or current preferences
NetMap *netmap.NetworkMap // if non-nil, the new or current netmap
LoginFinished *empty.Message // non-nil when/if the login process succeeded
State *State // if non-nil, the new or current IPN state
Prefs *PrefsView // if non-nil && Valid, the new or current preferences
// SelfChange, if non-nil, indicates that this node's own [tailcfg.Node]
// has changed: addresses, name, key expiry, capabilities, etc. It carries
@@ -125,15 +175,98 @@ type Notify struct {
// full netmap.
//
// Consumers that need additional state (peers, DNS config, packet
// filter) should react to SelfChange by fetching the relevant bits on
// demand via [LocalClient].
// filter) should react to SelfChange by fetching the full netmap on
// demand via [LocalClient.NetMap].
SelfChange *tailcfg.Node `json:",omitzero"`
// PeerChanges, if non-nil, is a list of [tailcfg.PeerChange] that have occurred since the last
// full netmap update. This is sent in lieu of a full NetMap when [NotifyPeerChanges] is set in
// the session's mask and a netmap update is derived from an incremental MapResponse.
// Full MapResponse updates from the control plane are sent as a full NetMap.
PeerChanges []*tailcfg.PeerChange `json:",omitzero"`
// InitialStatus, if non-nil, is the current [ipnstate.Status]. It is
// only set in the first Notify of a session when the watcher requested
// [NotifyInitialStatus]. Together with subsequent [Notify.SelfChange]
// and [Notify.PeerChanges] messages, it lets a watcher stitch together
// a continuous view of node state without fetching the netmap.
InitialStatus *ipnstate.Status `json:",omitzero"`
// NetMap, if non-nil, is the full network map. New consumers should prefer
// [LocalClient.NetMap] for one-shot fetches and [Notify.SelfChange] /
// [Notify.PeerChanges] for incremental reactive updates; NetMap on the bus
// is the legacy path retained for hosts whose GUIs have not yet finished
// migrating. It is delivered:
//
// - On the initial Notify if the watcher requested
// [NotifyInitialNetMap] (any platform).
// - On subsequent Notify messages, only when tailscaled is running
// on Windows, macOS, or iOS. On Linux and other platforms it is
// always nil after the initial notify.
//
// Deprecated: this field is only populated on Windows, macOS, and iOS and
// is slated for removal in favor of [Notify.InitialStatus] +
// [Notify.SelfChange] / [Notify.PeerChanges], etc, as this field
// doesn't scale.
NetMap *netmap.NetworkMap
// PeerChangedPatch, if non-empty, lists narrow per-field peer patches
// since the last Notify (currently Online, LastSeen, DERPHome,
// Endpoints). It mirrors [tailcfg.MapResponse.PeersChangedPatch].
//
// Peer additions and any peer change that can't be expressed as a
// [tailcfg.PeerChange] travel in [Notify.PeersChanged]; peer removals
// in [Notify.PeersRemoved].
//
// Watchers must opt in to receive this field by setting
// [NotifyPeerPatches]; without that bit (but with [NotifyPeerChanges])
// the producer promotes each patch into a full-Node entry in
// [Notify.PeersChanged] instead.
//
// The [tailcfg.PeerChange] type may grow more fields over time;
// consumers that see a [tailcfg.PeerChange] with a field they don't
// recognize should re-fetch the affected node by NodeID via
// [LocalClient.PeerByID] (an O(1) lookup) to learn its current value
// rather than ignoring the change.
PeerChangedPatch []*tailcfg.PeerChange `json:",omitzero"`
// PeersChanged, if non-empty, lists peers whose full [tailcfg.Node]
// has been added or replaced since the last Notify. A node ID may
// appear here either because it is a brand-new peer or because the
// control plane sent a fresh full Node for an existing peer when the
// change wasn't expressible as a [tailcfg.PeerChange] patch (e.g. a
// CapMap, Addresses, Hostinfo, or Tags change). Consumers should
// upsert by NodeID.
//
// This mirrors [tailcfg.MapResponse.PeersChanged] semantics; peer
// removals travel in [Notify.PeersRemoved] and narrow per-field
// patches in [Notify.PeerChanges].
PeersChanged []*tailcfg.Node `json:",omitzero"`
// PeersRemoved, if non-empty, lists [tailcfg.NodeID]s that have been
// removed from the netmap since the last Notify. See
// [Notify.PeersChanged]. This mirrors
// [tailcfg.MapResponse.PeersRemoved].
PeersRemoved []tailcfg.NodeID `json:",omitzero"`
// UserProfiles, if non-empty, carries [tailcfg.UserProfileView]
// entries that have been added or updated since the last Notify on
// this session. Watchers must opt in via [NotifyPeerChanges] or
// [NotifyPeerPatches]; this field is gated on the same bits as
// [Notify.PeersChanged] / [Notify.PeerChangedPatch] because its
// only purpose is to let those consumers resolve the [tailcfg.UserID]
// referenced by a peer Node.
//
// The producer guarantees that any UserID referenced by a peer in
// a [Notify.PeersChanged] / [Notify.PeerChangedPatch] entry will
// have its profile delivered either earlier on this same session
// (e.g. via the initial NetMap or via an earlier Notify carrying
// UserProfiles) or in this same Notify. A consumer that sees a
// UserID it doesn't recognize on a session that opted in to
// peer-change notifications can treat it as a bug; the
// [LocalClient.UserProfile] LocalAPI fallback exists for sessions
// that didn't subscribe with the peer-change bits or that need to
// look up a UserID for any other reason.
//
// The values are [tailcfg.UserProfileView] so they share backing
// memory with the producer's tracking maps; consumers should treat
// them as read-only and use [tailcfg.UserProfileView.AsStruct] or
// the per-field accessors to read them.
UserProfiles map[tailcfg.UserID]tailcfg.UserProfileView `json:",omitzero"`
Engine *EngineStatus // if non-nil, the new or current wireguard stats
BrowseToURL *string // if non-nil, UI should open a browser right now
@@ -204,14 +337,11 @@ func (n Notify) String() string {
if n.Prefs != nil && n.Prefs.Valid() {
fmt.Fprintf(&sb, "%v ", n.Prefs.Pretty())
}
if n.NetMap != nil {
sb.WriteString("NetMap{...} ")
}
if n.SelfChange != nil {
fmt.Fprintf(&sb, "SelfChange(%v) ", n.SelfChange.StableID)
}
if n.PeerChanges != nil {
fmt.Fprintf(&sb, "PeerChanges(%d) ", len(n.PeerChanges))
if n.PeerChangedPatch != nil {
fmt.Fprintf(&sb, "PeerChangedPatch(%d) ", len(n.PeerChangedPatch))
}
if n.Engine != nil {
fmt.Fprintf(&sb, "wg=%v ", *n.Engine)