In addition to printing goroutine stacks, explicitly track all in-flight
operations and print them when the watchdog triggers (along with the
time they were started at). This should make debugging watchdog failures
easier, since we can look at the longest-running operation(s) first.
Signed-off-by: Andrew Dunham <andrew@tailscale.com>
Signed-off-by: Andrew Dunham <andrew@tailscale.com>
Prior to this change, if BIRD stops responding wgengine.watchdogEngine
will crash tailscaled.
This happens because in wgengine.userspaceEngine, we end up blocking
forever trying to write a request to or read a response from BIRD with
wgLock held, and then future watchdog'd calls will block on acquiring
that mutex until the watchdog kills the process. With the timeout, we at
least get the chance to print an error message and decide whether we
want to crash or not.
Updates tailscale/coral#72
Signed-off-by: Andrew Dunham <andrew@tailscale.com>
Signed-off-by: Andrew Dunham <andrew@tailscale.com>
Andrew DunhamandGitHub