tstest/natlab/vmtest: serialize ensureDebugSSHKey across parallel boots

Env.Start boots all VM nodes in parallel; each calls
createCloudInitISO -> ensureDebugSSHKey concurrently. When
/tmp/vmtest_key doesn't yet exist, the first goroutine creates it
with os.WriteFile, which opens with O_CREATE|O_TRUNC and briefly
leaves the file existing-but-empty between the open and the
subsequent write. A concurrent goroutine that hits that window
sees ReadFile succeed with zero bytes, then fails ssh.ParsePrivateKey
with "ssh: no key found", causing boot to fail with:

  boot: creating cloud-init ISO: parse /tmp/vmtest_key: ssh: no key found

Observed in CI on TestSiteToSite (3 nodes). Wrap the function in
a package-level Mutex so the first caller fully writes the key
before any other caller reads it.

Updates #20228

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ie6399dcba0c397bb8041931d3de1c6063a11c568

tstest/natlab/vmtest/cloudinit.go

@@ -12,6 +12,7 @@ import (
 	"path/filepath"
 	"sort"
 	"strings"
+	"sync"
 
 	"github.com/creachadair/mds/shell"
 	"github.com/kdomanski/iso9660"
@@ -214,7 +215,16 @@ func tailscaledEnvPrefix(n *Node) string {
 	return strings.Join(parts, " ") + " "
 }
 
+// debugSSHKeyMu serializes ensureDebugSSHKey calls. Env.Start boots all
+// nodes in parallel and each calls createCloudInitISO -> ensureDebugSSHKey,
+// which would otherwise race on creating /tmp/vmtest_key: one goroutine
+// can observe a newly-created-but-still-empty key file written by another
+// and fail to parse it.
+var debugSSHKeyMu sync.Mutex
+
 func ensureDebugSSHKey() error {
+	debugSSHKeyMu.Lock()
+	defer debugSSHKeyMu.Unlock()
 	const keyPath = "/tmp/vmtest_key"
 	if privPEM, err := os.ReadFile(keyPath); err == nil {
 		if _, err := os.Stat(keyPath + ".pub"); err == nil {