tstest/natlab/vmtest: add Fedora + DNS-backend coverage, harden non-KVM boot (#20409)

* tstest/natlab/vmtest: make cloud VM boot robust without KVM

Adding heavier distro images (Fedora) surfaced several ways the cloud VM
boot path breaks under TCG software emulation (no /dev/kvm), especially
with multiple concurrent VMs on few cores.

- Add a virtio-rng device to the cloud path so early boot doesn't block in
  getrandom() waiting for the CRNG to seed.
- When no hardware acceleration is available, relax the stuck-console
  watchdog (tuned for KVM's ~1-2s first output) and serialize VM boots so a
  heavy guest doesn't starve its siblings' emulation threads.
- Bound the bring-up context to the test deadline and dump each VM's console
  on failure, so a hang surfaces as a diagnosable Fatalf instead of an
  opaque `go test -timeout` panic (which skips cleanups).

Fixes tailscale/corp#44794
Updates tailscale/corp#44793

Signed-off-by: Brendan Creane <bcreane@gmail.com>

* tstest/natlab/vmtest: add Fedora and DNS-backend test coverage

Add the first RHEL-family distro and the machinery to assert and provision
distinct DNS backends, so adding a distro isn't "basically equivalent" to
the others.

- Add a Fedora 43 image (NetworkManager + systemd-resolved, SELinux
  enforcing). restorecon-relabel the curl'd binaries so they exec under
  enforcing mode.
- Add DNSBackend/AssertDNSBackend, reading the dns_manager_linux_mode_*
  clientmetric to assert which backend tailscaled selected.
- Add a WithDNSMode node option. WithDNSMode(DNSDirect) masks
  systemd-resolved and writes a plain resolv.conf pointing at natlab's fake
  DNS, forcing the direct backend -- so one image covers multiple backends.

Fixes tailscale/corp#44796
Updates tailscale/corp#44793

Signed-off-by: Brendan Creane <bcreane@gmail.com>

---------

Signed-off-by: Brendan Creane <bcreane@gmail.com>

tstest/natlab/vmtest/cloudinit.go

@@ -18,6 +18,7 @@ import (
 	"github.com/creachadair/mds/shell"
 	"github.com/kdomanski/iso9660"
 	"golang.org/x/crypto/ssh"
+	"tailscale.com/tstest/natlab/vnet"
 )
 
 // createCloudInitISO creates a cidata seed ISO for the given cloud VM node.
@@ -129,12 +130,23 @@ func (e *Env) generateLinuxUserData(n *Node) string {
 	}
 	ud.WriteString("  - [\"chmod\", \"+x\", \"/usr/local/bin/tailscaled\", \"/usr/local/bin/tailscale\", \"/usr/local/bin/tta\"]\n")
 
+	// Apply the bin_t label for SELinux enforcement: we curl the binaries in
+	// rather than installing a package, so nothing else labels them, and
+	// enforcing mode would deny exec. No-op on non-SELinux systems, but only
+	// RHEL-family images ship restorecon.
+	if n.os.Family == LinuxRHEL {
+		ud.WriteString("  - [\"/bin/sh\", \"-c\", \"restorecon -v /usr/local/bin/tailscaled /usr/local/bin/tailscale /usr/local/bin/tta 2>&1 || true\"]\n")
+	}
+
 	// Enable IP forwarding for subnet routers.
 	if n.advertiseRoutes != "" {
 		ud.WriteString("  - [\"sysctl\", \"-w\", \"net.ipv4.ip_forward=1\"]\n")
 		ud.WriteString("  - [\"sysctl\", \"-w\", \"net.ipv6.conf.all.forwarding=1\"]\n")
 	}
 
+	// Provision the requested DNS backend before tailscaled starts.
+	writeLinuxDNSModeSetup(&ud, n.dnsMode)
+
 	// Start tailscaled, either via the stock systemd unit or directly in
 	// the background. --statedir provides a VarRoot so features like
 	// Taildrop (which needs a place to stash incoming files) have a
@@ -157,6 +169,30 @@ func (e *Env) generateLinuxUserData(n *Node) string {
 	return ud.String()
 }
 
+// writeLinuxDNSModeSetup appends cloud-init runcmd entries that provision the
+// guest so tailscaled selects the requested DNS backend. Must run before the
+// tailscaled launch entries. The zero value (DNSDefault) is a no-op. mode is
+// validated in AddNode, so any other unknown value is a bug and panics.
+func writeLinuxDNSModeSetup(ud *strings.Builder, mode DNSMode) {
+	switch mode {
+	default:
+		// AddNode validates the mode, so an unknown value here is a bug.
+		panic(fmt.Sprintf("unhandled DNSMode %q", mode))
+	case DNSDefault:
+		// The empty/zero value: leave the image's DNS config alone, so
+		// systemd-resolved stays enabled and tailscaled selects it (or
+		// whatever the image runs by default). No-op.
+	case DNSDirect:
+		// Mask systemd-resolved and drop a plain resolv.conf so dnsMode() in
+		// net/dns/manager_linux.go falls through to "direct". Point it at
+		// natlab's fake DNS VIP (not a public resolver): it's the only resolver
+		// reachable in vnet and it serves the internal *.tailscale names.
+		fmt.Fprintf(ud, "  - [\"/bin/sh\", \"-c\", \"systemctl disable --now systemd-resolved 2>/dev/null || true\"]\n")
+		fmt.Fprintf(ud, "  - [\"/bin/sh\", \"-c\", \"systemctl mask systemd-resolved 2>/dev/null || true\"]\n")
+		fmt.Fprintf(ud, "  - [\"/bin/sh\", \"-c\", \"rm -f /etc/resolv.conf && printf 'nameserver %s\\\\n' >/etc/resolv.conf\"]\n", vnet.FakeDNSIPv4())
+	}
+}
+
 // generateFreeBSDUserData creates FreeBSD nuageinit user-data (#cloud-config)
 // for a node. FreeBSD's nuageinit supports a subset of cloud-init directives
 // including runcmd, which runs after networking is up.