wgengine/magicsock: disable SIO_UDP_NETRESET on Windows

By default, Windows sets the SIO_UDP_CONNRESET and SIO_UDP_NETRESET
options on created UDP sockets. These behaviours make the UDP socket
ICMP-aware; when the system gets an ICMP message (e.g. an "ICMP Port
Unreachable" message, in the case of SIO_UDP_CONNRESET), it will cause
the underlying UDP socket to throw an error. Confusingly, this can occur
even on reads, if the same UDP socket is used to write a packet that
triggers this response.

The Go runtime disabled the SIO_UDP_CONNRESET behavior in 3114bd6, but
did not change SIO_UDP_NETRESET–probably because that socket option
isn't documented particularly well.

Various other networking code seem to disable this behaviour, such as
the Godot game engine (godotengine/godot#22332) and the Eclipse TCF
agent (link below). Others appear to work around this by ignoring the
error returned (anacrolix/dht#16, among others).

For now, until it's clear whether this ends up in the upstream Go
implementation or not, let's also disable the SIO_UDP_NETRESET in a
similar manner to SIO_UDP_CONNRESET.

Eclipse TCF agent: https://gitlab.eclipse.org/eclipse/tcf/tcf.agent/-/blob/master/agent/tcf/framework/mdep.c

Updates #10976
Updates golang/go#68614

Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I70a2f19855f8dec1bfb82e63f6d14fc4a22ed5c3

wgengine/magicsock/magicsock.go

@@ -2538,6 +2538,7 @@ func (c *Conn) bindSocket(ruc *RebindingUDPConn, network string, curPortFate cur
 			}
 		}
 		trySetSocketBuffer(pconn, c.logf)
+		trySetUDPSocketOptions(pconn, c.logf)
 
 		// Success.
 		if debugBindSocket() {

wgengine/magicsock/magicsock_notwindows.go

@@ -0,0 +1,13 @@
+// Copyright (c) Tailscale Inc & AUTHORS
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build !windows
+
+package magicsock
+
+import (
+	"tailscale.com/types/logger"
+	"tailscale.com/types/nettype"
+)
+
+func trySetUDPSocketOptions(pconn nettype.PacketConn, logf logger.Logf) {}

wgengine/magicsock/magicsock_windows.go

@@ -0,0 +1,58 @@
+// Copyright (c) Tailscale Inc & AUTHORS
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build windows
+
+package magicsock
+
+import (
+	"net"
+	"unsafe"
+
+	"golang.org/x/sys/windows"
+	"tailscale.com/types/logger"
+	"tailscale.com/types/nettype"
+)
+
+func trySetUDPSocketOptions(pconn nettype.PacketConn, logf logger.Logf) {
+	c, ok := pconn.(*net.UDPConn)
+	if !ok {
+		// not a UDP connection; nothing to do
+		return
+	}
+
+	sysConn, err := c.SyscallConn()
+	if err != nil {
+		logf("trySetUDPSocketOptions: getting SyscallConn failed: %v", err)
+		return
+	}
+
+	// Similar to https://github.com/golang/go/issues/5834 (which involved
+	// WSAECONNRESET), Windows can return a WSAENETRESET error, even on UDP
+	// reads. Disable this.
+	const SIO_UDP_NETRESET = windows.IOC_IN | windows.IOC_VENDOR | 15
+
+	var ioctlErr error
+	err = sysConn.Control(func(fd uintptr) {
+		ret := uint32(0)
+		flag := uint32(0)
+		size := uint32(unsafe.Sizeof(flag))
+		ioctlErr = windows.WSAIoctl(
+			windows.Handle(fd),
+			SIO_UDP_NETRESET,               // iocc
+			(*byte)(unsafe.Pointer(&flag)), // inbuf
+			size,                           // cbif
+			nil,                            // outbuf
+			0,                              // cbob
+			&ret,                           // cbbr
+			nil,                            // overlapped
+			0,                              // completionRoutine
+		)
+	})
+	if ioctlErr != nil {
+		logf("trySetUDPSocketOptions: could not set SIO_UDP_NETRESET: %v", ioctlErr)
+	}
+	if err != nil {
+		logf("trySetUDPSocketOptions: SyscallConn.Control failed: %v", err)
+	}
+}