types/geo: fix floating point bug causing NaN returns in SphericalAngleTo (#18777)

Subtle floating point imprecision can propagate and lead to
trigonometric functions receiving inputs outside their
domain, thus returning NaN. Clamp the input to the valid domain
to prevent this.

Also adds a fuzz test for SphericalAngleTo.

Updates tailscale/corp#37518

Signed-off-by: Amal Bansode <amal@tailscale.com>

types/geo/point.go

@@ -125,6 +125,9 @@ func (p Point) SphericalAngleTo(q Point) (Radians, error) {
 	sLat, sLng := float64(qLat.Radians()), float64(qLng.Radians())
 	cosA := math.Sin(rLat)*math.Sin(sLat) +
 		math.Cos(rLat)*math.Cos(sLat)*math.Cos(rLng-sLng)
+	// Subtle floating point imprecision can lead to cosA being outside
+	// the domain of arccosine [-1, 1]. Clamp the input to avoid NaN return.
+	cosA = min(max(-1.0, cosA), 1.0)
 	return Radians(math.Acos(cosA)), nil
 }