Files whose names contain characters with Unicode decompositions (such
as umlauts or voiced kana) could not be opened or written over
Taildrive.
Background: keyboards and IMEs emit NFC (precomposed) characters on
every platform, so filenames on Linux (ext4 etc) and Windows (NTFS)
disks are usually NFC bytes. NFD (decomposed) names mostly come from
Apple software: HFS+ forced a variant of NFD on write, and Apple's
frameworks still decompose paths via fileSystemRepresentation. APFS
preserves whatever bytes it is given but does normalization-insensitive
lookups (it stores a hash of the normalized name), so canonically
equivalent names find the same file. ext4 and NTFS lookups, by
contrast, are byte-exact.
On the wire, the macOS WebDAV client sends paths in NFD form (they
pass through the decomposing file system representation, and unlike
Apple's NFS client there is no "nfc" mount option). Windows and Linux
WebDAV clients pass names through as the application provided them,
typically NFC. WebDAV itself mandates no normalization, and PROPFIND
hrefs reflect the server's on-disk bytes.
The two forms are canonically equivalent but byte-wise different, so a
macOS client requesting the NFD form of an NFC-named file on a Linux
or Windows host got a 404 from the exact-byte lookup. Even against an
APFS host, where the filesystem absorbs the mismatch, the client-side
StatCache could still infer a 404: a cached directory listing in one
form caused depth 0 PROPFINDs in the other form to be treated as not
found without ever reaching the server. The inverse direction (NFD
bytes on a Linux disk, copied there from a Mac, requested in NFC form
by a Windows or Linux client) was broken too.
Alternative regimes considered: normalizing names at storage time (as
Nextcloud and Syncthing's autoNormalize do) would rename user files in
shared directories as a side effect of serving them; normalizing
request paths to a fixed form on the wire is unsound because the
on-disk form is unknowable a priori (ext4 can hold either form, or
both). Instead, adopt the APFS model: preserve bytes, but make lookups
normalization-insensitive.
Concretely, wrap the remote file server's webdav.Dir in a
normalizingFS that, when an exact path lookup fails, rescans the
parent directory for an entry whose name is canonically equivalent,
comparing the NFC form of both sides (which also sidesteps Apple's
nonstandard decomposition tables). Exact matches always win, and newly
created files keep the exact bytes the client sent. Also NFC-normalize
StatCache keys so canonically equivalent names share a cache entry.
The change is covered at three levels: unit tests for the StatCache,
an in-process two-node test in drive/driveimpl, and a new TestTaildrive
VM integration test in tstest/natlab/vmtest that shares a directory
between two Ubuntu VMs and exercises the NFC/NFD cases over the real
stack with curl playing the part of a macOS WebDAV client.
Fixes#15020
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9c2f157e604efc629828581e08d5b3191dbb7d4e
This file was never truly necessary and has never actually been used in
the history of Tailscale's open source releases.
A Brief History of AUTHORS files
---
The AUTHORS file was a pattern developed at Google, originally for
Chromium, then adopted by Go and a bunch of other projects. The problem
was that Chromium originally had a copyright line only recognizing
Google as the copyright holder. Because Google (and most open source
projects) do not require copyright assignemnt for contributions, each
contributor maintains their copyright. Some large corporate contributors
then tried to add their own name to the copyright line in the LICENSE
file or in file headers. This quickly becomes unwieldy, and puts a
tremendous burden on anyone building on top of Chromium, since the
license requires that they keep all copyright lines intact.
The compromise was to create an AUTHORS file that would list all of the
copyright holders. The LICENSE file and source file headers would then
include that list by reference, listing the copyright holder as "The
Chromium Authors".
This also become cumbersome to simply keep the file up to date with a
high rate of new contributors. Plus it's not always obvious who the
copyright holder is. Sometimes it is the individual making the
contribution, but many times it may be their employer. There is no way
for the proejct maintainer to know.
Eventually, Google changed their policy to no longer recommend trying to
keep the AUTHORS file up to date proactively, and instead to only add to
it when requested: https://opensource.google/docs/releasing/authors.
They are also clear that:
> Adding contributors to the AUTHORS file is entirely within the
> project's discretion and has no implications for copyright ownership.
It was primarily added to appease a small number of large contributors
that insisted that they be recognized as copyright holders (which was
entirely their right to do). But it's not truly necessary, and not even
the most accurate way of identifying contributors and/or copyright
holders.
In practice, we've never added anyone to our AUTHORS file. It only lists
Tailscale, so it's not really serving any purpose. It also causes
confusion because Tailscalars put the "Tailscale Inc & AUTHORS" header
in other open source repos which don't actually have an AUTHORS file, so
it's ambiguous what that means.
Instead, we just acknowledge that the contributors to Tailscale (whoever
they are) are copyright holders for their individual contributions. We
also have the benefit of using the DCO (developercertificate.org) which
provides some additional certification of their right to make the
contribution.
The source file changes were purely mechanical with:
git ls-files | xargs sed -i -e 's/\(Tailscale Inc &\) AUTHORS/\1 contributors/g'
Updates #cleanup
Change-Id: Ia101a4a3005adb9118051b3416f5a64a4a45987d
Signed-off-by: Will Norris <will@tailscale.com>
Fix the index out of bound panic when a request is made to the local
fileserver mux with a valid secret-token, but missing share name.
Example error:
http: panic serving 127.0.0.1:40974: runtime error: slice bounds out of range [2:1]
Additionally, we document the edge case behavior of utilities that
this fileserver mux depends on.
Signed-off-by: Craig Hesling <craig@hesling.com>
This prevents Mark-of-the-Web bypass attacks in case someone visits the
localhost WebDAV server directly.
Fixestailscale/corp#19592
Signed-off-by: Percy Wegmann <percy@tailscale.com>
This prevents Mark-of-the-Web bypass attacks in case someone visits the
localhost WebDAV server directly.
Fixestailscale/corp#19592
Signed-off-by: Percy Wegmann <percy@tailscale.com>
This prevents Mark-of-the-Web bypass attacks in case someone visits the
localhost WebDAV server directly.
Fixestailscale/corp#19592
Signed-off-by: Percy Wegmann <percy@tailscale.com>
This prevents Mark-of-the-Web bypass attacks in case someone visits the
localhost WebDAV server directly.
Fixestailscale/corp#19592
Signed-off-by: Percy Wegmann <percy@tailscale.com>
This prevents Mark-of-the-Web bypass attacks in case someone visits the
localhost WebDAV server directly.
Fixestailscale/corp#19592
Signed-off-by: Percy Wegmann <percy@tailscale.com>
This change updates all tailfs functions and the majority of the tailfs
variables to use the new drive naming.
Updates tailscale/corp#16827
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
This change updates the tailfs file and package names to their new
naming convention.
Updates #tailscale/corp#16827
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
Brad FitzpatrickandBrad Fitzpatrick
Will NorrisandWill Norris
Alex ChanandAlex Chan
Craig HeslingandGitHub
Percy WegmannandPercy Wegmann
Charlotte Brandhorst-SatzkornandGitHub