Getting back to Philipp's original bug report, Apple documentation says macOS
has a facility like the Linux renameat2 system call (i.e., it's like 'renameat'
except it can be told to fail if the destination already exists). Attached is a
proposed patch to use this facility, which means that the case-insensitivity
test would no longer need to be done in macOS. If there's some way to implement
renameat_noreplace on MS-Windows we could get rid of the case-insensitivity test
there too.
I don't have easy access to macOS so I have not installed this patch. It'd be
nice, Philipp, if you could try it out.
Thanks, the patch fixes the problem. However, it's still not 100% correct: now casing changes such as from "A" to "a" where macOS treats the file names as equivalent trigger the "file already exists" signal as well. I don't think that can be fixed, though; there's no way to special-case casing changes while keeping atomicity intact. So I'd rather have Emacs react conservatively and skip the casing check entirely.
Note that the manpage says:
RENAME_EXCL On file systems that support it (see getattrlist(2) VOL_CAP_INT_RENAME_EXCL), it will cause EEXIST to be returned if the destination already exists.
I interpret this such that if the filesystem doesn't support RENAME_EXCL the rename will succeed even if the destination exists.
Since we probably won't be able to solve all issues across operating systems and filesystems, probably we should have at least a warning in the documentation that rename-file attempts to be race-free and atomic, but only on a best-effort basis.