Yes, in NTFS. From here:
As an aside if you are running under NTFS then file operations are atomic at the file system level. A rename will occur in a single operation as far as any higher code is concerned. The problem you are seeing almost appears to be an issue where the FileInfo object is being shared across applications. It is a MarshalByRef object and therefore can be used in remoting environments. Don't know if this applies to you.