4

I have been looking for software which will checksum and verify/compare NTFS volumes with deep paths on Windows, and failing verification on every one I've tried. I've even tried writing my own python script and run into the same problems.

The problems start when files end up with paths which are more than 256 characters. This is bad enough on the primary drive, but once these files have been backed up and are even deeper on the backup filesystem it gets even worse. For example, my python program, ExactFile (suggested here), md5deep and HashMyFiles will all happily hash

D:\My Music\CDs\Bach, Johann Sebastian\The Art of Fuge _ Musical Offering (Sir Neville Marriner)\Bach, Johann Sebastian+The Art Of Fuge _ Musical Offering (Sir Neville Marriner, CD 1)+09+Contrapunctus 7, A 4, Per Augmentationem Et Diminutionem.mp3

since it is only 247 characters, but when it comes to verify it against the backup, none of these programs are able to open the file:

W:\WD SmartWare.swstor\XYZZY\Volume.1234567890abcdef1234567890abcdef0123\My Music\CDs\Bach, Johann Sebastian\The Art of Fuge _ Musical Offering (Sir Neville Marriner)\Bach, Johann Sebastian+The Art Of Fuge _ Musical Offering (Sir Neville Marriner, CD 1)+09+Contrapunctus 7, A 4, Per Augmentationem Et Diminutionem.mp3

as it's full path is now 317 characters.

Microsofts fciv even crashes when having to deal with such long file paths.

Overall, this makes it very difficult to verify that the backups have been successful.

I'm sure I can't be the only person that needs to calculate, verify and compare hashes of files in deep directory structures. It seems crazy that NTFS can support file paths up to 32k in length, yet so many utilities which aim to audit those filesystems seem unable to cope with such valid filepaths.

Can anyone verify that there are any tools which actually support hashing, verifying and comparing file hashes of files on Windows with long paths?

Community
  • 1
Mark Booth
  • 2,799
  • 1
  • 31
  • 47
  • I do not know any tools, but I do know two work-arounds: 1) use the good old fashioned `subst` command to shorten the paths. Ugly as hell, but should work. 2) You can bypass the windows API which limits you to 260 chars by adding a ["\?\" prefix](http://superuser.com/questions/471794/is-there-a-successor-to-ntfs). If you find tools which support this they should work fine up to 32k chars. – Hennes Jun 16 '13 at 22:08
  • Thanks @Hennes, `subst` really isn't a viable option, and I tried the "\\?\" fix in several of these pieces of software and at best it made no difference and at worst made them hash even fewer files. In my python program it just changes errors from `[WinError 3] The system cannot find the path specified` to `[WinError 206] The filename or extension is too long`. But that's really a [so] question. *8'( – Mark Booth Jun 17 '13 at 08:26
  • cygwin + find + md5sum + bash glueing should do the trick. If that doesn't work, turn to the real thing: linux + find + md5sum + bash glueing – wnrph Jul 05 '13 at 18:17
  • @MarkBooth: I have some old, klutzy C code I can give you which does MD5 checksums of deep paths. It probably isn't suitable as-is to your needs but could be adapted. Email me if interested. – Harry Johnston Jul 09 '13 at 03:35

2 Answers2

2

7zip 9.25 alpha can compute CRC-32, CRC-64, SHA-256 and SHA-1 and supports long file paths.

Martin
  • 243
  • 3
  • 10
  • I can't see any way to get 7-zip to calculate the checksums without creating an archive. Could you give an example of what you mean? – Mark Booth Sep 18 '13 at 23:47
  • 1
    @MarkBooth: I tried 9.30 alpha: `7z h *` or `7z h -mm=sha1 -r *`. This does not create any archive. I hope I did not misunderstand you, and sorry for my late reply. – Martin Sep 25 '13 at 12:29
  • At least 7z 16.04 uses `-scrc` option to set one of available hash function (`CRC32`, `CRC64`, `SHA1`, `SHA256`). E.g. `7z h -scrcSHA1 -r folder > report.txt`. – A-IV Apr 12 '17 at 10:46
0

Total Commander is capable of handling long file paths (and it even warns about them). It is shareware.

It can create MD5 and SHA1 checksums; it writes the sums into files, all sums into one file or for each file one sum file. I don't know if it can be used in batch files etc. to automatically generate checksums.

Moreover, it can compare directories and complete directory trees including each bit in each file, and I think this feature might be interesting for you.

Martin
  • 243
  • 3
  • 10