4

I've got a couple of disks which have a large set of files which are mostly the same. However, in a few cases there are files on one disk which differ from files on the other disk. There are also a lot of files where the files are identical on both drives, but the timestamps differ.

For my purposes I need to find just the files which are actually different. If I run:

rsync --dry-run -HPrlt

it find not only the files that are different, but also the files that differ only by timestamp, leaving me with extra work to determine if these are false positives or not.

I also thought to try:

rsync -c --dry-run -HPrlt

But this command takes a lot longer. In fact, the former command ran in a few seconds (presumably because the directory structure was already in cache from a previous rsync) whereas the latter command is still running. I suspect that this is because rsync is relying entirely on the checksum to determine what files need transferring, instead of something slightly more intelligent like only using a checksum if the timestamps differ.

How can I quickly see only the files that actually differ?

Note: This is not a duplicate of How to print files that would have been changed using rsync? because as the highest rated comment to the highest rated answer points out, --dry-run will show files that are identical if their timestamps differ.

Michael
  • 2,614
  • 6
  • 31
  • 51
  • In "But this command takes a lot faster." I think you meant "... is a lot slower". – bitinerant Feb 06 '21 at 18:25
  • @bitinerant thanks, I have fixed it – Michael Feb 06 '21 at 18:27
  • So, do I understand correctly that want `rsync --dry-run` and then, of the files that are listed, `rsync -c --dry-run`? In other words, compare everything by timestamp first, and then checksum files who's timestamps don't match? – bitinerant Feb 06 '21 at 18:31
  • @bitinerant i don't know... if i don't use "-c" it shows me files that differ only by timestamp. if I use "-c" it takes too long for the command to run. – Michael Feb 06 '21 at 18:31
  • Try `rsync --dry-run -HPrl` (i.e. without the `--times`, if you don't want files listed that only differ by timestamp). – bitinerant Feb 06 '21 at 18:33
  • @bitinerant I would have guessed that would work, but it's still returning files that `cmp` as identical but differ by timestamp. – Michael Feb 06 '21 at 20:32
  • Besides your wish to get the results fast: Is it possible that you also have files that are identical, but differ in filename or have been moved elsewhere? — Such "identical files" `rsync` will not find. – dodrg Jul 24 '23 at 12:01
  • I don't think you can do what you want. You either have the overhead of file comparison OR the speed of timestamp comparison. The problem is that many programs update the modified timestamp even if the file has not changed. – rfportilla Jul 24 '23 at 20:06

3 Answers3

3

rsync -HPrl --itemize-changes --dry-run source/ destination/ | grep -Fv "f..T......"| grep -Fv "d..T......"| cut -d " " -f 2

*Do Not Miss Trailing Slash For The Source Directory.

'--itemize-changes' outputs a change-summary for all updates. When combined with --dry-run and greping out files/dirs for which only timestamp update is required, gives the required output (quickly).

mittal
  • 131
  • 5
  • This doesn't work. It still shows ALL the files. – shivams Jul 28 '23 at 12:52
  • The command is now mainly comparing file sizes, to identify differing files. Could you check size of a file which you know is same but is wrongly shown in the output? If the sizes are different, try checking their checksum. – mittal Jul 29 '23 at 05:45
  • Check this: https://superuser.com/questions/1448465/exact-files-copied-have-a-different-size-on-disk-on-the-same-drive – mittal Jul 29 '23 at 05:45
  • They are the same size. Their checksum is same. Their timestamp is same. – shivams Jul 29 '23 at 21:22
  • You can reproduce the issue using this gist: https://gist.github.com/shivams/4371e8ee68d39d9b1b7f0c2ccccd725e – shivams Jul 29 '23 at 21:22
  • Add a trailing slash for the source directory, as indicated in the answer (added a warning too). Else it's trying to put dir1 inside dir2. It works. – mittal Jul 31 '23 at 05:38
  • Aah. Your solution actually works! It was a matter of trailing slashes. Too bad that the bounty expired though, and I saw it late. My 100 points were gone and I couldn't award them to you! – shivams Jul 31 '23 at 11:54
0

It's possible to run rsync in two stages :

  1. Generate a list of all files differing in size or timestamp (which might wrongly include some files that are identical)
  2. rsync using this list and with the checksum-compare option to find the real differences.

This answer is based on the post
Reuse rsync --dry-run output to speed up the actual transfer later on.

To use the file list generated during a dry run as an include file, requires removing the extra lines at the top and bottom of the dry-run output.

Example output:

sending incremental file list
[LIST OF FILES]

sent 226 bytes  received 34 bytes  520.00 bytes/sec
total size is 648,373,274  speedup is 2,493,743.36 (DRY RUN)

To remove the superfluous lines and leave only the files-list:

rsync --dry-run -avz source/ destination/ | head --lines=-3 | tail --lines=+3 > include.txt

For rsync to use this file (add additional options as desired):

rsync -c --include-from=include.txt --exclude=* source/ destination/

EDIT: I have reproduced the problem according to the poster's gist and can add to the description that whenever --dry-run is specified, then all the files are marked for sync, no matter which combination of parameters is used.

I think that the problem is actually with --dry-run, perhaps because it's checking too many metadata attributes. Seems perhaps like a bug.

harrymc
  • 455,459
  • 31
  • 526
  • 924
  • Hi @harrymc, thanks for your response. However, this does not fully answer the key bounty issue. The key bounty issue is this: "when running `--dry-run`, rsync seems to print ALL the files". Why is this happening? I later found this Unix.SE post which discusses the issue but the solutions provided there still don't work: https://unix.stackexchange.com/questions/341413/see-only-changes-when-running-rsync – shivams Jul 27 '23 at 11:33
  • @shivams: Could you add more information about source and target disks and how do the files differ in such a way that they are all different? – harrymc Jul 27 '23 at 11:53
  • I am on Ubuntu 22.04 using ext4 partition. Here is a gist to reproduce the issue: https://gist.github.com/shivams/4371e8ee68d39d9b1b7f0c2ccccd725e – shivams Jul 27 '23 at 12:24
  • Try to use `cp -p` in the gist. – harrymc Jul 27 '23 at 12:34
  • Yes, used that (and updated the gist). Still the same issue. Can you reproduce it at your end? – shivams Jul 27 '23 at 12:38
  • Yes, reproduced. I guess that `--dry-run` checks too many metadata items, so everything comes up as different. – harrymc Jul 27 '23 at 17:29
0

This could be an XY problem: you have an issue to solve, but are asking how to solve it with rsync.

The OP question asks about rsync, but its possible that a timestamp difference will always show as "different". I'm not sure theres a "-c but ignore timestamp" option. No matter what tool you decide to use, you must read the entire file to verify its content.

Here is a possible alternative (non-rsync) solution:

Hash the trees and find the differences. This will produce a list of files that differ. By "differ" I mean any of:

  • the content changed
  • the file exists on one side but not the other
cd /tree1
find -type f -print0 | sort -z | xargs -0 md5sum > /tmp/tree1.log

cd /tree2
find -type f -print0 | sort -z | xargs -0 md5sum > /tmp/tree2.log

diff -uw /tmp/tree1.log /tmp/tree2.log | grep '^[+-]' | awk '{print $2}' | sort -u

(For the md5sum naysayers: I know md5 is broken in the sense of of finding a second preimage attack...but the OP (probably) isn't looking for something cryptographically critical and md5 is faster than sha256.)

KJ7LNW
  • 351
  • 2
  • 10