65

I'm a bit puzzled by this? Why is rsync faster than scp? Doesn't rsync use scp beneath or does it do something more efficient? Is there some way to speed up scp?

studiohack
  • 13,468
  • 19
  • 88
  • 118
grm
  • 2,504
  • 3
  • 28
  • 29

6 Answers6

88

Rsync will obviously be faster than scp if the target already contains some of the source files, since rsync only copies the differences. But I suspect your question was about doing a straightforward copy to an empty target.

You've passed the -z option to rsync; this turns on compression. If the network bandwidth is the limiting factor (it often is), compression can improve the transfer speed by a noticeable amount.

You can also enable compression with scp by passing the -C option. This should about even things out with rsync. Compression is not enabled by default in ssh because it saves bandwidth but adds latency and CPU overhead; latency is bad for interactive sessions (this doesn't apply to scp), and the CPU overhead is useless if the files you're copying are already compressed.

Older versions of rsync used rsh rather than ssh as the default transport layer, so a fair comparison would be between rsync and rcp. But ssh has been the default since 2.6.0 released on 2004-01-01.

With identical compression settings, I'd expect rsync and scp to have essentially the same speed. Please share benchmarks if you find otherwise.

Community
  • 1
Gilles 'SO- stop being evil'
  • 69,786
  • 21
  • 137
  • 178
  • 6
    Nice, using -C made my transfer speed increasing from 5MB to 20MB – ıɾuǝʞ Apr 03 '14 at 08:00
  • 1
    I just tested that for a directory containing 16 files around 3-9 MB and `rsync -z` is still WAY faster than `scp` with any compression suggested in these answers enabled. Its also faster than compressing and archiving into one file manually and `scp`'ing that file (`scp` with compression is even slower than that). So the question of the OP actually remains unanswered: wth is `scp` so slow compared to `rsync`? – ohcibi Feb 20 '17 at 19:13
  • I agree, the -C flag does not explain OP's question. I tested this on a directory of 2k json files where time scp provided (total: 6m, user: 0.028s, system: 2.453s). Adding the -C flag did not appreciably change things (total: 6m, user: 0.5s, system: 3s) perhaps because each json is small. However, rsync was significantly faster running in (total: 5.6s, user: 0.26s, system: 3.16s) – curiousgeorge May 31 '20 at 00:18
19

try scp in a fast way

scp -p -C -o 'CompressionLevel 9' -o 'IPQoS throughput'  -c aes128-cbc machine:file .

these options speed up scp 5 times in my setup compared to plain scp machine:file .

Update, 2017

Actually scp is slow due to poor management of TCP details such as MTU and buffer size. Luckily this has been fixed by the HPN SSH project. To my understanding you can use HPN SSH as a transport for rsync.

Peter K
  • 359
  • 3
  • 5
  • 2
    Awesome, thanks!! However always always do `scp -p` (preserve date/time) as default, and probably `-r` (recursive) so `scp -pr -C ...`. (I just had to scrub and restart a 40Gb scp job using these because I forgot the `-p`) – smci Oct 24 '17 at 21:26
  • Support for Arcfour was disabled by default in OpenSSH 7.2 (Feb 2016), and was subsequently removed completely in OpenSSH 7.6 (Oct 2017). macOS 10.14 ships with OpenSSH 7.9, which is several major versions beyond this removal, so say goodbye to `-c arcfour` – Samuel Prevost Mar 05 '21 at 21:30
  • 1
    Thanks, @SamuelPrevost. I'll check which of the current cyphers is the fastest. – Peter K Apr 09 '21 at 14:31
  • 1
    This was helpful for picking a new cypher https://bash-prompt.net/guides/bash-ssh-ciphers/ – jmcgrath207 Apr 14 '22 at 03:49
  • @jmcgrath207, thanks, I updated the cipher in the answer – Peter K Sep 02 '22 at 10:17
5

It used to be the other way around, but I believe rsync's speed has improved greatly the past few revisions. It also depends on how many files you're copying. If it's a lot, rsync will usually be faster because scp spawns a new process for each file you're copying. You can try weakening the cipher scp uses to see if it speeds up. Last I recall, the arcfour cipher was the fastest.

churnd
  • 4,898
  • 26
  • 24
  • 2
    Scp doesn't actually spawn a new process for each file being copied. You have one process on the sending side and one on the receiving side, just as with rsync. – Kenster Jun 21 '14 at 22:29
4

For my testing, rsync is faster than scp, you can use iotop to test them on transfering the same file:

sudo iotop -o

Maybe you'll get different result, but you can test them youself. BTW, while using scp, do not foget to choose its cipher by:

scp -c arcfour <source> <dest>

While arcfour can speed up the encryption.

coanor
  • 141
  • 3
3

For a large number of small files rysnc is much faster than scp. They say it's because it has smaller overhead. For one large file, I would expect similar results.

sencer
  • 491
  • 5
  • 18
0

Are you re-copying files over existing ones? If so, rsync's ability to block compare and only copy the differences will be relevant.

Linker3000
  • 27,498
  • 3
  • 52
  • 73