0

I have a directory with 10GB (1M files) in it. rclone sync is taking multiple days to upload it to OneDrive.

I have a dedicated 11Mbps uplink. With my uplink speed this should take 2 hours. As proof, I was able to sync directories with 100GB (but with fewer files) in less time than it is taking to sync this little 10GB directory. So I conclude that the 1M files is adding days of time. I don't even see how this is possible. My ping time is 30ms. Even if software is dumb enough to handshake every file individually, and only do one handshake at a time, and each handshake is 3 roundtrips, it shouldn't be this slow. What could be going on and how might I fix it?

personal_cloud
  • 141
  • 5
  • A lot of small files can take longer to copy than larger files overall but a lot less of those files accumulated. This is why people compress lots of small files to an archive or zip file, and then send that file instead. The CPU to compress and decompress will take a little time on both sides, but it will speed up the transmission considerably. Give that a trial run to test. – Vomit IT - Chunky Mess Style Jul 03 '22 at 02:25

2 Answers2

1

I have a server with 1gbps link, other tools have no problem reaching line capacity, but rclone defaults were very low, less than 10% use!

Setting this variable to 128M completely solved it!

(I'm uploading multi gb files with rclone)

--drive-chunk-size 128M

Defaults value

--drive-chunk-size SizeSuffix Upload chunk size (default 8Mi)

Pedro
  • 111
  • 3
0

My advice is to pack the files all in one folder and compress them, which reduces the size of the files, and it's faster for OneDrive to upload files in the same folder (merge multiple small files into one large file if you have the conditions).

Ceasar
  • 389
  • 1
  • 7