Is there a way to transfer thousand of small files faster on Windows 7?

Question

I noticed that if I transfer a few large files between two hard drives, it's pretty speedy, at around at least 30 MB per second, but if I transfer thousands of files less than 5 KB, it is pretty damn too slow.. around 1 to 2 MB per second.

Is there a way to speed up the copy/paste process with thousands of small files on Windows 7?

Chris · Answer 1 · 2020-12-13T07:07:29.170

25

You might want to take a look at TeraCopy which is a program designed to copy and move files at the maximum possible speed by dynamically adjusting buffers to reduce seek times. TeraCopy can also do asynchronous copying which speeds up file transfer between two physical hard drives.

I have personally used this application and have found that it does speed up file transfers which usually would take some period of time to accomplish.

Hope this helps some.

edited Dec 13 '20 at 07:07

answered Mar 28 '10 at 23:29

Chris

1,748
2
15
15

1

@netrox: highly, highly recommended Teracopy. +1 – studiohack Mar 29 '10 at 04:40
2

TeraCopy crashed for me so I wouldn't recommend it as there's potential for data loss. – Simon Morgan Jun 10 '15 at 16:37
3

I had the opposite problem - copying 1m tiny files caused explorer to crash, but TeraCopy worked well! – RobinL Jan 17 '16 at 10:02
1

TeraCopy is not freeware. I suggest [FastCopy](https://fastcopy.jp/en/) instead. – luca76 Aug 29 '20 at 08:33

score 22 · Answer 2 · edited Aug 11 '18 at 18:15

22

The solution is to archive with WinRAR, but, when asked how to archive, choose store. This means that there will be no compression. Thus in approximately one minute you will end up with one large file to copy, which will copy very fast.

I tried to copy 19890 small files (5K or so each) and Windows told me that it would take 3 hours, TeraCopy said 3.2 hours, but with my method it only took 1.5 minutes.

edited Aug 11 '18 at 18:15

robinCTS

4,327
4
20
29

answered Aug 11 '18 at 17:43

Ion Apostol

229
2
2

2

[7Zip](https://www.7-zip.org/) will let you set the compression type to "store" too. This is waaaayyyy faster than copying individual files. – Gabriel Luci Dec 17 '19 at 19:51
It works much faster than window copy. saved me hours – HungNM2 Nov 25 '21 at 09:41

score 8 · Answer 3 · answered Mar 28 '10 at 23:00

8

ZIP the files and then transfer the larger ZIP file? I don't know how long it would take to ZIP though (and if the total time is faster).

answered Mar 28 '10 at 23:00

Kevin Yap

1,017
10
14

9

You still have to read all files on the source side and write all of them at the target side. It won't be faster. You'll just lose time for zipping them up. – Joey Mar 28 '10 at 23:02
2

Actually, this works for me in my situation as I was really wanting to back up files... I don't care about unzipping. Zipping thousands of files sure worked... a zipped file was much faster than the usual copying. But yeah, I did try to unzip and it takes a while. Thanks for the suggestion. – netrox Mar 29 '10 at 03:10
1

No problem; glad I helped. :) – Kevin Yap Mar 29 '10 at 03:38
2

This works extremely well. I was able to get the transfer time of 200mb of files from 30 seconds using a plain copy down to 2 seconds (no compression, different drive). The reason it works is because there is a lot of overhead associated with creating and closing a new file. With an archive, it's has a single file handle (the archive) that you are appending data to. If you are moving to another hard drive, create the archive on the destination don't create it then move. – Despertar Dec 13 '16 at 04:09
1

Another usage case where this can help speed things up a lot is when transferring to/from portable flash drives or other slower media and interfaces. Zipping/Unzipping thousands of small files on an SSD with a SATA III connection is a lot faster than transferring thousands of files to/from a portable flash drive on USB. Cleanup is made a lot simpler too, as deleting a single zip file on the USB drive takes less than a second versus several minutes to delete a folder with thousands of small files in it. – positlux Jan 30 '18 at 00:24
@Joey: but given the archive file, the transfer time would be improved, right? I am talking exclusively about trasfer speed. – Albert Apr 10 '23 at 11:18

score 1 · Answer 4 · answered Mar 28 '10 at 23:06

It's possible that part of what is slowing you down for many small files is if they are not in the same physical area of the disk. On a drive which is not very fragmented, a single large file will mostly all be read from one place, but if you have to read a bunch of separate files, they may be scattered across the disk.

ZIPing was the first idea that came to mind for me as well, but as pointed out above, you'd lose time to that process anyhow. I have noticed that in general, copying with RoboCopy.exe goes faster than doing it through the GUI. You might want to play around with that and see how it works for you.

You'll lose most time due to the frequent switching between file content and the MFT entries. Where the individual files are located on the disk is nigh irrelevant here. — Joey, Mar 28 '10 at 23:20

score 1 · Answer 5 · edited Sep 24 '12 at 13:21

1

ZIP has a 4 GB file limit (or something like that) - I usually use RAR archiver (it doesn't have that file size limit) and specify to not compress at all - this way archiving into a single file happens very fast, and then I simply copy that big file.

edited Sep 24 '12 at 13:21

Peter Mortensen

12,090
23
70
90

answered Mar 29 '10 at 00:50

Andrey

187
2
9

score 0 · Answer 6 · answered Mar 28 '10 at 23:12

It might be worth trying a quick defrag before ZIPing anything, but this is only really if you will be moving lots of small files, very often. If not then I suggest just zipping it (with 7zip or something - which will often compress better than just Windows standard compression) and then copy across.

And it also depends on the harddrives. Is this a USB external hard drive or 2 that are in the same system? If it's an old external hard drive it may be using USB1.0 or you may be better off having one with a power supply.

score 0 · Answer 7 · answered Jul 14 '20 at 00:44

0

Seems to be lots of indeterminate answers here, so I'll add a datapoint.

For me it was significantly faster to zip (using store-only compression) and then transfer the zip. Significantly.

answered Jul 14 '20 at 00:44

user37309

111
1

Vilém Duha · Answer 8 · 2023-07-03T16:02:25.097

0

On Windows 11, just was trying to solve same issue (!). Total commander helped me a lot, since it's much faster.

https://www.ghisler.com/

edited Jul 03 '23 at 16:02

answered Jul 03 '23 at 16:01

Vilém Duha

101
1

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 03 '23 at 16:02

score 0 · Answer 9 · edited Jul 07 '15 at 11:10

0

If its a unix based system you can use tar over ssh.
-The following command compresses all files and folders.

tar -cf - /home | ssh root@192.168.1.1 tar -xvf - -C /

-Transfer it and decompress it at the second computer.
This is much faster than copying only per scp

First read the man page and BE CAREFUL

edited Jul 07 '15 at 11:10

mic84

2,363
2
21
17

answered Jul 07 '15 at 11:00

user466567

11

2

The question contains the "windows" tag so your answer doesn't apply. – Nasreddine Jul 07 '15 at 11:08
2

indeed. next time, read the tags, eh? – td512 Jul 07 '15 at 11:15
Not to mention, read the age of the question – qasdfdsaq Jul 07 '15 at 11:30

Is there a way to transfer thousand of small files faster on Windows 7?

9 Answers9

Linked

Related