8

I was copying a few very very large files to my computer. In total, 28 files and roughly 200GB (meaning each one was . I noticed that it was going at more or less the same speeds the entire time:

enter image description here

However, transferring multiple small files has much larger fluctuations in speeds:

enter image description here

(Yes, this is from google drive but they're all locally downloaded to my computer)

Jon
  • 9,181
  • 39
  • 95
  • 127
  • It's got to do with the overhead of file transfer. Small files may require random disk access and the transfer speed also depends on the disk fragmentation. [More](https://superuser.com/questions/237774/why-is-my-file-copying-slowly) [info](http://serverfault.com/a/9743/218766) [here](http://www.makeuseof.com/answers/transfer-times-small-files-slower-big-file/) – Vinayak Sep 07 '14 at 02:17
  • @Vinayak Why would you need a random disk access/what would it be used for? – Jon Sep 07 '14 at 02:29
  • Files aren't always stored sequentially. [This video](https://www.youtube.com/watch?v=zGVXp-BkseA) explains it quite well. – Vinayak Sep 07 '14 at 02:36
  • I understand how defrag work, but why would small files require random disk accesses? – Jon Sep 07 '14 at 02:44
  • 1
    Because to reduce disk fragmentation, the files will will be written to wherever free space is still available within the HDD. For instance, there might be a 10 MB free space somewhere between two large files. The 10 megabytes of space isn't enough to store a large 5 GB movie, but it can accommodate hundreds of small, 1-100 KB files (text files, INI configuration files, documents, GIFs, etc.). When that block is filled up, the HDD will look for other small blocks of free space that it can use to fill with data. – Vinayak Sep 07 '14 at 02:49
  • [A better example here](http://www.tech-faq.com/how-data-is-stored-in-your-hard-disk.html) – Vinayak Sep 07 '14 at 03:23

1 Answers1

0

To reduce disk fragmentation, the files will will be written to wherever free space is still available on the hard drive.

For instance, there might be a 10 MB contiguous free space somewhere between two large, sequentially stored files.

The 10 megabytes of space isn't enough to store a large 5 gigabyte movie, but it can accommodate hundreds of small, 1-100 KB files (text files, INI configuration files, documents, GIFs, etc.)

When that block is filled up, the HDD will look for other small blocks of free space that it can use to fill with data, which causes the head to move around a lot.

That, along with the overhead mentioned in these linked articles causes small files to take a long time to copy.

Community
  • 1
Vinayak
  • 10,625
  • 10
  • 54
  • 89
  • 2
    Your answer would seem to imply that writing small files should be faster than large files, since you think that the small files would not be fragmented. The OP's charts do not support your explanation. You have misidentified the reason for R/W head movement during file copying. – sawdust Sep 07 '14 at 09:31
  • 1
    [File system fragmentation increases disk head movement or seeks, which are known to hinder throughput](https://en.wikipedia.org/wiki/File_system_fragmentation). Also, I never said small files aren't fragmented. And I have no idea what information you could've gleaned from the 'charts'. – Vinayak Sep 07 '14 at 19:37
  • 1
    Sure, file fragmentation reduces throughput. But fragmentation is not the reason why copying many small files takes longer than a large file. *"I never said small files aren't fragmented"* -- Then what's the point of your 3rd sentence/paragraph? If a contiguous space *"can accommodate hundreds of small, 1-100 KB files"*, would these small files still be stored in fragments? You seem to make the case that large files are more likely to be fragmented, and therefore require more seeks and have lower data throughput. So tell us, what information did you glean from those charts? – sawdust Sep 08 '14 at 01:07
  • 1
    I think you meant, "fragmentation is not the **only** reason why [copying small files takes longer]". Fragmentation definitely [affects write speeds](http://goo.gl/ZrHPPF) and what do you think happens when files (large and small) are being copied? You don't seem to understand [how files get fragmented](http://goo.gl/mGq5fY) and how de-fragmentation works. And yes, [large files on average are more likely to be fragmented](http://goo.gl/DiwRlA) since contiguous free clusters might not always be available. You also don't seem to have read the articles I linked in my answer. – Vinayak Sep 08 '14 at 06:02
  • 1
    If [this is why you're here](https://sslimgs.xkcd.com/comics/duty_calls.png), please stop now. If not, add a relevant answer to the question. – Vinayak Sep 08 '14 at 06:05
  • 1
    No, I already wrote what I meant; fragmentation does not explain the *difference* between copying a lot of small files versus a large file, especially since the large file is more likely to be fragmented. You persist in not expanding on your post. Instead you guess at what I don't know or understand or read. I've already provided a link to my answer in the 8th comment to the OP. – sawdust Sep 08 '14 at 06:35