What's the fastest way to combine two or more files in Linux?

Question

I want to combine two or more files in Linux, so I am using the following command:

cat small_file LARGE_File LARGER_FILE > SUM_OF_FILES

However this runs very slow.

Does anyone know a Linux tool that combines the files in the fastest time?

Get yourself faster hard drives. The bottleneck is not in the command, but in the speed of reading the data. — Dark Falcon, Nov 22 '11 at 17:14
How often do you need to do this? If frequently then consider a different system of running things. — Ed Heal, Nov 22 '11 at 17:20
@macki: "periodically" could mean 10 times per second or once every 10 years. — Keith Thompson, Nov 22 '11 at 18:51
If you're doing this periodically, you aren't by chance including the `SUM_OF_FILES` from a previous run as an input to the next one, are you? — phs, Nov 22 '11 at 19:38

score 7 · Accepted Answer · edited Nov 23 '11 at 04:13

7

You could try a variation on the dd command, such as:

dd if=small_file bs=4k of=SUM_OF_FILES

dd if=LARGE_FILE bs=4k of=SUM_OF_FILES oflag=append

dd if=LARGER_FILE bs=4k of=SUM_OF_FILES oflag=append

edited Nov 23 '11 at 04:13

Gaff

18,569
15
57
68

answered Nov 22 '11 at 18:45

this is what I'm trying right now, I'm having problem with the exact size – Nov 22 '11 at 22:10

score 1 · Answer 2 · answered Nov 22 '11 at 17:34

1

I've found mmv (Mass Move and rename - Move, copy, append or link Multiple files using wildcard patterns.) from this useful bash reference. So you could do:

cp small_file SUM_OF_FILES
mmv -a LARGE_File SUM_OF_FILES
mmv -a LARGER_FILE SUM_OF_FILES

(note: mmv isn't installed by default, use sudo apt-get install mmv)

answered Nov 22 '11 at 17:34

CapelliC

119
4

`brew install mmv` to install mmv on max osx. – suzanshakya Dec 21 '14 at 06:01

score 0 · Answer 3 · answered Nov 22 '11 at 17:14

0

Maybe

cat small_file >> LARGE_File

will do what you want? If you need LARGE_FILE to be unchanged

cp LARGE_File SUM_OF_FILES
cat small_file >> SUM_OF_FILES

is better, but this will only be slightly faster than your original code.

answered Nov 22 '11 at 17:14

Gunther Piez

123
4

4

this is very slow when merging LARGE FILES – macki Nov 29 '11 at 18:42

What's the fastest way to combine two or more files in Linux?

3 Answers3