How to compress extremely large file in place

Question

I need to compress a 646.4GB file on a 1TB external HDD. This HDD currently has about 238.6GB free. However, since the straightforward compression methods create a new file, I'll not be able to compress this file, which is a .vhdx file (a Windows system image). I have access to both Windows (10 Tech Preview) and Unix systems (both Mac and Ubuntu, if that helps).

I either need some way to compress these files in place or some way to split this file into several smaller files without leaving the huge 646.4GB as is (ie, some way that deletes data from this file as it is done using it). Thanks!

I *assume* that you can do this with dd to get part of the file (e.g. the last 100GB, then `trunc`ate the original file to 646.4-100GB and compress the 100GB part. Rinse and repeat. Note that this will leave you with 7 compressed files all of which need the reverse actions to restore the vhdx to its original state (as in uncompress first, concattenate, next). However I really would try to avoid this. If at all possible use a second disk or use the network. Possibly in combination with netcat. — Hennes, Jul 01 '15 at 04:49
I suggest You do not use any software that modifies the original file while compressing/splitting it. If anything goes wrong, the vhdx file might get corrupted. Also if You are experimenting with new programs, create a backup if possible. — Divin3, Jul 01 '15 at 07:34
Indeed. Step 1: make a backup. Step 2: Oh, I already have a copy and all this is moot. Still, proof of concept posted as an intelectual exercise. :) — Hennes, Jul 01 '15 at 18:25
If the disk is formatted in NTFS you can turn on compression on the drive, which will perform an in place comresssion. Right click on the drive from Windows File Explorer, mark the "Compress this drive to save space" box. — Peter, Jul 01 '15 at 18:33
you might need to compact the vhdx file by zeroing out all clusters and run something similar to [`VBoxManage modifymedium --compact`](https://superuser.com/q/529149/241386) — phuclv, Sep 22 '18 at 12:25

score 3 · Answer 1 · answered Jul 02 '15 at 16:30

3

In windows, right click on the file and select Properties, then General Tab, then Advanced Button, tick compress contents to save disk space. This works for individual files or folders.

answered Jul 02 '15 at 16:30

Moab

58,044
21
113
176

score 2 · Accepted Answer · answered Jul 01 '15 at 18:24

Yes, it can be done.

Proof of concept:

Let me get a file to test with:

cp /bin/sh mylargefile

ls -l
-rw-r--r--  1 hennes  users  137208 Jul  1 20:05 my_large_file

Lets chop it into 3 pieces, first piece will be 50k, second will be 50k, third will be 37k. We start at the end with the third part.

 dd if=my_large_file of=part3 bs=1k skip=100
 33+1 records in
 33+1 records out
 34808 bytes transferred in 0.000232 secs (150046592 bytes/sec)

 ~/test$ ls -l
total 180
-rw-r--r--  1 hennes  users  137208 Jul  1 20:05 my_large_file
-rw-r--r--  1 hennes  users   34808 Jul  1 20:09 part3

Ok, so we can copy part of the file. Now lets truncate the original large file to 100000 bytes

truncate  -s 100000 my_large_file
[hennes@dragon] ~/test$ ls -l
total 144
-rw-r--r--  1 hennes  users  100000 Jul  1 20:17 my_large_file
-rw-r--r--  1 hennes  users   34808 Jul  1 20:09 part3

Compress with your favourite program. E.g.

bzip2 -9 part3
[hennes@dragon] ~/test$ ls -l part3.bz2
-rw-r--r--  1 hennes  users  11773 Jul  1 20:09 part3.bz2

Rinse and repeat:

dd if=my_large_file of=part2 bs=1k skip=50
47+1 records in
47+1 records out
48800 bytes transferred in 0.024526 secs (1989735 bytes/sec)

New dd with different numbers. New truccate with different numbers. ...

If you do this without uttermost care you are likely to mess up!

Above examples are already a mix of KB and KiB's. Take extreem care with your numbers.

Also, take a backup. Which means you already have twice the space so this should not be nessecary. Use only if you need to demo something (e.g. as homework or as a proof of concept during a job interview) and at your own risk.

illiterate · Answer 3 · 2018-09-28T13:57:28.043

BIG WARN:I CAN'T GUARANTEE THIS SCRIPT CODE SAFE!!

On my Debian Stretch I encounter same problem.
I don't know any released tool do this task, so I make a simple shell script for me($1 is the large file):
For compress:

#!/bin/sh -e
#in-place compress single large file
compressor="lz4"
! test -d ./small-files && mkdir ./small-files
while true; do
size="$(stat -c%s ${1})"
block="$((1024*1024*1024))"
if [ "${size}" -gt "${block}" ] ; then
tail --bytes "${block}" "$1" | ${compressor} > "./small-files/$((${size}-${block}))"
sync
truncate -s "$((${size}-${block}))" "${1}"
sync
elif [ "${size}" -gt "0" ] ; then
tail --bytes "${block}" "$1" | ${compressor} > "./small-files/0"
sync
truncate -s "0" "${1}"
sync
else
break
fi
done
echo "success"

For decompress:

#!/bin/sh -e
#in-place decompress single large file
decompressor="lz4cat"
for size in $(ls -1 ./small-files | sort -n) ; do
truncate -s "${size}" "./${1}"
sync
${decompressor} "./small-files/${size}" >> "./${1}"
sync
rm "./small-files/${size}"
sync
done
echo "success!"

For checksum:

#!/bin/sh -e
#in-place check compressed single large file
origincat () {
decompressor="lz4cat"
for size in $(ls -1 ./small-files | sort -n) ; do
${decompressor} "./small-files/${size}"
done
}
origincat | md5sum

score 0 · Answer 4 · answered Sep 22 '18 at 12:41

If the VHD file has windows on it, which I assume it does because you say "Windows System Image".

You might be better off mounting the VHD. Then use a tool like microsoft's imagex to compress the drive into an image file.

imagex /capture /flags "professional" c: n:\images\windows.wim "Win"

How to compress extremely large file in place

4 Answers4

Yes, it can be done.

If you do this without uttermost care you are likely to mess up!