21

I was going to make an image of my old USB drive. I had good reasons to expect there were some blocks filled with zeros on the device, so to save some space I used conv=sparse option:

dd if=/dev/sdb of=myusb.img conv=sparse bs=32M

However it saved me nothing:

$ ls -hls myusb.img
250M -rw-r--r-- 1 root root 250M Oct 18 21:31 myusb.img

I'm sure there are zero-filled blocks on the device. Why didn't dd conv=sparse save space?

Note I already know the answer (I think). I'm posting it below. The question is for future reference.

Kamil Maciorowski
  • 69,815
  • 22
  • 136
  • 202

1 Answers1

36

If you are absolutely sure there were zero-filled blocks then the reason you saved no space was the large buffer you used. From man dd:

sparse try to seek rather than write the output for NUL input blocks

You used bs=32M, so you needed a whole 32 MiB block of zeros at a right offset for the conv=sparse option to do its job if only just once.

The option bs sets ibs (input block size) and obs (output block size). While the manual mentions input blocks, it is actually the obs that matters.

Here are the results of some tests. (As I am the OP, I did the tests with the very same device.) Each file is named according to <obs_used>.img pattern. Pay attention to the first column:

$ ls -hlst *.img
250M -rw-r--r-- 1 root root 250M Oct 18 22:02 4M.img
250M -rw-r--r-- 1 root root 250M Oct 18 22:02 2M.img
249M -rw-r--r-- 1 root root 250M Oct 18 22:02 1M.img
248M -rw-r--r-- 1 root root 250M Oct 18 22:01 512K.img
248M -rw-r--r-- 1 root root 250M Oct 18 22:01 256K.img
247M -rw-r--r-- 1 root root 250M Oct 18 22:00 128K.img
247M -rw-r--r-- 1 root root 250M Oct 18 21:57 64K.img
247M -rw-r--r-- 1 root root 250M Oct 18 21:56 32K.img
246M -rw-r--r-- 1 root root 250M Oct 18 21:55 16K.img
246M -rw-r--r-- 1 root root 250M Oct 18 21:54 8K.img
246M -rw-r--r-- 1 root root 250M Oct 18 21:53 4K.img
246M -rw-r--r-- 1 root root 250M Oct 18 21:52 2K.img
246M -rw-r--r-- 1 root root 250M Oct 18 21:51 1K.img
246M -rw-r--r-- 1 root root 250M Oct 18 21:44 512.img

The conclusion is: you shouldn't use large obs with conv=sparse option. The common sector size is 512 bytes, so bs=512 seems just right. Your command should have been:

dd if=/dev/sdb of=myusb.img conv=sparse bs=512
Kamil Maciorowski
  • 69,815
  • 22
  • 136
  • 202
  • It's good to see the `sparse` option actually working, but the saving is so small that it doesn't seem worth the trouble. Maybe you should fill the free space on the old device with an all-0x00 file and then delete it, before actually dumping the image with dd? – Chang Qian Oct 23 '22 at 02:05
  • @ChangQian [I know the trick](https://superuser.com/a/1097391/432690). – Kamil Maciorowski Oct 23 '22 at 02:28
  • You wrote in the question that you had *"good reasons to expect there were some blocks filled with zeros"*, so at the end how can you explain the very small amount of saved space even with `bs=512`? What were your good reasons? Given the limited total size and the name you gave I can imagine that the source is a USB key? If yes, unless this is a brand new key or unless you have explicitly written a file full of zeros on it, I wouldn't expect many sectors containing only zeros. – PierU Oct 24 '22 at 08:18
  • @PierU Two or more blocks are "some blocks". After 6 years I don't remember the reasons. Why should I ever explain the very small amount of saved space? IMO the fact it's "small" does not change the conclusion that smaller `bs` seems better when `conv=sparse` is used. – Kamil Maciorowski Oct 24 '22 at 08:28
  • Your question/answer has been linked from another question to support the idea that `conv=sparse` would not save a significant amount of written blocks, without taking into account that the context in not the same (this is a SSD involved, so there are reasons to expect a different behavior compared to a USB key that has likely not even a garbage collection mechanism). You're not responsible for such wild extrapolation, but nonetheless it would be useful IMHO to update your answer and comment about the amount of saved space. – PierU Oct 24 '22 at 08:46