10

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything.

I tried

parallel 'gunzip {}' ::: `ls *.gz`
parallel gunzip `ls *.gz`

with no results, bash tells me:

/bin/bash: archive1.gz: command not found
...

What am I doing wrong?

Thanks

gc5
  • 330
  • 1
  • 2
  • 14

2 Answers2

12

I found this, which suggests using the --gnu flag:

parallel --gnu gunzip  ::: *gz

If this works, you should either delete /etc/parallel/config or change its contents to --gnu rather than --tollef (as root):

echo "--gnu" > /etc/parallel/config

Also, never parse the output of ls., use globbing as I have above or find instead:

find . -name "*gz*" -print0 | parallel -q0 gunzip
terdon
  • 52,568
  • 14
  • 124
  • 170
  • It totally works.. thanks for tips and corrections. – gc5 Jun 08 '13 at 15:46
  • 1
    Parsing the output of `ls` is less of an issue when piping into GNU Parallel. As long as you do not have malicious users making filenames containing \n you are safe. In this case, however, `::: *.gz` would probably be the best. – Ole Tange Jun 10 '13 at 20:00
  • Putting \n in filenames is pretty fun indeed, thanks for the tip. However for small admin tasks it shouldn't be an issue so it's not worth taking a habit of using some other tool IMHO. – Aki Jan 15 '14 at 15:53
  • @Aki In fact it is VERY much worth making a habit of using the correct tool for the job. `ls` output changes between *nix flavors, and locale settings, it tends to break on any kind of white space, not just newlines, you willavoid a lot of pain if you get used to not parsing ls. Globbing is both simpler and safer. – terdon Jan 15 '14 at 15:55
  • @terdon: `for i in $(ls) do command "$i"; done`, I use this a lot and, globing is not always an option. Making habits < understanding the limitations of the tools I use, of course when performing sensitive tasks it's important to be careful, but space should be left for quick idioms that just work regardless of the fact they are not suitable for everything – Aki Jan 15 '14 at 18:38
  • 1
    @Aki `for i in *; do command "$i"; done` is easier to write and safer. Why would you want to learn an idiom that is harder and more risky? Try `for i in $(ls) ` in a directory that contains files with spaces in their names. Seriously, parsing `ls` is almost always not the right or easiest way to do things. If globbing is not an option, use `find`. If you insist on using `ls` at least don't use a `for` loop, this is much safer: `ls | while read i; do command "$i"; done` at least that won't break on spaces. – terdon Jan 15 '14 at 18:44
  • @terdon: didn't think of using globing there and thanks for the while loop advice. I kept using something because I wouldn't bother to learn better, that's fixed now – Aki Jan 16 '14 at 17:05
  • @terdon : GNU parallel isn't available on my platform. Is there a way to do this with xargs (I don't want to run more gz instances than my number of cores). – user2284570 Aug 29 '14 at 21:32
0

Doing this works:

   ls *.gz | parallel -t gunzip

The -t is optional but is useful as it shows you the commands that are executed on stderr.

I'm not sure you are doing anything wrong ::: should work (it's meant to be equivalent) but not even the examples in the man page work for me.

Update: the --gnu flag makes it work as terdon said.

parkydr
  • 2,297
  • 1
  • 18
  • 18