Super User Stack Exchange
Super User Stack Exchange

Questions tagged [gnu-parallel]

GNU Parallel is a command-line tool that allows one to run multiple commands in parallel

From the website:

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.

GNU Parallel was written by Ole Tange, who also happens to have a Super User account.

The documentation is available online.

47 questions
10
votes
2 answers

How to use GNU parallel with gunzip

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything. I tried parallel 'gunzip {}' ::: `ls *.gz` parallel gunzip `ls *.gz` with no results, bash tells me: /bin/bash:…
gc5
  • 330
  • 1
  • 2
  • 14
10
votes
3 answers

How can I install GNU Parallel alongside Moreutils?

Homebrew has a formula for moreutils and GNU parallel. GNU Parallel conflicts with Moreutils since it also has a binary called parallel, which is just less useful. However I'd still like to install both formulae at the same time. How can I do…
slhck
  • 223,558
  • 70
  • 607
  • 592
5
votes
2 answers

Suppressing stderr in GNU Parallel

I'm using GNU Parallel to concurrently run a command several thousand times. To get logs of the execution I'm using --files and --results. To get nice progress bar while it's running I'm using --eta and --progress. Now, my problem is that while…
Jasiu
  • 171
  • 1
  • 2
4
votes
3 answers

Gnu parallel and ack not playing nicely due to stdin, pipe

I'm trying to use parallel and ack together to do some searching in parallel. However, ack seems to insist on using stdin if it finds itself in a pipe, even if you give it files to search: $ echo hello > test.txt $ ack hello test.txt hello $ echo…
mgalgs
  • 2,312
  • 4
  • 23
  • 33
4
votes
2 answers

How do I use GNU split's "filter" option with GNU parallel?

I am trying to split a number of huge gz file into N-line compressed gzipped chunks. To demonstrate, let us consider the following: seq 100 | gzip > big_file0.gz I can split this into multiple 10-line compressed files as follows: zcat big_file0.gz…
saffsd
  • 143
  • 1
  • 4
4
votes
2 answers

GNU "parallel --pipe" doesn't process stdin by lines

I'm super confused about how to use GNU parallel to pass stdin to the job command. I have what I imagined to be a really common use case. I have some process xxd that does something with stdin and outputs to stdout. I have some way to generate or…
ThorSummoner
  • 1,130
  • 1
  • 15
  • 25
4
votes
2 answers

GNU Parallel - global variables and function

I have this script: GLOBAL_VAR="some global value" function test { echo $1 echo ${GLOBAL_VAR} } export -f test parallel --jobs 5 --linebuffer test ::: "${files[@]}" How can I have $GLOBAL_VAR visible from parallel?
Martin Perry
  • 143
  • 5
3
votes
1 answer

Using sed with parallel gives empty output when redirecting to file

I'm using the zsh shell. I am trying to use sed to substitute some text in many files, using parallel to speed up the process. When I tested this on one file and let the command output go to stdout I saw the expected result. When I tried to redirect…
Simon O'Hanlon
  • 135
  • 2
  • 8
3
votes
1 answer

Gnu Parallel hangs as one process is "sleeping"

I am running a command in parallel using Gnu Parallel, which has two parameters as input, a directory and a conf file: parallel --gnu my_command ::: (ls -d dir*test) ::: properties.conf I run it on top of a multi core cpu (24 cores) and…
Randomize
  • 543
  • 1
  • 6
  • 12
3
votes
1 answer

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl >> OUTPUT Is there a better option that will show…
William Entriken
  • 2,224
  • 5
  • 27
  • 39
3
votes
1 answer

GNU parallel does not split work evenly

My understanding is that the -X option should distribute arguments evenly among the jobs. Yet, I get a very skewed distribution: user@host:/tmp/ptest$ count() { > echo $# > } user@host:/tmp/ptest$ export -f count user@host:/tmp/ptest$ count…
Zoltan
  • 195
  • 1
  • 9
3
votes
2 answers

Achieve better compression for multiple gzipped files

I have several directories containing thousands of gzip files (overall we are talking about 1M files). Some of these files are corrupted and most of them are really small in size (a couple of KB). Almost all of them are highly similar in content,…
nopper
  • 131
  • 3
3
votes
2 answers

Multiple read from a txt file in bash (parallel processing )

Here is a simple bash script for HTTP status code while read url do urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "${url}" --max-time 5 ) echo "$url $urlstatus" >> urlstatus.txt done < $1 I am…
user7423959
  • 45
  • 1
  • 7
3
votes
2 answers

execute a command for each line of stdin immediately?

How do I execute a command for each line of stdin immediately? Example: (echo abc; sleep 10; echo def;) | ??? The xargs tool does not do the job because it blocks until it sees all the data (waits for stdout to close). Yet I have to execute…
VasyaNovikov
  • 3,366
  • 3
  • 25
  • 33
2
votes
1 answer

gnu-parallel encrypting files with spaces or special characters?

I'm trying to encrypt a bunch of files with the code below: find . -name "*.vi" | sort | parallel --gnu -j 4 --workdir "$PWD" ' echo "Encrypting {/.} ..." gpg -r user@myemail.com -o "/tank/test/{/.}.gpg" -e "{}" '; This works fine, but only…
Weekender
  • 45
  • 6
1
2 3 4