How can I run mogrify over the 3 million JPG files?

Question

I have 3 million JPG files stored in a Linux CentOS 6 server.

I want to change quality to %50 file size over 1 megabyte. I wrote this command but got an "argument list too long" error:

$ find -type f -name "*..jpg" -size +1M | xargs mogrify -quality 50 *.jpg
bash: /usr/bin/xargs: Argument list too long

How can I change the quality of millions of files?

Why do you add `*.jpg` to `xargs`? It will get the files from `find`. — choroba, Apr 24 '14 at 13:31
Possible duplicate of [How to pass a list of files to ImageMagick's montage?](https://superuser.com/questions/620053/how-to-pass-a-list-of-files-to-imagemagicks-montage) — malat, Apr 15 '19 at 10:35

choroba · Answer 1 · 2014-04-24T13:45:14.380

1

When using find and xargs, you don't need to name the files for xargs. It will get the list of files from find:

find -print0 -type f -name '*.jpg' -size +1M | xargs -0 -n100 mogrify -quality 50

-n100 will process the images by 100s. -print0 and -0 will make the pipe work even if the filenames contain whitespace.

You can also call mogrify directly from find, ideally if it supports the + ending for exec:

find  -type f -name '*.jpg' -size +1M -exec mogrify -quality 50 {} +

edited Apr 24 '14 at 13:45

answered Apr 24 '14 at 13:35

choroba

18,638
4
48
52

This will still make the argument list too long in the first command, if all are passed to xargs in one. Also, you should pipe from `find` to `xargs` with `find ... -print0 | xargs -0`. – slhck Apr 24 '14 at 13:39
Well, luckily, `find` *can do it all*. :D Instead of ending the -exec command with `+`, it can also be terminated with `;`. This executes the command once per result. Don’t forget to escape `;`! – Daniel B Apr 24 '14 at 13:49
@DanielB: What is the advantage of `;` over `+` in this case? – choroba Apr 24 '14 at 13:51
1

It doesn’t result in a command line that’s too long. Although `find` might already work around that, who knows. It’s also suitable for commands that don’t accept batch input. – Daniel B Apr 24 '14 at 13:57
Well, although it may be a bit spammy, I discovered `xargs` and `find` are, in fact, smart enough to split the arguments collection automatically. – Daniel B Apr 24 '14 at 14:02

score 1 · Answer 2 · answered Apr 24 '14 at 13:40

1

xargs supports a -n argument to limit the amount of arguments passed to whatever it calls:

find -type f -name '*.jpg' -size +1M -print0 | xargs -0 -n1 mogrify -quality 50

This will launch mogrify once per image. As mogrify can only process one file at the time, this is the way to go.

answered Apr 24 '14 at 13:40

Dennis Kaarsemaker

129
2

1

`mogrify`'s documentation shows examples with `*.jpg`. – choroba Apr 24 '14 at 13:48
Ah, I blindly trusted the manpage: "SYNOPSIS mogrify [options] input-file" - indicating it only supports one. – Dennis Kaarsemaker Apr 24 '14 at 14:29
@DennisKaarsemaker can you edit your answer then? – codepringle Mar 19 '21 at 03:13

Franck Dernoncourt · Answer 3 · 2017-08-10T02:15:04.980

A cross-platform solution with Python+convert: it will convert all PDF files of the current directory into PNG files (you can change to JPG if you prefer) multithreadedly.

from __future__ import print_function
import os
import glob
import multiprocessing      

def convert_to_png(pdf_filepath):
    '''
    Convert PDF file to PNG file
    '''
    png_filepath = '{0}.png'.format(pdf_filepath[:-4])
    print('pdf_filepath: {0}'.format(pdf_filepath))
    print('png_filepath: {0}'.format(png_filepath))
    command = 'convert -background white -alpha off -geometry 1600x1600 -density 200x200 -quality 100 -resize 800x {0} {1}'.format(pdf_filepath, png_filepath)
    print(command)
    os.system(command)

def main():
    pdf_filepaths = glob.iglob(os.path.join('.','*.pdf'))
    pool = multiprocessing.Pool(processes=4)
    pool.map(convert_to_png, pdf_filepaths)
    pool.close()
    pool.join()   
    print('done')

if __name__ == "__main__":
    main()
    #cProfile.run('main()') # if you want to do some profiling

This requires Imagemagick and Ghostscript to be installed. Works on Linux/Mac OS X/Microsoft Windows.

If you prefer to add the filename on each image, you can replace the command in convert_to_png() by:

command = 'convert  -background white -alpha off -geometry 1600x1600 -density 200x200 -quality 100 -annotate +50+50 {2} -resize 800x {0} {1}'.format(pdf_filepath, png_filepath, os.path.basename(pdf_filepath))

(See -annotate documentation)

score 0 · Answer 4 · answered Apr 15 '19 at 10:33

0

As mentioned on SO, you could also do:

$ find -type f -name "*..jpg" -size +1M > my_jpeg.txt
$ mogrify -quality 50 @my_jpegs.txt

answered Apr 15 '19 at 10:33

malat

1,223
2
15
36

How can I run mogrify over the 3 million JPG files?

4 Answers4