Speeding up find command result with folder exclusions

Question

I have the following folder structure:

Approvals_Share
    1 CUST APPROVAL
    2 LICENSE APPROVAL
    CANADA INSCRIBE
    z-remove

There are multiple subfolders and illustrator files “.ai” inside each folder

If I perform a find individually on each folder I get the results back in the amount of time specified below between square brackets:

1 CUST APPROVAL: [14m 19s]
2 LICENSE APPROVAL: [10m 17s]
CANADA INSCRIBE: [13s]
z-remove: [2s]

Here is a sample shell command that I use to lists the z-remove folder

cd '/Volumes/Box/Approvals_Share/z-remove/'
find .  -name '*.ai' -not -name '.*' > output.txt

This task takes only 2 seconds to complete, however if I run the same command on the parent folder Approvals_Share excluding all folders except z-remove the task takes 3m 27s. I was expecting it to be 2s or way closer. Here is the shell command I’m using:

cd '/Volumes/Box/Approvals_Share/'
find .  -name '*.ai' -not -name '.*' -not -path './2 LICENSE APPROVAL/*' -not -path './1 CUST APPROVAL/*' -not -path './CANADA INSCRIBE/*' > output.txt

Is there a way to speed up the result?

Reda Salih · Answer 1 · 2020-03-13T14:35:08.487

1

Using not -path requires traversing all the excluded directories with ignoring the output, reason why it's taking more time.

The solution here consists of using -prune, it forbids find to access to the children of every pattern within -path option:

cd /Volumes/Box/Approvals_Share/
find . -path './2 LICENSE APPROVAL' -prune -o -path './1 CUST APPROVAL' -prune -o -path './CANADA INSCRIBE' -prune -o -name '*.ai' -not -name '.*' -print

Another way consists of grouping the excluded directories :

find . -type d \( -path './2 LICENSE APPROVAL' -o -path './1 CUST APPROVAL' -o -path './CANADA INSCRIBE' \) -prune -o -name '*.ai' -not -name '.*' -print

edited Mar 13 '20 at 14:35

answered Mar 11 '20 at 21:57

Reda Salih

906
7
9

This is the right approach since the excluded folders are really the known thing here. Eventually, there could be new folders like `z-remove` that will need to be scanned and listed as well. Unfortunately, there is something off with your script, it started listing the content of the excluded folders. – Gio Mar 12 '20 at 15:05
Where are you executing the command from, because i have already tested this method and it's working you should execute it on the parent directory : cd /Volumes/Box/Approvals_Share/ – Reda Salih Mar 12 '20 at 15:44
Yes, from Approvals_Share of course. I'm generating the shell commands programmatically but I'm checking at the string before running it and it matches yours, no double spaces, etc. Do you have a bunch of ai files inside your test folders? – Gio Mar 13 '20 at 13:58
Yes in each folder and in subfolders, i have made an edit to try the same in another way. – Reda Salih Mar 13 '20 at 14:30
The grouped version you posted also lists the content of the excluded paths. – Gio Mar 13 '20 at 18:10
Have you edited something on this command or just copy past because the order is important ? – Reda Salih Mar 13 '20 at 18:53

score 0 · Answer 2 · answered Mar 11 '20 at 23:12

This other answer is right but it requires you to explicitly specify all directories you want to exclude. The solution doesn't scale up well. What if there are hundreds of directories in .?

Your requirement is to exclude all directories except z-remove (and . implicitly in the first place, otherwise it wouldn't make sense). A more scalable solution should specify z-remove (and .) only, as this information is enough to exclude the rest.

If you know there are no matching files directly in . then all of them (if any at all) will be in z-remove. In such case use this straightforward approach:

find z-remove -name '*.ai' ! -name '.*'

^{(Note: I used ! because it's POSIX while -not is not.)}

If there can be matching files directly in . then you need to start in . (obviously) and prune every directory that is not . or ./z-remove or inside ./z-remove. Like this:

find . -type d ! -path . ! -path ./z-remove ! -path './z-remove/*' -prune \
       -o -name '*.ai' ! -name '.*' -print

Note your original code does not use -type f, so e.g. a directory ./z-remove/foo.ai will pass all the tests. You're talking about files and in Unix directory is also a file. But you're talking about "illustrator files", so maybe you mean regular files and -type f is what you want. If so, my solution becomes:

find . -type d ! -path . ! -path ./z-remove ! -path './z-remove/*' -prune \
       -o -type f -name '*.ai' ! -name '.*' -print

Speeding up find command result with folder exclusions

2 Answers2