Bash: Iterating over lines in a variable

Question

How does one properly iterate over lines in bash either in a variable, or from the output of a command? Simply setting the IFS variable to a new line works for the output of a command but not when processing a variable that contains new lines.

For example

#!/bin/bash

list="One\ntwo\nthree\nfour"

#Print the list with echo
echo -e "echo: \n$list"

#Set the field separator to new line
IFS=$'\n'

#Try to iterate over each line
echo "For loop:"
for item in $list
do
        echo "Item: $item"
done

#Output the variable to a file
echo -e $list > list.txt

#Try to iterate over each line from the cat command
echo "For loop over command output:"
for item in `cat list.txt`
do
        echo "Item: $item"
done

This gives the output:

echo: 
One
two
three
four
For loop:
Item: One\ntwo\nthree\nfour
For loop over command output:
Item: One
Item: two
Item: three
Item: four

As you can see, echoing the variable or iterating over the cat command prints each of the lines one by one correctly. However, the first for loop prints all the items on a single line. Any ideas?

Answer 1

With bash, if you want to embed newlines in a string, enclose the string with $'':

$ list="One\ntwo\nthree\nfour"
$ echo "$list"
One\ntwo\nthree\nfour
$ list=$'One\ntwo\nthree\nfour'
$ echo "$list"
One
two
three
four

And if you have such a string already in a variable, you can read it line-by-line with:

while IFS= read -r line; do
    echo "... $line ..."
done <<< "$list"

---

@wheeler makes a good point about <<< adding a trailing newline.

Suppose the variable ends with a newline

list=$'One\ntwo\nthree\nfour\n'

Then the while loop outputs

while IFS= read -r line; do
    echo "... $line ..."
done <<< "$list"

... One ...
... two ...
... three ...
... four ...
...  ...

To get around that, use a redirection from a process substitution instead of a here-string

while IFS= read -r line; do
    echo "... $line ..."
done < <(printf '%s' "$list")

... One ...
... two ...
... three ...
... four ...

But now this "fails" for strings without a trailing newline

list2=$'foo\nbar\nbaz'
while IFS= read -r line; do
    echo "... $line ..."
done < <(printf '%s' "$list2")

... foo ...
... bar ...

The read documentation says

The exit status is zero, unless end-of-file is encountered

and because the input does not end with a newline, EOF is encountered before read can get a whole line. read exits non-zero and the while loop completes.

The characters are consumed into the variable though.

So, the absolutely proper way to loop over the lines of a string is:

while IFS= read -r line || [[ -n $line ]]; do
    echo "... $line ..."
done < <(printf '%s' "$list2")

This outputs the expected for both $list and $list2

Answer 2

You can use while + read:

some_command | while read line ; do
   echo === $line ===
done

Btw. the -e option to echo is non-standard. Use printf instead, if you want portability.

Answer 3

#!/bin/sh

items="
one two three four
hello world
this should work just fine
"

IFS='
'
count=0
for item in $items
do
  count=$((count+1))
  echo $count $item
done

Answer 4

Here's a funny way of doing your for loop:

for item in ${list//\\n/
}
do
   echo "Item: $item"
done

A little more sensible/readable would be:

cr='
'
for item in ${list//\\n/$cr}
do
   echo "Item: $item"
done

But that's all too complex, you only need a space in there:

for item in ${list//\\n/ }
do
   echo "Item: $item"
done

You $line variable doesn't contain newlines. It contains instances of \ followed by n. You can see that clearly with:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo $list | hexdump -C

$ ./t.sh
00000000  4f 6e 65 5c 6e 74 77 6f  5c 6e 74 68 72 65 65 5c  |One\ntwo\nthree\|
00000010  6e 66 6f 75 72 0a                                 |nfour.|
00000016

The substitution is replacing those with spaces, which is enough for it to work in for loops:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013

---

Demo:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C
for item in ${list//\\n/ } ; do
    echo $item
done

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013
One
two
three
four

Answer 5

You can also first convert the variable into an array, then iterate over this.

lines="abc
def
ghi"

declare -a theArray

while read -r line
do
    theArray+=("$line")            
done <<< "$lines"

for line in "${theArray[@]}"
do
    echo "$line"
    #Do something complex here that would break your read loop
done

This is only usefull if you do not want to mess with the IFS and also have issues with the read command, as it can happen, if you call another script inside the loop that that script can empty your read buffer before returning, as it happened to me.