Newlines in sed on Mac OS X

Question

I find that \n doesn't work in sed under Mac OS X. Specifically, say I want to break the words separated by a single space into lines:

# input
foo bar

I use,

echo "foo bar" | sed 's/ /\n/'

But the result is stupid, the \n is not escaped!

foonbar

After I consulted to google, I found a workaround:

echo 'foo bar' | sed -e 's/ /\'$'\n/g'

After reading the article, I still cannot understand what \'$'\n/g' means. Can some one explain it to me, or if there is any other way to do it? Thanks!

this would probably work too: `echo "foo bar" | tr ' ' '\n'` — glenn jackman, Jul 06 '11 at 18:17
Thanks for the advice. But currently I just use the above case as an example, I do need to know how to escape a `\n`. — Ivan Xiao, Jul 06 '11 at 22:50
Possible duplicate: http://stackoverflow.com/questions/21621722/removing-carriage-return-on-mac-os-x-using-sed — hyiltiz, Apr 19 '17 at 07:05
Maybe you could also use Perl instead of sed which would make it simpler. See my [answer here](https://stackoverflow.com/a/63777484/111036) for details. Basically, it would be `echo "foo bar" | perl -pe 's/ /\n/` or `perl -pe 's/ +/\n/g` to replace all spaces or groups of spaces by a new line. — mivk, Sep 07 '20 at 12:26
I just noticed a script running on Big Sur escaped the newline just fine, but the same script on Catalina showed your output... not a newline. I think Apple updated it to address this issue. — Mark A. Donohoe, Dec 23 '20 at 05:21

Ahmed Fasih · Answer 1 · 2019-12-19T02:14:44.003

38

You can brew install gnu-sed and replace calls to sed with gsed.

To use it as sed instead of gsed, brew helpfully prints the following instructions after you install:

GNU "sed" has been installed as "gsed".
If you need to use it as "sed", you can add a "gnubin" directory
to your PATH from your bashrc like:

    PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

That is, append the following line to your ~/.bashrc or ~/.zshrc:

export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

edited Dec 19 '19 at 02:14

answered Apr 14 '13 at 01:33

Ahmed Fasih

859
1
8
9

Using the same software on all platform is way easier (at last to me) than dealing with every specificities of the Mac version. Beware, the option is now `--with-default-names` and *not* `--default-names`. However, this option did not worked on my installation, so I had to put a `alias gsed=sed` in my `~/.profile` to make it work. – Clément May 11 '15 at 11:40
5

This is an ugly workaround. It doesn't explain why `sed` on OS X behaves the way it does and makes the incorrect assumption that `gnu-sed` is more correct. Don't be a GNU-addict and stick to POSIX standards to avoid problems in the long run. – octosquidopus Oct 08 '15 at 02:13
It is how Apple should make its OS works by default. Why use the name of a program and then make it works differently? I waste an hour because of this... – rascio Apr 04 '17 at 10:58
@rascio - because OS X is BSD-like and Linux is Linux-like. – iAdjunct May 30 '18 at 14:30
@rascio I think you'll find that GNU sed is the newcomer; BSD sed dates back to 1979 (see https://en.wikipedia.org/wiki/Version_7_Unix). – Duncan Bayne Sep 10 '18 at 05:19
This is the preferred answer in most cases I think. If you get a compile script in your lap you don't want to spend time modifying it just to make it work with OSX. Instead, simply do `brew install gnu-sed`, `alias sed='gsed'`, `sh compile.sh`, and remove the alias when you're done (`unalias sed`). – Robin Keskisarkka Apr 02 '19 at 12:53
Correction: For the alias to work correctly inside the script, the command should be `source compile.sh` – Robin Keskisarkka Apr 02 '19 at 13:03
`brew install gnu-sed --with-default-names` gives me `Error: invalid option: --with-default-names` – k1eran Dec 18 '19 at 16:01

Lri · Answer 2 · 2013-04-14T11:48:44.927

33

These would also work:

echo 'foo bar' | sed 's/ /\
/g'

echo 'foo bar' | sed $'s/ /\\\n/g'

lf=$'\n'; echo 'foo bar' | sed "s/ /\\$lf/g"

OS X's sed doesn't interpret \n in the replace pattern, but you can use a literal linefeed preceded by a line continuation character. The shell replaces $'\n' with a literal linefeed before the sed command is run.

edited Apr 14 '13 at 11:48

answered Jul 07 '11 at 05:58

Lri

40,894
7
119
157

7

Why does `sed $'s/ /\\\n/g'` work, but not `sed $'s/\\\n/ /g'`? – Alec Jacobson Jan 03 '16 at 23:03
2

@alec You can't use `sed` even in linux/unix to remove newlines because its parsed/split at each newline. If you run this in linux/unix, it won't do anything either: `echo -e 'foo\nbar' | sed 's/\n//'` – wisbucky Oct 27 '17 at 01:05

score 12 · Answer 3 · answered Jul 07 '11 at 03:57

The workaround you found passes a single argument string to sed -e.

That argument ends up being a string in the familiar sed s/ / /g format.

That string is created in two parts, one after the other.

The first part is quoted in '...' form.

The second part is quoted in $'...' form.

The 's/ /\' part gets the single-quotes stripped off, but otherwise passes through to sed just as it looks on the command-line. That is, the backslash isn't eaten by bash, it's passed to sed.

The $'\n/g' part gets the dollar sign and the single-quotes stripped off, and the \n gets converted to a newline character.

All together, the argument becomes

s/ /\newline/g

[That was fun. Took a while to unwrap that. +1 for an interesting question.]

score 8 · Answer 4 · answered Jul 06 '11 at 16:15

8

The expression $'...' is a bash-ism which produces ... with the standard escape sequences expanded. Th \' before it just means a backslash followed by the end of the quoted section, the resulting string is s/ /\. (Yes, you can switch quoting in the middle of a string; it doesn't end the string.)

POSIX standard sed only accepts \n as part of a search pattern. OS X uses the FreeBSD sed, which is strictly POSIX compliant; GNU, as usual, adds extra stuff and then Linux users all think that is some kind of "standard" (maybe I'd be more impressed if either of them had a standards process).

answered Jul 06 '11 at 16:15

geekosaur

11,723
32
40

Now I understand the `$'...'` part. But... what is `s/ /\ `? What do you mean by `switch quoting`? – Ivan Xiao Jul 06 '11 at 22:53
2

`'...'` is one kind of shell quoting; `$'...'` is another. There's also `"..."` and `\x` to quote a single character. You can combine those in a single word, which is what was being done there, switching from a normal `''` string to a `$''` string to translate the `\n`. As for the rest, it's building up a `sed` command (`s/text/replacement/flags`). In this case the command is started, including a backslash at the end to protect the literal newline that the `$'\n/g'` appends. The result is to replace all (the `/g` flag) spaces with newlines. – geekosaur Jul 06 '11 at 23:01

score 7 · Answer 5 · answered Oct 27 '17 at 01:36

7

There's a very easy to visually see what's happening. Simply echo the string!

echo 's/$/\'$'\n/g'

results in

s/$/\
/g

which is equivalent to s/$/\newline/g

If you didn't have the extra \ before the newline, the shell would interpret the newline as the end of the command prematurely.

answered Oct 27 '17 at 01:36

wisbucky

2,928
31
29

this should have been the accepted answer – grepit Aug 27 '21 at 21:26

score 3 · Answer 6 · answered Mar 10 '20 at 17:06

3

This workaround works on Mac and the script can be executed also on Linux

NL="\n"
if [[ $uname -eq "Darwin" ]]; then
    NL=$'\\\n'
fi

echo 'foo bar' | sed -e "s| |${NL}|"

answered Mar 10 '20 at 17:06

Massimo Zerbini

131
2

slm · Answer 7 · 2020-06-03T19:14:35.963

0

A bit more compact form of @max-zerbini's answer:

[[ $(uname) -eq "Darwin" ]] && NL=$'\\\n' || NL="\n"
echo 'foo bar' | sed -e "s| |${NL}|"

edited Jun 03 '20 at 19:14

answered Jun 03 '20 at 16:14

slm

9,959
10
49
57

Newlines in sed on Mac OS X

7 Answers7

Linked