Short version: After much investigation, I was unable to install software on MacOS from source with either llvm and -mretpoline flag, or with gcc and -mindirect-branch=thunk -mfunction-return=thunk -mindirect-branch-register flags. This seems more general than R and specific to MacOS, so I changed the title accordingly. I suspect that this cannot be done on a Mac as of 27 April 2018. I am running macOS High Sierra version 10.13.3 (17D102).
Long version: The following applies to GnuPG, which I decided to install before R (because R requires gfortran, which requires gcc, which requires MPFR, which comes with a GPG signature that I wanted to verify). I followed the steps to install GPG from Git.
Latest LLVM (with Homebrew)
LLVM from Apple fails (see below), so I used LLVM clang 6 to fix this and I installed with homebrew (it defeats a bit the purpose of compiling from source with specific flags, but I was running out of time). I installed homebrew with:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
then updated it twice with
brew update
Installing clang with Homebrew required XCode, so I installed that from the App Store. Then I followed the steps at this page:
brew install --with-toolchain llvm
Then I added the -mretpoline flag to both C, I added the path to both C and C++ compilers, and I called the shell scripts from GPG:
export CFLAGS='-mretpoline'
export CC=/usr/local/opt/llvm/bin/clang
export CXX=/usr/local/opt/llvm/bin/clang++
./autogen.sh
./configure --sysconfdir=/etc --enable-maintainer-mode && make
I got this error:
checking whether the C compiler works... no
The log file config.log gives more details:
configure:4049: /usr/local/opt/llvm/bin/clang -mretpoline conftest.c >&5
fatal error: error in backend: MachO doesn't support COMDATs, '__llvm_retpoline_r11' cannot be lowered.
clang-6.0: error: clang frontend command failed with exit code 70 (use -v to see invocation)
This thread with a comment from January 2018 at the bottom saying that Mac is not yet supported with -mretpoline:
samkellett: @chandlerc is MacOS support omitted intentionally (ie. not implemented yet)?
chandlerc: Not at all, I just don't have a Mac system to test with...
It has no response since then. So installing with LLVM failed.
GCC (with Homebrew)
Another alternative was to compile with gcc instead of LLVM. I installed the latest version of clang (6.0) with
brew install gcc
And I added the gcc flags from this page, which are different from LLVM. This bash script adds the flags to both C and C++, gives the paths to both compilers, and call the shell scripts from GPG:
export CFLAGS='-mindirect-branch=thunk -mfunction-return=thunk -mindirect-branch-register'
export CXXFLAGS=$CFLAGS
export CC=/usr/local/opt/gcc/bin/gcc-7
export CXX=/usr/local/opt/gcc/bin/g++-7
./autogen.sh
./configure --sysconfdir=/etc --enable-maintainer-mode && make
and again I get the error:
checking whether the C compiler works... no
The log file config.log gives more details:
configure:4027: checking whether the C compiler works
configure:4049: /usr/local/opt/gcc/bin/gcc-7 -mindirect-branch=thunk -mfunction-return=thunk -mindirect-branch-register conftest.c >&5
Undefined symbols for architecture x86_64:
"__x86_return_thunk", referenced from:
_main in ccZuBhFQ.o
(maybe you meant: ___x86_return_thunk)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
configure:4053: $? = 1
configure:4091: result: no
It's weird that the compiler knows about the similar sounding name with an extra underscore.
So gcc failed too. And now I'm at a loss.
Appendix: LLVM from Apple
The following bash script exports the flags for make and calls the shell scripts from GnuPG:
export CFLAGS='-mretpoline'
export CXXFLAGS=$CFLAGS
echo $CFLAGS
echo $CXXFLAGS
./autogen.sh
./configure --sysconfdir=/etc --enable-maintainer-mode && make
It fails with the compiler that ships from Apple, but it shows that the flags are getting through to the compiler:
configure:4045: gcc -mretpoline -mindirect-branch=thunk -mfunction-return=thunk -mindirect-branch-register conftest.c >&5
clang: error: unknown argument: '-mretpoline'
So @seth's comment is right in how to send flags to the compiler.