bash: using scp in cron job fails, but runs succesfully when run from command line

Question

I am trying to use scp in a bash script run by cron (I am running this on Ubuntu 10.0.4 LTS).

The script works fine (i.e. transfers and copies file1 and file2 to/from the remote server, when I run it from the command line. However, when I run the script as a cron job, it fails.

Th is is what the script looks like:

#!/bin/bash

cd /home/oompah/scripts/tests/
scp -P 12345 file1 oompah@someserver.com:~/uploads

if scp -P 12345 oompah@someserver.com:/path/to/file2.dat local.dat >&/dev/null ; then 
    echo "INFO: transfer OK" ; 
else 
    echo "ERROR: transfer failed" ; 
fi

The error message I get (redirected to a log file) when I run it as a cron job is:

ERROR: transfer failed

The error message I get sent to my mail inbox is:

Permission denied (publickey).
lost connection

Why is happening, and how may I fix it?.

[Edit]

I modified the 1st scp command with an -i command (as suggested by M Jenkins), i also added -v for debug messages. Here is the full debug message log. Hopefully, it can shed some light as to what is going on:

Executing: program /usr/bin/ssh host 12.34.56.78, user oompah, command scp -v -t ~/uploads
OpenSSH_5.3p1 Debian-3ubuntu6, OpenSSL 0.9.8k 25 Mar 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 12.34.56.78 [12.34.56.78] port 12345.
debug1: Connection established.
debug1: identity file /home/oompah/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu3
debug1: match: OpenSSH_5.3p1 Debian-3ubuntu3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '[12.34.56.78]:12345' is known and matches the RSA host key.
debug1: Found key in /home/oompah/.ssh/known_hosts:3
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/oompah/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: No more authentication methods to try.
Permission denied (publickey).
lost connection
Permission denied (publickey).

What output do you get if you don't redirect stdout and stderr to /dev/null? — bmk, Mar 31 '11 at 13:02
@bmk: without the stdout redirection, I get the following messages: Permission denied (publickey). lost connection Permission denied (publickey). — oompahloompah, Mar 31 '11 at 13:25
Suggestion: Don't ever discard stderr in scripts like this. It is more useful than having a single "ERROR" message. — u1686_grawity, Mar 31 '11 at 13:44
could it be that a.) you use an agent when interactively running the command, b.) you run it as a different user without own key pair (or home folder) or c.) that the authorized_keys on the target restricts source of the connection ...? — 0xC0000022L, Mar 31 '11 at 17:23

u1686_grawity · Accepted Answer · 2011-03-31T17:28:41.057

9

My guess:

You have a password-protected SSH keypair, which is automatically loaded by GNOME Keyring when you login. However, cron does not have access to the keyring, and ssh cannot ask for a password either (due to lack of a tty).

To quote the ssh log you added:

debug1: Offering public key: /home/oompah/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: PEM_read_PrivateKey failed
debug1: read PEM private key done: type
debug1: read_passphrase: can't open /dev/tty: No such device or address

edited Mar 31 '11 at 17:28

answered Mar 31 '11 at 13:42

u1686_grawity

426,297
64
894
966

@grawity: Thanks for the information. How do I fix the problem though? – oompahloompah Mar 31 '11 at 17:14
@oompah: Remove the password from your keypair. (If you want, you can create a second one purely for automated use, and give it to `scp -i`.) – u1686_grawity Mar 31 '11 at 17:26
@grawity: Thanks for the feedback. I am not comfortable removing a password from my Keypair (I wouldn't know how to do that in any case). You mention creating a "second one" - presumably you mean a second keypair - but this one with no password - so I can use it for automated logins. BTW, do you mean PASSPHRASE when you say password? – oompahloompah Mar 31 '11 at 17:55
@oompah: If your CentOS machine is physically secure, [it's okay.](http://superuser.com/questions/261361/) The second keypair suggestion will probably only be useful if you have the primary keypair on many systems. (OpenSSH calls it "passphrase", but not many people actually use an entire _phrase_ for their keys.) – u1686_grawity Mar 31 '11 at 18:01
I finally got this to work, after a lot of screwing around with keychain etc. In the end, I settled for you suggestion to create a passwordless keypair for cron and pass that to scp in the shell script. It shouldn't have to be this hard really ... SMH – oompahloompah Mar 31 '11 at 19:12
@oompah: The thing is, at logon your primary SSH key is unlocked by the passphrase kept in GKeyring. The GKeyring keyring is unlocked by your login password, the one _you_ enter. There's no way cron or cron>ssh could access any of these passwords. – u1686_grawity Mar 31 '11 at 19:28
To remove the password from the SSH key, see https://stackoverflow.com/questions/112396/ – iafisher Jul 18 '21 at 02:02

score 3 · Answer 2 · answered Mar 31 '11 at 13:31

It sounds like scp isn't picking up your public/private key pair from your ~/.ssh directory.

Try adding

HOME=/home/oompah

into the top of your crontab file (it should already be setting that anyway automatically)

You could also try adding

echo "DEBUG: My home dir is $HOME"

into your script to make sure it's getting the right value.

Another option is to specify the -i parameter to scp to force a specific key pair to use:

scp -i /home/oompah/.ssh/id_rsa ...

for example.

I tried your suggesting (using the -i option). Please see my updated question — oompahloompah, Mar 31 '11 at 16:58

score 3 · Answer 3 · answered Mar 31 '11 at 13:32

3

What user is cron running as? It looks like that user doesn't have access to your public key.

answered Mar 31 '11 at 13:32

quanticle

876
2
9
24

score 0 · Answer 4 · answered Apr 23 '13 at 07:38

Although not the problem in this case, cron interprets the percent-sign (%) as newline-character, so it has to be escaped (\%) or you'll end up with half a command wondering why cron simply does nothing (though it will complain in syslog).

This can cause trouble if you are working with /bin/date in your crontab.

bash: using scp in cron job fails, but runs succesfully when run from command line

4 Answers4