systemctl start service command is not working in userdata of ec2

Question

I am trying to run systemctl service in ec2 instance on launching of ec2 instance. So for to run my script on start of ec2 instance i am running my script using user data of ec2 instance. So my script has the following lines of code in bash

start.sh

#!/usr/bin/env sh
set -e

FILE=/etc/rsyslog.d/"$SERVICE_NAME".conf
  sudo touch "$FILE"

  cat <<EOM | sudo tee "$FILE"
if \$programname == "$SERVICE_NAME" then "$ROUTER_LOGS"
& stop
EOM

ROUTER_LOGS=/var/log/router/out.log
  FILE=/usr/local/bin/upload_logs.sh
  sudo touch $FILE
  chmod +x $FILE
  S3_BUCKET=my-bucket
  cat <<EOM | sudo tee $FILE
aws s3 cp $ROUTER_LOGS s3://$S3_BUCKET/logs/$HOSTNAME/$SERVICE_NAME.log.$(date +%Y-%m-%d_%H%M%S)
EOM

FILE=/etc/logrotate.d/router
  sudo touch $FILE
  cat <<EOM | sudo tee $FILE
/var/log/router/*.log {
    hourly
    rotate 24
    missingok
    notifempty
    compress
    delaycompress
    endscript
}

FILE="/etc/systemd/system/router.service"
  cat <<EOM | sudo tee $FILE
[Unit]
  Description=$SERVICE_NAME
  After=network.target
  Requires=network.target

[Service]
  WorkingDirectory=$APP_DIR/target/release/
  ExecStart=$APP_DIR/target/release/router
  ExecStopPost=/bin/bash /usr/local/bin/upload_logs.sh
  StandardOutput=syslog
  StandardError=syslog
  SyslogIdentifier=$SERVICE_NAME
  TimeoutStartSec=1000s
  Environment="RUST_LOG=info"
  Environment="RUST_ENV=$ENV"
  Environment="APP_DIR=$APP_DIR"
  Restart=on-failure
  Type=simple

[Install]
  WantedBy=multi-user.target

EOM


  sudo systemctl daemon-reload
  sudo systemctl enable router.service
  sudo systemctl status router.service #
  sudo systemctl start router.service
  sudo systemctl status router.service

The output of the above script is

● router.service - gql-router
   Loaded: loaded (/etc/systemd/system/router.service; enabled; vendor preset: disabled)
   Active: inactive (dead)

I am using Centos 7 linux in EC2 of AWS.

But It is working when i tried running it inside machine using ssh. So the problem is running using user data of EC2. I tried checking logs using journalctl but no entries found of the service. Can anyone please help me with this problem in solving.

Did you try to replace `set -e` with `set -x` to debug what's happening instead of existing? And keep in mind that user-data runs only once at the time of the EC2 instance creation, not at every restart — Algeriassic, Mar 25 '23 at 20:54
Thanks @Algeriassic for suggestion. When i tried set -x it starts the router.service. But how does set -e is terminating the systemctl start command, even it is returning zero exit code. Can it because of systemctl status command returning non-zero exit code — dhruv mehta, Mar 26 '23 at 08:44
`set -e` exits at any error / non- zero code. Any useful logs after enabling debugging? And did you try to add some time `sleep 5000` for example at the beginning of the script to make sure the instance loaded all needed services? — Algeriassic, Mar 26 '23 at 14:34
```systemctl status``` before ```systemctl start``` produces non zero exit code which can stop the script due to ```set -e```. So the script will never reach ```systemctl start``` command. ```systemctl status``` will return non zero because the service is not started. it is returning 3 as exit code. — dhruv mehta, Mar 26 '23 at 17:48
Please check my answer and let me know if that fixes the issue. — Algeriassic, Mar 29 '23 at 19:47

score 0 · Accepted Answer · answered Mar 27 '23 at 12:49

The script will exit at any error because of the set -e.

For better troubleshooting, you need have set -x to debug and see what's happening.

Potential cause:

After=network.target Requires=network.target

It looks like the scrip needs network services to be up, and if they are not, it exits.

The fix would be to wait for the network services to come up.

For this, you need to replace these 2 lines with:

After=network-online.target Wants=network-online.target

This will make it wait for the network services to come online. It will fail (timeout) after 90 seconds.

Hope this help, and fixes your issue.

systemctl start service command is not working in userdata of ec2

1 Answers1