I have a process which is executed by init.d script on the background. Eg:
case "$1" in
start)
/bin/myprocess &
stop)
killall myprocess
restart)
killall myprocess
/bin/myprocess &
esac
In certain conditions, myprocess can fail and return. Is there any (standard) way how to detect its fail and restart in automatically?
Buildroot has three possible init systems, so there are three ways to do this:
initWith this, one adds an entry to /etc/inittab.
::respawn:/bin/myprocess
Note that BusyBox init has an idiosyncratic /etc/inittab format. The second field is meaningless, and the first field is not an ID but a device basename.
initAgain, one adds an entry to /etc/inittab.
myprocess:2345:respawn:/bin/myprocess
systemdOne writes a unit file in, say, /etc/systemd/system/myprocess.service:
[Unit]
Description=My Process
[Service]
ExecStart=/bin/myprocess
Restart=always
[Install]
WantedBy=multi-user.target
Enable this to autostart at bootup with:
systemctl enable myprocess.service
Start it manually with:
systemctl start myprocess.service
What about creating a subshell with a loop that calls constantly the same process?
If it ends, the next iteration of the loop goes on and starts it again.
(while true; do
/bin/myprocess
done) &
If the subshell dies, it's over though. The only possibility in that case would be to create another process (I'll call it necromancer) that checks whether yourprocess is alive, start it if it isn't and run this necromancer with cron, so that you can check that regularly.
Next step would be wondering what could happen if cron dies, but at some point you should feel safe and stop worrying.
The easiest way would be to add it to /etc/inittab, which is designed to do this sort of thing:
respawn If the process does not exist, start the process. Do not wait for its termination (continue scanning the /etc/inittab file). Restart the process when it dies. If the process exists, do nothing and continue scanning the /etc/inittab file.
For example, you could do this:
# Run my stuff
myprocess:2345:respawn:/bin/myprocess
You could make use of Monit . It's really easy to use and quite flexible. See for example this configuration for restarting the Tomcat process on failure.
check process tomcat with pidfile /var/run/tomcat.pid
start program = "/etc/init.d/tomcat start"
stop program = "/etc/init.d/tomcat stop"
if failed port 8080 type tcp then restart
It also has a lot of configuration examples for many use cases.
If you're not a super user or root, and if your Linux system has Docker installed, then you can create a docker image of your process, using docker to restart your process if the system is rebooted.
File: docker-compose.yml
version: "3"
services:
lserver:
image: your_docker_image:latest
ports:
- 8080:8080 # just use 8080 as an example
restart: always # this is where your process can be guaranteed to restart
To start your docker container,
docker-compose up -d
I find it's easy to handle my-own-process with auto-restart if I am not a super user of the system.
For a sample example of how to create a docker image, here is a quick example:
File: Dockerfile
FROM alpine:3.5
RUN apk update && apk upgrade && rm -rf /var/cache/apk/*
WORKDIR /app
COPY my-process-server /app
RUN ln -s /app/my-process-server /usr/local/bin/my-process-server
EXPOSE 8080
CMD ["my-process-server"]
In my case, as a quick-fix, I modified and used the solution of @Trylks to wrap the program I was launching. I wanted it to end only on clean exit.
Should run in most shells:
#!/bin/sh
echo ""
echo "Use: $0 ./program"
echo ""
#eg="/usr/bin/apt update"
echo "Executing $1 ..."
EXIT_CODE=1
(while [ $EXIT_CODE -gt 0 ]; do
$1
# loops on error code: greater-than 0
EXIT_CODE=$?
done) &
(Edit): Sometimes programs hang without quitting, for no apparent reason. (Yes, of course there's always a reason but it can take a lot of time and effort to find it, particularly if it's not your own code.)
The problem I had was that the process (a Python server) was hanging from time to time, so I needed to regularly kill and restart it. I did it with a cron task that runs every couple of hours. Here's the shell script:
#!/bin/sh
# This cron script restarts the server
# Look for a running instance of myapp.py
p=$(ps -eaf | grep "[m]yapp.py")
# Get the second item; the process number
n=$(echo $p | awk '{print $2}')
# If it's not empty, kill the process
if [ "$n" ]
then
kill $n
fi
# Start a new instance
python3 myapp.py
The process can be launched with until loop as it's been answered here: How do I write a bash script to restart a process if it dies?.
Until loop works with the requirement to run it the background, see sample:
#!/bin/sh
(until /bin/myprocess; do
echo "'/bin/myprocess' crashed with exit code $?. Restarting..." >&2
sleep 1
done) &
Comparing to the @Trylks answer with while loop the last couldn't restart process when it's written in entrypoint.sh script of a Dockerfile. Despite it the until loop is working there too..
