Systemd Refuses to Start Redis Server

Question

I have duplicate redis.conf & /etc/systemd/system/redis.service files on my test and prod servers via scp.

Where prod is able to start and run redis with a unix socket. My test server refuses to start redis, with the following error output:

● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-07-31 01:42:29 HKT; 27s ago
       Docs: http://redis.io/documentation,
             man:redis-server(1)

redis-server.service: Scheduled restart job, restart counter is at 5.
Stopped Advanced key-value store.
redis-server.service: Start request repeated too quickly.
redis-server.service: Failed with result 'exit-code'.
Failed to start Advanced key-value store.

I have read many threads related to restart counter is at 5, but none of them have been specific to redis.service.

/etc/systemd/system/redis.service contains:

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)

[Service]
Type=forking
ExecStart=/usr/bin/redis-server --loglevel verbose /etc/redis/redis.conf
PIDFile=/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis-socket
RuntimeDirectory=redis
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWritePaths=-/var/lib/redis
ReadWritePaths=-/var/log/redis
ReadWritePaths=-/var/run/redis


NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# redis-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target
Alias=redis.service

But when I run systemctl restart redis along with tail -f /var/log/redis/redis-server.log

Nothing is written to the log.... nothing.

This log command journalctl -f -u redis-server gave output journalctl -f -u redis-server ... seems redis doesn't like my custom group redis-socket (members redis and a virtuamin server owner called server-owner, necessary to give server owner access to unix socket, works on Prod...previously worked on Test. Better way to config/define the group? EDIT Problem solved. The core problem was that root had somehow become the owner of /etc/redis correcting ownership on that directory to redis:redis got things working again with my custom group as configured below.

-- Logs begin at Sun 2022-07-31 07:37:12 HKT. --
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:55:09 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Jul 31 08:55:09 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:55:09 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:55:13 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:55:13 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:55:13 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8330]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8330]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 1.
Jul 31 08:56:36 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8333]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8333]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 2.
Jul 31 08:56:36 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8336]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8336]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 3.
Jul 31 08:56:36 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8339]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8339]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 4.
Jul 31 08:56:37 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:37 test.example.test systemd[8342]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:37 test.example.test systemd[8342]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:37 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Jul 31 08:56:37 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:37 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:43 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:56:43 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:43 test.example.test systemd[1]: Failed to start Advanced key-value store.

lil' help?

Could you start `redis-server` with verbose output and add the logs? — mashuptwice, Jul 30 '22 at 18:04
@mashuptwice thx Its 2AM here.. might you have a command for that? I am searching now — mjones, Jul 30 '22 at 18:06
https://www.mankier.com/1/redis-server edit the systemd service file with the desired loglevel or start it from an interactive shell — mashuptwice, Jul 30 '22 at 18:09
thanks for that, but no luck.. I just edited the post... so that you can see my systemd config — mjones, Jul 30 '22 at 18:18
The service file is unlikely to help without knowing the cause of the failure. Please add the verbose logs. To get verbose output , edit the service file from `ExecStart=/usr/bin/redis-server /etc/redis/redis.conf` to `ExecStart=/usr/bin/redis-server --loglevel verbose /etc/redis/redis.conf`. If it is too late to wrap your head around enabling logging, go to sleep and try again tomorrow. — mashuptwice, Jul 30 '22 at 19:14
@mashuptwice took your advice and closed my eyes a bit. I also edited the service as requested and did systemctl daemo-reload. The tail command above still shows nothing being written to the redis server log when I try to restart redis. — mjones, Jul 31 '22 at 00:29
Use `journalctl -r -u redis-server` instead of tail or `journalctl -f` to get live output of the syslog while you try to start the service with `systemctl start redis-server`. I suspect logs aren't configured to this location or the user `redis` has no privileges to write to the various directories. This is basic troubleshooting and you should familiarize yourself with the different approaches for getting detailed logs on your system. — mashuptwice, Jul 31 '22 at 00:39
Problem solved. Thx for your help. The core problem was not the custom group. As I pondered above. After I reset the group to the default it still wouldn't start, so I had to 1.) remove your loglevel config from the service and 2.) give /etc/redis correct permission for the redis user... unbeknownst to me root had become the owner of /etc/redis instead of redis... correcting that got everything to work, including my custom group as originally configured above — mjones, Jul 31 '22 at 01:38
Also you should consider to add a (hopefully detailed) answer to your own question, as you got it solved by yourself. — mashuptwice, Jul 31 '22 at 02:31

score 0 · Answer 1 · answered Aug 07 '22 at 22:55

Ok problem solved.. thanks to the help of @mashuptwice and a good night cleep...

Cause of problem:
Me. I work in multiple terminals and accidentally opened /etc/redis.conf in nano in 2 different terminal windows. While I caught part of my error and deleted .redis.conf.swp file that was created, I simply didn't notice, that the original /etc/redis.conf had changed its permissions to be root/root instead of redis/redis.

The tail -f /var/log/redis/redis-server.log threw no useful error mesages at all to indicate the problem above.

Solution:
The journalctl -f -u redis-server provided by @mashuptwice enabled to the permissions errors beginning with my redis-server.service config and eventually leading me to to find the permissions error on /etc/redis.conf.

In a nutshell, the native redis log, does not seem to very useful when troubleshooting, use journalctl -f -u redis-server if have issues with redis.

Might slap in there what permissions should look like too for something for folks to check per your solutions and finding applicable detail in the logs. Perhaps just checking security requirements on application config files set as user x with x permission and some output examples with ls commands would be a great addition/extension to this answer. — Vomit IT - Chunky Mess Style, Jul 05 '23 at 18:34

Systemd Refuses to Start Redis Server

1 Answers1