Sometimes you have to face nasty service crashes, e.g. due to memory leaks (yes, logstash, I’m talking about you). Using the systemd timers feature is a handy and idiomatic way to restart systemd services (ofc we could use cron, but that’s not what this is about anyway). However, everyone knows by now that systemd can be mystical sometimes and its arcanes are somehow complex to understand.

That being said, I want to restart my logstash service 4 times after it’s been started. Therefore I need two files:

  • A service unit file that restarts my logstash service
  • A timer file that triggers my restart service

The restart service unit file looks straightforward:

# /etc/systemd/system/logstash-restart.service
[Unit]
Description=restart logstash

[Service]
Type=oneshot
ExecStart=/bin/systemctl restart logstash

However, I struggled a bit on how to restart the service every 4 hours after activation. I did not want to use the OnCalendar option because that would be akin to using cron and wouldn’t leverage the granularity of systemd timer functions. As it turns out, timers can be scheduled to be activated at a given time interval after they’ve been started with the OnActiveSec directive but that would only kick in once. The timer needs to be told to activate again after its initial activation with the OnUnitActiveSec directive. Eventually, the time when the timer was activated can be stored on disk and that can be done with the Persistent option. This option is false by default, which is OK because it doesn’t need to be enabled in my case - I want the timer to activate only after the service has been running for a given number of hours. This gives us the following timer file:

# /etc/systemd/system/logstash-restart.timer
[Timer]
OnActiveSec=4h
OnUnitActiveSec=4h

[Install]
WantedBy=timer.target

With both of those files on disk, there remains the need to enable and start the timer (that creates a symbolic link at the appropriate target location so it will be started at next boot, and also starts the countdown from now on):

# systemctl enable logstash-restart.timer
# systemctl start logstash-restart.timer

Checking if the timer shows up:

# systemctl list-timers --all
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
Fri 2017-11-17 20:32:19 CET  3h 59min left Thu 2017-11-16 19:50:23 CET  20h ago      logstash-restart.timer       logstash-restart.service

systemd tracks execution times closely so it’s easy to tell whenever the timer has fired and when it will activate next.