Controlling Erlang’s Heart

Erlang’s heart feature provides a heartbeat-based monitoring capability for Erlang runtime systems, with the ability to restart a runtime system if it fails. It works reasonably well, but one issue with it is that if an error occurs such that it causes repeated immediate runtime crashes, heart will happily keep restarting the runtime over and over again, ad infinitum.

For yaws 1.80, released a few days ago on Feb. 12, I added a check to the heart setup in the yaws startup script to prevent endless restarts. I thought I’d share it here because it’s useful for Erlang systems in general and is in no way specific to yaws. It works by passing startup information from one incarnation to the next, checking that information to detect multiple restarts within a given time period. We track both the startup time and the restart count, and if we detect 5 restarts within a 60 second period, we stop completely. This is not to say that yaws is in dire need of this capability — it’s extremely stable in general and 1.80 in particular is a very good release — but I added it mainly because other Erlang apps sharing the same runtime instance as yaws may not enjoy that same high level of stability, especially while they’re still under development.

The command heart runs to start a new instance is set in the HEART_COMMAND environment variable. For yaws, it’s set like this (I’ve split this over multiple lines for clarity, but it’s just one line in the actual script):

HEART_COMMAND="${ENV_PGM} \ HEART=true \ YAWS_HEART_RESTARTS=$restarts \ YAWS_HEART_START=$starttime \ $program "${1+"$@"}

where

• ${ENV_PGM} is /usr/bin/env, which allows us to set environment variables for the execution of a given command.
• HEART is an environment variable that we use to indicate the command was launched by heart.
• YAWS_HEART_RESTARTS is an environment variable that we use to track the number of restarts already seen. The yaws script initially sets this to 1 and increments it for each heart restart.
• YAWS_HEART_START is an environment variable that we use to track the time of the current round of restarts. This is tracked as UNIX time, obtained by the script via the “date -u +%s” command.
• $program is the yaws script itself, i.e., $0.
• ${1+"$@"} is a specific shell construct that passes all the original arguments of the script unchanged along to $program.

The yaws script looks for HEART set to true, indicating that it was launched by heart. For that case, it then checks YAWS_HEART_RESTARTS and YAWS_HEART_START to see how many restarts we’ve seen since the start time. We get the current UNIX time and subtract the YAWS_HEART_START time; if it’s less than or equal to 60 seconds and the restart count is 5, we exit completely without restarting the Erlang runtime. Otherwise we restart, first adjusting these environment variables. If the restart count is less than 5 within the 60 second window, we increment the restart count and set the new value into YAWS_HEART_RESTARTS but keep the same YAWS_HEART_START time. But if the current time is more than 60 seconds past the start time, we reset YAWS_HEART_RESTARTS to 1 and set a new start time for YAWS_HEART_START. Look at the yaws script to see the details of this logic — scroll down to the part starting with if [ “$HEART” = true ].

Note that this approach is much like the way Erlang receive loops generally track state, by recursively passing state information to themselves.