Custom Prolog and Epilog scripts

Slurm's prolog and epilog features allow users to specify scripts to run on each node before the first job step/at job termination. This enables admins to perform tasks like system monitoring or cleanup in a non-disruptive way at fairly frequent intervals.

TensorWave's managed Slurm solution provides /mnt/customer/prolog.d and /mnt/customer/epilog.d for cluster admins to add their custom prolog/epilog scripts, respectively. Per-node summaries of custom prolog/epilog script runs are logged in /mnt/customer/logs.

Prolog/Epilog scripts are a powerful tool for managing a cluster, but there are also a few easy ways to 'shoot yourself in the foot' with them. If a Prolog/Epilog script returns a non-zero exit code, the node will be placed in DRAIN state, so if a buggy script is deployed, it can bring down the entire cluster. Prolog/Epilog scripts are run as the root user, this provides broad access for system monitoring, but can also enable disrupting running jobs if performing cleanup tasks.

Test example of a custom prolog

In this example, we have a prolog and an epilog script. Both scripts print an output to stdout. The epilog script returns 1 to simulate a failure event.

[e-mail]@slurm-login-skip-849dbcf5c-q7ffr:~$ sudo cat /mnt/customer/prolog.d/test-prolog-1.sh
#!/usr/bin/env bash
echo "Hello from job id $SLURM_JOB_ID on node $SLURM_NODENAME"

[e-mail]@slurm-login-skip-849dbcf5c-q7ffr:~$ sudo cat /mnt/customer/epilog.d/test-epilog-1.sh
#!/usr/bin/env bash
echo "Goodbye from job id $SLURM_JOB_ID on node $SLURM_NODENAME"
exit 1

To trigger the prolog/epilog, we submit an srun job. Since the test-epilog-1.sh 'fails', the node our job ran on drains.

[e-mail]@slurm-login-skip-849dbcf5c-q7ffr:~$ srun -N 1 --gpus-per-node=8 hostname
tus1-p2-g6

[e-mail]@slurm-login-skip-849dbcf5c-q7ffr:~$ sinfo
PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpuworker*    up   infinite      1  drain tus1-p2-g6
gpuworker*    up   infinite      1   idle tus1-p2-g5

Investigating logs, the prolog runs successfully, but since the epilog failed, the full output is saved in the logs.

[e-mail]@slurm-login-skip-849dbcf5c-q7ffr:~$ cat /mnt/customer/logs/prolog/tus1-p2-g6.log
timestamp=2026-04-01T18:43:39Z script=/mnt/customer/prolog.d/test-prolog-1.sh exit_code=0 job_id=5995 job_user=[e-mail]

[e-mail]@slurm-login-skip-849dbcf5c-q7ffr:~$ cat /mnt/customer/logs/epilog/tus1-p2-g6.log
timestamp=2026-04-01T18:43:42Z script=/mnt/customer/epilog.d/test-epilog-1.sh exit_code=1 job_id=5995 job_user=[e-mail]
--- output ---
Goodbye from job id 5995 on node

Refrences

Slurm Prolog and Epilog Guide: https://slurm.schedmd.com/prolog_epilog.html