Prolog / Epilog

Overview

Slurm supports prolog and epilog scripts that run automatically on each worker pod at the start and end of every job. Prologs run upon allocation; epilogs run after the job completes or is cancelled.

Common uses include:

• Verifying node health before a job starts

• Cleaning up temporary files or resetting state after a job ends

• Logging job metadata for monitoring or auditing

• Enforcing site-specific policies around GPU, network, or filesystem state

Prolog and epilog scripts run as root on the worker pod, which gives them broad access but also means a failing or buggy script can affect the pod and any jobs running on it. Specifically, if a prolog or epilog script exits with a non-zero code, Slurm will place the node in DRAIN state, taking it out of service. Test scripts carefully before deploying them.

Prolog and epilog scripts also run in the critical path of every job on every allocated node, so a slow script degrades scheduling for the whole cluster. Read Best practices before writing one.

For full background on how Slurm handles prolog and epilog execution, see the Slurm Prolog and Epilog Guide.

---

Built-in scripts

TensorWave runs a set of managed prolog and epilog scripts on every job automatically. These handle node health checks (see Health Checks), GPU metrics collection for the dashboard, and dispatching your custom scripts. Your scripts always run after the built-in health checks.

---

Adding custom scripts

Custom prolog and epilog scripts go in the following directories on the shared storage volume:

Directory	Script Execution
/mnt/customer/prolog.d/	Upon allocation, on every allocated node
/mnt/customer/epilog.d/	After each job completes, on every allocated node

Scripts are executed in lexicographic order by filename. Use numeric prefixes to control ordering, and use leading zeroes if necessary to ensure accurate sorting:

/mnt/customer/prolog.d/
  01-check.sh
  10-check-something-else.sh
  20-setup-environment.sh
  99-final-step.sh

Requirements

• Scripts must be executable (chmod +x). Non-executable files are skipped with a warning in the log.

• Scripts must include a shebang on the first line (#!/usr/bin/env bash).

• Scripts run as root. A non-zero exit code will drain the node.

Example prolog script

#!/usr/bin/env bash
echo "Hello from job $SLURM_JOB_ID on node $SLURMD_NODENAME"

Install it:

sudo cp my-prolog.sh /mnt/customer/prolog.d/10-my-prolog.sh
sudo chmod +x /mnt/customer/prolog.d/10-my-prolog.sh

Example epilog script

#!/usr/bin/env bash
echo "Goodbye from job $SLURM_JOB_ID on node $SLURMD_NODENAME"
# Clean up any job-specific scratch
rm -rf /tmp/job-${SLURM_JOB_ID}

Install it:

sudo cp my-epilog.sh /mnt/customer/epilog.d/10-my-epilog.sh
sudo chmod +x /mnt/customer/epilog.d/10-my-epilog.sh

---

Best practices

Every prolog and epilog script runs on every allocated node, on every job, and the job cannot start (prolog) or release its nodes (epilog) until the scripts finish. Scripts in prolog.d/ and epilog.d/ are run serially in filename order, so their runtimes add up.

Two rules follow from that, taken directly from the Slurm documentation:

Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). Long running scripts can cause scheduling problems when jobs take a long time to start or finish. Slurm commands in these scripts can potentially lead to performance issues and should not be used.

Never call Slurm commands

Do not run squeue, scontrol, sinfo, sacct, sacctmgr, sbatch, srun, salloc, scancel, or any other Slurm client command from a prolog or epilog script.

Each of those commands opens a connection to the Slurm controller (slurmctld) and asks it to do work. Because prologs and epilogs fire on every node of every job, a single Slurm command in a script is multiplied by nodes × jobs. A few hundred nodes cycling jobs is enough to saturate the controller's RPC threads, and the symptoms show up cluster-wide:

• squeue, sinfo, and sbatch hang or time out for all users

• Jobs sit in CF (CONFIGURING) or CG (COMPLETING) for minutes

• Scheduling stalls even though nodes are idle

• Nodes drain because prolog/epilog scripts time out waiting on the controller

Use the environment instead. Slurm exports the job's details to the script, so almost everything you would query for is already available for free:

Variable	Contents
SLURM_JOB_ID	Job ID
SLURM_JOB_USER / SLURM_JOB_UID	Submitting user and UID
SLURM_JOB_ACCOUNT	Account the job is charged to
SLURM_JOB_PARTITION	Partition
SLURM_JOB_QOS	QOS
SLURM_JOB_NODELIST / SLURM_JOB_NUM_NODES	Allocated nodes
SLURM_JOB_CPUS_PER_NODE	CPUs allocated per node
SLURM_JOB_GPUS / ROCR_VISIBLE_DEVICES	GPUs allocated to the job
SLURM_JOB_WORK_DIR	Job working directory
SLURMD_NODENAME	Node this script is running on
SLURM_SCRIPT_CONTEXT	Which script is running (prolog_slurmd, epilog_slurmd, …)
SLURM_JOB_EXIT_CODE / SLURM_JOB_DERIVED_EC	Job exit code (epilog only)

Avoid this:

#!/usr/bin/env bash
# BAD: hits slurmctld on every node of every job
ACCOUNT=$(scontrol show job "$SLURM_JOB_ID" | grep -oP 'Account=\K\S+')
NODES=$(squeue -h -j "$SLURM_JOB_ID" -o %N)

Do this instead:

#!/usr/bin/env bash
# GOOD: same information, zero controller load
ACCOUNT="${SLURM_JOB_ACCOUNT}"
NODES="${SLURM_JOB_NODELIST}"

If you need something that genuinely is not in the environment, collect it outside the prolog/epilog path.

Keep scripts short and bounded

• Target well under a second of runtime.

• Bound every external call. Wrap anything that touches the network, a shared filesystem, or an external API in timeout, and give it a small limit:

  timeout 5 /usr/bin/curl -sf https://internal.example/register || true

• No retry or polling loops. sleep-and-retry loops in a prolog turn a transient blip into a cluster-wide scheduling stall.

• Avoid expensive filesystem work. Do not walk large directory trees, chown -R home directories, or write large files to shared storage. Scope cleanup to the job, for example:

  rm -rf /tmp/job-${SLURM_JOB_ID}

Fail deliberately

A non-zero exit drains the node, and a failing prolog also requeues the job. Only exit non-zero when the node is genuinely unfit to run work.

For best-effort steps such as logging, metrics, registration with an external system, swallow the failure explicitly so a transient error does not take a node out of service:

#!/usr/bin/env bash
set -uo pipefail

# Best effort: never drain the node for a logging failure.
timeout 5 /usr/bin/logger -t my-prolog "job ${SLURM_JOB_ID} starting on ${SLURMD_NODENAME}" || true

exit 0

Keep output quiet as well. Script output is captured in the per-node log only on failure (see Viewing logs), so verbose success-path output is wasted work on every job.

---

Viewing logs

Per-node prolog and epilog logs are written to:

Path	Contents
/mnt/customer/logs/prolog/<node>.log	Output from all prolog scripts on that node
/mnt/customer/logs/epilog/<node>.log	Output from all epilog scripts on that node

Each entry includes a timestamp, script path, exit code, job ID, and user. Script output is only captured in the log when the script fails.

Viewing a node's prolog log:

cat /mnt/customer/logs/prolog/tus1-p2-g6.log

Example output for a successful prolog:

timestamp=2026-04-01T18:43:39Z script=/mnt/customer/prolog.d/10-my-prolog.sh exit_code=0 job_id=5995 job_user=[e-mail]

Example output when an epilog script fails (output is included):

timestamp=2026-04-01T18:43:42Z script=/mnt/customer/epilog.d/10-my-epilog.sh exit_code=1 job_id=5995 job_user=[e-mail]
--- output ---
Goodbye from job 5995 on node

If a script fails and the node drains, check the log for the affected node first, then inspect node state with sinfo:

sinfo -n tus1-p2-g6

Once the issue is resolved, contact your cluster administrator to resume the node.

---

Scripts in /mnt/customer/prolog.d and /mnt/customer/epilog.d are writable by administrators only (chmod 1700). Logs in /mnt/customer/logs are readable by all users.