> For the complete documentation index, see [llms.txt](https://docs.tensorwave.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tensorwave.com/slurm/prolog-epilog.md). # Prolog / Epilog ### Overview Slurm supports prolog and epilog scripts that run automatically on each worker pod at the start and end of every job. Prologs run upon allocation; epilogs run after the job completes or is cancelled. Common uses include: * Verifying node health before a job starts * Cleaning up temporary files or resetting state after a job ends * Logging job metadata for monitoring or auditing * Enforcing site-specific policies around GPU, network, or filesystem state Prolog and epilog scripts run as **root** on the worker pod, which gives them broad access but also means a failing or buggy script can affect the pod and any jobs running on it. Specifically, **if a prolog or epilog script exits with a non-zero code, Slurm will place the node in DRAIN state**, taking it out of service. Test scripts carefully before deploying them. Prolog and epilog scripts also run in the critical path of every job on every allocated node, so a slow script degrades scheduling for the whole cluster. Read Best practices before writing one. For full background on how Slurm handles prolog and epilog execution, see the [Slurm Prolog and Epilog Guide](https://slurm.schedmd.com/prolog_epilog.html). *** ### Built-in scripts TensorWave runs a set of managed prolog and epilog scripts on every job automatically. These handle node health checks (see Health Checks), GPU metrics collection for the dashboard, and dispatching your custom scripts. Your scripts always run after the built-in health checks. *** ### Adding custom scripts Custom prolog and epilog scripts go in the following directories on the shared storage volume: | Directory | Script Execution | | ------------------------- | ------------------------------------------------- | | `/mnt/customer/prolog.d/` | Upon allocation, on every allocated node | | `/mnt/customer/epilog.d/` | After each job completes, on every allocated node | Scripts are executed in **lexicographic order** by filename. Use numeric prefixes to control ordering, and use leading zeroes if necessary to ensure accurate sorting: ``` /mnt/customer/prolog.d/ 01-check.sh 10-check-something-else.sh 20-setup-environment.sh 99-final-step.sh ``` #### Requirements * Scripts must be **executable** (`chmod +x`). Non-executable files are skipped with a warning in the log. * Scripts must include a **shebang** on the first line (`#!/usr/bin/env bash`). * Scripts run as **root**. A non-zero exit code will drain the node. #### Example prolog script ```bash #!/usr/bin/env bash echo "Hello from job $SLURM_JOB_ID on node $SLURMD_NODENAME" ``` Install it: ```bash sudo cp my-prolog.sh /mnt/customer/prolog.d/10-my-prolog.sh sudo chmod +x /mnt/customer/prolog.d/10-my-prolog.sh ``` #### Example epilog script ```bash #!/usr/bin/env bash echo "Goodbye from job $SLURM_JOB_ID on node $SLURMD_NODENAME" # Clean up any job-specific scratch rm -rf /tmp/job-${SLURM_JOB_ID} ``` Install it: ```bash sudo cp my-epilog.sh /mnt/customer/epilog.d/10-my-epilog.sh sudo chmod +x /mnt/customer/epilog.d/10-my-epilog.sh ``` *** ### Best practices Every prolog and epilog script runs on **every allocated node, on every job**, and the job cannot start (prolog) or release its nodes (epilog) until the scripts finish. Scripts in `prolog.d/` and `epilog.d/` are run **serially** in filename order, so their runtimes add up. Two rules follow from that, taken directly from the [Slurm documentation](https://slurm.schedmd.com/prolog_epilog.html): > Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). Long running scripts can cause scheduling problems when jobs take a long time to start or finish. Slurm commands in these scripts can potentially lead to performance issues and should not be used. #### Never call Slurm commands Do not run `squeue`, `scontrol`, `sinfo`, `sacct`, `sacctmgr`, `sbatch`, `srun`, `salloc`, `scancel`, or any other Slurm client command from a prolog or epilog script. Each of those commands opens a connection to the Slurm controller (`slurmctld`) and asks it to do work. Because prologs and epilogs fire on every node of every job, a single Slurm command in a script is multiplied by nodes × jobs. A few hundred nodes cycling jobs is enough to saturate the controller's RPC threads, and the symptoms show up cluster-wide: * `squeue`, `sinfo`, and `sbatch` hang or time out for **all** users * Jobs sit in `CF` (CONFIGURING) or `CG` (COMPLETING) for minutes * Scheduling stalls even though nodes are idle * Nodes drain because prolog/epilog scripts time out waiting on the controller **Use the environment instead.** Slurm exports the job's details to the script, so almost everything you would query for is already available for free: | Variable | Contents | | ---------------------------------------------- | ------------------------------------------------------------- | | `SLURM_JOB_ID` | Job ID | | `SLURM_JOB_USER` / `SLURM_JOB_UID` | Submitting user and UID | | `SLURM_JOB_ACCOUNT` | Account the job is charged to | | `SLURM_JOB_PARTITION` | Partition | | `SLURM_JOB_QOS` | QOS | | `SLURM_JOB_NODELIST` / `SLURM_JOB_NUM_NODES` | Allocated nodes | | `SLURM_JOB_CPUS_PER_NODE` | CPUs allocated per node | | `SLURM_JOB_GPUS` / `ROCR_VISIBLE_DEVICES` | GPUs allocated to the job | | `SLURM_JOB_WORK_DIR` | Job working directory | | `SLURMD_NODENAME` | Node this script is running on | | `SLURM_SCRIPT_CONTEXT` | Which script is running (`prolog_slurmd`, `epilog_slurmd`, …) | | `SLURM_JOB_EXIT_CODE` / `SLURM_JOB_DERIVED_EC` | Job exit code (**epilog only**) | Avoid this: ```bash #!/usr/bin/env bash # BAD: hits slurmctld on every node of every job ACCOUNT=$(scontrol show job "$SLURM_JOB_ID" | grep -oP 'Account=\K\S+') NODES=$(squeue -h -j "$SLURM_JOB_ID" -o %N) ``` Do this instead: ```bash #!/usr/bin/env bash # GOOD: same information, zero controller load ACCOUNT="${SLURM_JOB_ACCOUNT}" NODES="${SLURM_JOB_NODELIST}" ``` If you need something that genuinely is not in the environment, collect it **outside** the prolog/epilog path. #### Keep scripts short and bounded * **Target well under a second of runtime.** * **Bound every external call.** Wrap anything that touches the network, a shared filesystem, or an external API in `timeout`, and give it a small limit: ```bash timeout 5 /usr/bin/curl -sf https://internal.example/register || true ``` * **No retry or polling loops.** `sleep`-and-retry loops in a prolog turn a transient blip into a cluster-wide scheduling stall. * **Avoid expensive filesystem work.** Do not walk large directory trees, `chown -R` home directories, or write large files to shared storage. Scope cleanup to the job, for example: ```bash rm -rf /tmp/job-${SLURM_JOB_ID} ``` #### Fail deliberately A non-zero exit drains the node, and a failing prolog also requeues the job. Only exit non-zero when the node is genuinely unfit to run work. For best-effort steps such as logging, metrics, registration with an external system, swallow the failure explicitly so a transient error does not take a node out of service: ```bash #!/usr/bin/env bash set -uo pipefail # Best effort: never drain the node for a logging failure. timeout 5 /usr/bin/logger -t my-prolog "job ${SLURM_JOB_ID} starting on ${SLURMD_NODENAME}" || true exit 0 ``` Keep output quiet as well. Script output is captured in the per-node log only on failure (see Viewing logs), so verbose success-path output is wasted work on every job. *** ### Viewing logs Per-node prolog and epilog logs are written to: | Path | Contents | | -------------------------------------- | ------------------------------------------- | | `/mnt/customer/logs/prolog/.log` | Output from all prolog scripts on that node | | `/mnt/customer/logs/epilog/.log` | Output from all epilog scripts on that node | Each entry includes a timestamp, script path, exit code, job ID, and user. Script output is only captured in the log when the script fails. **Viewing a node's prolog log:** ```bash cat /mnt/customer/logs/prolog/tus1-p2-g6.log ``` Example output for a successful prolog: ``` timestamp=2026-04-01T18:43:39Z script=/mnt/customer/prolog.d/10-my-prolog.sh exit_code=0 job_id=5995 job_user=user@example.com ``` Example output when an epilog script fails (output is included): ``` timestamp=2026-04-01T18:43:42Z script=/mnt/customer/epilog.d/10-my-epilog.sh exit_code=1 job_id=5995 job_user=user@example.com --- output --- Goodbye from job 5995 on node ``` If a script fails and the node drains, check the log for the affected node first, then inspect node state with `sinfo`: ```bash sinfo -n tus1-p2-g6 ``` Once the issue is resolved, contact your cluster administrator to resume the node. *** > Scripts in `/mnt/customer/prolog.d` and `/mnt/customer/epilog.d` are writable by administrators only (`chmod 1700`). Logs in `/mnt/customer/logs` are readable by all users. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.tensorwave.com/slurm/prolog-epilog.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.