Jobs

Jobs are submitted and managed through standard Slurm commands. There are two primary modes:

• Interactive (srun, salloc): allocate resources and run commands directly, useful for development and debugging.

• Batch (sbatch): submit a script that runs when resources become available, the standard approach for production workloads.

All jobs must request GPU resources explicitly. The default partition is configured with reasonable CPU-per-GPU and mem-per-CPU defaults, but these can be overridden per job. For command reference, see the official Slurm documentation.

---

srun

srun is a multi-node job launcher. When run from a login pod or a salloc session, it allocates resources, launches the command directly, and waits for it to complete. This is useful for quick one-off commands and interactive debugging.

When called from within an sbatch script, srun executes a job step against the resources already allocated to the batch job. Multiple srun calls can be sequenced in a single script to perform complex multi-step workloads.

Run a command on one node with 8 GPUs:

srun -N 1 --gpus-per-node=8 amd-smi

Open an interactive shell:

srun -N 1 --gpus-per-node=8 --pty bash

Run a command across multiple nodes:

srun -N 4 --gpus-per-node=8 hostname

srun inside a batch script or salloc session inherits resources from the allocation and runs across all of them by default. Resource flags can be used to target a subset of the allocated resources, but requesting more than was allocated will cause an error.

---

Batch Jobs

Batch jobs are submitted with sbatch and run when the scheduler grants the allocation. Resource requests, environment setup, and the actual workload are all defined in the job script. Any #SBATCH directive in the script can also be overridden at submission time by passing the corresponding flag directly to sbatch.

Basic structure

#!/bin/bash
#SBATCH --job-name=my-job
#SBATCH --output=jid-%j.name-%x.log
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=96
#SBATCH --gpus-per-node=8
#SBATCH --time=01:00:00

srun my-program <my-program-args>

Submit with:

sbatch my-job.sh

Check status:

squeue -u $USER

View output once the job runs:

tail -f jid-<jobid>.name-my-job.log

Common resource flags

Flag	Description
--nodes / -N	Number of nodes
--ntasks-per-node	Number of tasks (processes) per node
--gpus-per-node	GPUs per node
--cpus-per-task	CPU cores per task
--time	Wall-clock time limit (HH:MM:SS)
--nodelist	Run on specific nodes

Multi-node distributed training example

The pattern below works for PyTorch torch.distributed.run (torchrun) across multiple nodes. The first node allocated by Slurm acts as the rendezvous host.

#!/bin/bash
#SBATCH --job-name=ddp-training
#SBATCH --output=jid-%j.name-%x.log
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=48
#SBATCH --gpus-per-node=8
#SBATCH --time=04:00:00

GPUS_PER_NODE=8
MASTER_ADDR=$(hostname)
MASTER_PORT=6000

srun bash <<EOF
LOCAL_ADDR=\$(hostname)
REMOTE_ADDR=${MASTER_ADDR}
IS_HOST=0
if [ "\$LOCAL_ADDR" == "${MASTER_ADDR}" ]; then
  IS_HOST=1
  REMOTE_ADDR=localhost
fi

export OMP_NUM_THREADS=8

python -u -m torch.distributed.run \
  --nproc_per_node $GPUS_PER_NODE \
  --nnodes $SLURM_NNODES \
  --rdzv_endpoint \${REMOTE_ADDR}:${MASTER_PORT} \
  --rdzv_backend c10d \
  --rdzv_id=1 \
  --rdzv_conf=is_host=\$IS_HOST \
  --local_addr "\$(hostname)" \
  train.py
EOF

Example scripts are available on the cluster under /opt/tw/examples/libexec/.

---

MPI (PMIx)

The cluster uses PMIx as the default MPI launch interface (MpiDefault=pmix). Open MPI and other PMIx-compatible MPI libraries work with srun without needing mpirun.

The following environment is set cluster-wide and applied automatically to all login and compute sessions:

OMPI_MCA_btl_tcp_if_include=eno0,eno1   # Route TCP over front-end to avoid back-end contention
OMPI_MCA_btl=^openib                    # Disable legacy OpenIB transport
PMIX_MCA_gds=hash                       # Required PMIx GDS backend

Running an MPI job

Use srun directly. Slurm handles process launch and PMIx initialization:

#!/bin/bash
#SBATCH --job-name=mpi-job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --time=01:00:00

srun ./my-mpi-program

We recommend using srun rather than mpirun or mpiexec, as it integrates directly with Slurm's process placement and PMIx initialization.

RCCL collective communication

For GPU collective benchmarks and validation, RCCL tests are pre-installed on compute nodes. A minimal all-reduce test across 2 nodes:

#!/bin/bash
#SBATCH --job-name=rccl-test
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --time=00:05:00

export NCCL_IB_QPS_PER_CONNECTION=2
export NCCL_BUFFSIZE=8388608
export UCX_NET_DEVICES=eno0

srun /opt/rccl-tests/all_reduce_perf -b 512M -e 8G -f 2 -g 1

---

For containerized jobs (Pyxis/Enroot, Apptainer) and environment modules, see Containers and Modules.