> For the complete documentation index, see [llms.txt](https://docs.tensorwave.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tensorwave.com/slurm-old/running-jobs-with-modules.md).

# Running Jobs With Modules

Software modules are provided for commonly used software packages. 

#### Modules Quickstart

As an example, we provide a module for [Huggingface's Transformer Reinforcement Learning](https://github.com/huggingface/trl/tree/v0.28.0) package. So getting a working TRL environment is as easy as `module load trl`. We also provide a sample sbatch script that uses the TRL module `/opt/examples/libexec/trl-module.sbatch` :

{% code title="/opt/examples/libexec/trl-module.sbatch" lineNumbers="true" expandable="true" %}

```bash
#!/usr/bin/bash
#SBATCH --job-name=trl-finetuner
#SBATCH --output=jid-%j.name-%x.log
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=48
#SBATCH --gpus-per-node=8
#SBATCH --time=01:00:00
#SBATCH --nodes=4

set -exuo pipefail

module load trl

TRLFT_PY="/opt/examples/scripts/trl_tune/trl_tune.py"

GPUS_PER_NODE=8
MASTER_ADDR=$(hostname)
MASTER_PORT=6000

if [[ -d $HOME/.cache/huggingface/datasets/mlech26l___shell-helper ]]; then
  HF_OFFLINE=1
else
  HF_OFFLINE=0
fi

srun bash <<EOF

export MIOPEN_CUSTOM_CACHE_DIR=/tmp/miopen-cache
export MIOPEN_USER_DB_PATH=/tmp/miopen-user-db

export HF_DATASETS_OFFLINE=${HF_OFFLINE}
export HF_HUB_DISABLE_PROGRESS_BARS=1

MY_IP=\$(hostname)
IS_HOST=0
REMOTE_ADDR=${MASTER_ADDR}
if [ "\$MY_IP" == "${MASTER_ADDR}" ];then
 IS_HOST=1
 REMOTE_ADDR=localhost
fi

export OMP_NUM_THREADS=8

echo "MY IP is \$MY_IP and am I host? \$IS_HOST and what is master addr ${MASTER_ADDR}"
USE_ROCM=1

python -u -m torch.distributed.run \
 --nproc_per_node $GPUS_PER_NODE \
 --nnodes $SLURM_NNODES \
 --rdzv_endpoint \${REMOTE_ADDR}:${MASTER_PORT} \
 --rdzv_backend c10d \
 --max_restarts 0 \
 --rdzv_id=1 \
 --rdzv_conf=is_host=\$IS_HOST \
 --local_addr "\$(hostname)" \
 $TRLFT_PY

EOF
```

{% endcode %}

Line 12 (`module load trl`) loads the TRL software, and this is maintained through subsequent sub-shell calls, like the `srun bash`  on line 26.

You can test-run the script with `sbatch /opt/examples/libexec/trl-module.sbatch`. It runs accross 4 nodes by default, but you can scale it with `sbatch --nodes <num-nodes> /path/to/script.sbatch`.

{% code expandable="true" %}

```
$ sbatch /opt/examples/libexec/trl-module.sbatch
Submitted batch job 268
$ tail -f jid-268.name-trl-finetuner.log
++ hostname
+ MASTER_ADDR=tus1-p13-g2
+ MASTER_PORT=6000
+ [[ -d /home/tensorwave/.cache/huggingface/datasets/mlech26l___shell-helper ]]
+ HF_OFFLINE=1
+ srun bash
MY IP is tus1-p13-g2 and am I host? 1 and what is master addr tus1-p13-g2
MY IP is tus1-p14-g24 and am I host? 0 and what is master addr tus1-p13-g2
MY IP is tus1-p14-g37 and am I host? 0 and what is master addr tus1-p13-g2
MY IP is tus1-p16-g17 and am I host? 0 and what is master addr tus1-p13-g2
Rank: 0 out of 32
Number of gpus available: 8
  GPU 0: AMD Instinct MI325X
  GPU 1: AMD Instinct MI325X
  GPU 2: AMD Instinct MI325X
  GPU 3: AMD Instinct MI325X
  GPU 4: AMD Instinct MI325X
  GPU 5: AMD Instinct MI325X
  GPU 6: AMD Instinct MI325X
  GPU 7: AMD Instinct MI325X
Loading model: LiquidAi/LFM2.5-1.2B-Instruct

...

Found the latest cached dataset configuration 'default' at /home/tensorwave/.cache/huggingface/datasets/mlech26l___shell-helper/default/0.0.0/bf4e04b465240544350f49c89cd108c35698f588 (last modified on Sat Feb 14 08:59:42 2026).
Launching training
{'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'}
{'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'}
{'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'}
{'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'}
{'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'}
{'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'}
{'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'}
{'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'}k
```

{% endcode %}

#### Module Management

To explore available modules, run `module avail` or `module spider`. Avail gives a simplified output, while spider is more detailed and useful for sorting out module dependencies.

If you need a specific piece of software, contact us and we can provide a module that fits your needs.

#### Resources

Lmod user guide: <https://lmod.readthedocs.io/en/latest/010_user.html>&#x20;


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.tensorwave.com/slurm-old/running-jobs-with-modules.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.