> For the complete documentation index, see [llms.txt](https://docs.tensorwave.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tensorwave.com/slurm/performance.md).

# Performance

### Network Topology

TensorWave GPU clusters are designed around two complementary networking optimizations that directly affect multi-GPU workload performance.

**Rail-optimized networking.** Each GPU has a dedicated RDMA NIC, giving it an exclusive high-bandwidth path for collective communications. With 8 GPUs per node, there are 8 independent rails (`rdma0`-`rdma7`), one per GPU. This means RCCL ring and tree algorithms can saturate all available bandwidth simultaneously without GPUs competing for shared NIC resources.

**Topology-aware scheduling.** Nodes are organized into physical pods, each sharing a top-of-rack network fabric. Slurm's tree topology plugin uses a `topology.conf` that maps nodes to pods and pods to a spine, allowing the scheduler to preferentially allocate nodes within the same pod for a given job. For multi-node jobs, this reduces cross-switch hops and keeps the majority of collective traffic on the lower-latency intra-pod fabric.

***

### RCCL All-Reduce Test

RCCL (ROCm Collective Communications Library) tests are pre-installed on compute nodes at `/opt/rccl-tests/`. The `all_reduce_perf` benchmark measures collective communication bandwidth across GPUs and is useful for validating interconnect performance and identifying nodes with degraded network throughput. A sample sbatch job driving `all_reduce_perf` is provided at `/opt/tw/examples/libexec/rccl.sbatch`.

#### Running the test

Create an output directory and submit the job:

```bash
sbatch /opt/tw/examples/libexec/rccl.sbatch
```

`**rccl.sbatch`:\*\*

```bash
#!/bin/bash
#SBATCH --job-name=rccl_tests
#SBATCH --output=jid-%j.name-%x.log
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --gpus-per-node=8
#SBATCH --time=01:00:00
#SBATCH --nodes=2

set -euxo pipefail

# Use 2 InfiniBand queue pairs per connection between ranks
export NCCL_IB_QPS_PER_CONNECTION=2

# Double buffer size for NCCL communications
export NCCL_BUFFSIZE=8388608

# Prevent MPI from using InfiniBand
export UCX_NET_DEVICES=eno0

srun /opt/rccl-tests/all_reduce_perf -b 512M -e 8G -f 2 -g 1
```

To run on more nodes, override the `--nodes` value at submission time:

```bash
sbatch --nodes <nnodes> /opt/tw/examples/libexec/rccl.sbatch
```

#### Script parameters

**Environment variables**

| Variable                     | Value      | Purpose                                                                                                   |
| ---------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- |
| `NCCL_IB_QPS_PER_CONNECTION` | `2`        | Increases InfiniBand queue pairs per connection, improving routing entropy and throughput.                |
| `NCCL_BUFFSIZE`              | `8388608`  | Sets the RCCL communication buffer to 8 MB. Larger buffers can improve performance at high message sizes. |
| `UCX_NET_DEVICES`            | `eno0`     | Directs UCX control traffic over Ethernet, leaving InfiniBand dedicated to RCCL data traffic.             |
| `NCCL_IB_GID_INDEX`          | `1` or `3` | Specifies which GID index RCCL should use, values are dependent on the NIC vendor of your cluster.        |

**RCCL test arguments**

| Argument | Value  | Description                                            |
| -------- | ------ | ------------------------------------------------------ |
| `-b`     | `512M` | Minimum message size                                   |
| `-e`     | `8G`   | Maximum message size                                   |
| `-f`     | `2`    | Step factor (doubles each step: 512M, 1G, 2G, ..., 8G) |
| `-g`     | `8`    | GPUs per process                                       |

#### Reading the output

A successful run completes without errors and shows increasing bus bandwidth as message size grows. Key fields in the output:

#### Results on an 4-node MI355X Cluster

```
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
   536870912     134217728     float     sum      -1   2945.0  182.30  353.20      0   2947.0  182.17  352.96      0
  1073741824     268435456     float     sum      -1   5452.6  196.92  381.54      0   5446.2  197.16  381.99      0
  2147483648     536870912     float     sum      -1    10806  198.72  385.03      0    10817  198.52  384.63      0
  4294967296    1073741824     float     sum      -1    21843  196.63  380.97      0    21846  196.60  380.91      0
  8589934592    2147483648     float     sum      -1    44986  190.95  369.96      0    44930  191.19  370.42      0
# Errors with asterisks indicate errors that have exceeded the maximum threshold.
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 374.161
#
# Collective test concluded: all_reduce_perf
```

* `**algbw**` — algorithm bandwidth: message size divided by time. Reflects how quickly one collective operation completes.
* `**busbw**` — bus bandwidth: `algbw` corrected for the number of ranks. Better reflects peak hardware utilization.
* `**#wrong**` — should be `0`. Any non-zero value indicates a data correctness error.
* `**Avg bus bandwidth**` — average `busbw` across all message sizes. Useful as a single summary figure for comparison.

A healthy cluster shows `busbw` increasing steadily with message size and leveling off at a stable peak at larger sizes. Nodes with degraded interconnect will show lower `busbw` or fail to complete. Target `busbw` values will be dependent on your cluster architecture.

***

### Training Benchmarks (MFU / TFLOPS)

Beyond raw interconnect bandwidth, it's often useful to measure end-to-end training performance — throughput (tokens/sec), compute utilization (TFLOPS), and Model FLOPs Utilization (MFU). For this, TensorWave recommends the [Primus Auto Benchmark tool](https://github.com/AMD-AGI/Primus/blob/main/tools/auto_benchmark/Primus_Auto_Benchmark_README.md) from AMD, which runs standardized model configs on the Megatron or TorchTitan backend and automatically generates a metrics summary table.

The tool runs inside the `rocm/primus` container on a single GPU node. It is interactive: after launching, it prompts you to choose a backend, auto-detects the GPU (MI300X / MI355X), and lets you select which model config(s) to run. Selected configs run sequentially, with logs and metrics written to the `results/` directory.

> **Note:** MFU is reported by the **TorchTitan** backend. The **Megatron** backend reports TPS, TFLOPS, memory utilization, and step time. Choose the backend based on the metric you need.

#### Running the benchmark

Because the tool runs interactively in Docker, first allocate a GPU node and open a shell on it. From a login pod, allocate a node and find its Slurm node name:

```bash
salloc -N 1 --gpus-per-node=8
squeue -u $USER
```

Then SSH to the allocated worker pod by its node name:

```bash
ssh <node-name>
# example:
ssh tus1-p1-g1
```

Pull the Primus container:

```bash
docker pull rocm/primus:v26.5
```

Launch the container:

```bash
docker run -it --rm \
  --gpus all \
  --network host \
  --ipc host \
  -v $HOME/.ssh:/root/.ssh \
  --name primus \
  rocm/primus:v26.5
```

Change into the auto-benchmark directory and launch the interactive tool:

```bash
cd /workspace/Primus/tools/auto_benchmark/
bash run_primus_autobenchmark.sh
```

Follow the on-screen prompts to select a backend, model config(s), and any parameter overrides. You will be asked for a HuggingFace token to pull model and tokenizer assets.

#### Reading the results

When all runs finish, the tool automatically generates a metrics table. Logs and the exact configs used are preserved for reproducibility under `/workspace/Primus/tools/auto_benchmark/results/logs_<backend>/`.

| Metric         | Megatron | TorchTitan | Meaning                                                  |
| -------------- | -------- | ---------- | -------------------------------------------------------- |
| **TPS**        | ✓        | ✓          | Tokens processed per second.                             |
| **TFLOPS**     | ✓        | ✓          | Achieved compute throughput.                             |
| **MFU**        | —        | ✓          | Model FLOPs Utilization — achieved vs. theoretical peak. |
| **Memory (%)** | ✓        | ✓          | Peak GPU memory utilization.                             |
| **Time (ms)**  | ✓        | —          | Per-step elapsed time.                                   |

***

### Preflight Smoke Test

`preflight` is a fast, self-contained health check meant to be dropped into the **top of a job** to efficiently validate hardware health before your real workload starts. It runs the pre-installed PyTorch environment at `/opt/venv` and exercises three things in under a minute:

1. **matmul** — every GPU runs a `bf16` GEMM; validates the result is finite and measures throughput (TFLOPs).
2. **intra-node RCCL** — an `all_reduce` within each node's local GPUs; measures intra-node bus bandwidth.
3. **inter-node RCCL** — an `all_reduce` across the full allocation; measures cross-node bus bandwidth. **Only runs when launched across more than one node** (see below).

Each measurement must clear two bars: a per-SKU **absolute floor** and a **peer-relative** check (no GPU/link may sit more than a configurable tolerance below the fleet median), which catches a single throttled GPU or degraded link regardless of hardware SKU. On any failure `preflight` exits non-zero, so a `set -e` job **aborts before wasting the allocation**.

The launcher lives at `/opt/tw/examples/bin/preflight` and a ready-to-copy job is provided at `/opt/tw/examples/libexec/preflight.sbatch`.

#### Injecting it into your job

Add a single line near the top of your sbatch script, before your workload. Because the script uses `set -e`, a failed preflight stops the job before your workload runs:

```bash
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=48

set -euo pipefail

# Preflight: validates GPUs + RCCL across the whole allocation, then exits.
srun --ntasks-per-node=1 /opt/tw/examples/bin/preflight

# --- your real workload runs only if preflight passed -----------------------
srun ... my_training_job
```

You do not need to configure `torchrun`, master addresses, or GPU counts — the launcher detects the allocation from the SLURM environment and bootstraps `torch.distributed.run` for you.

#### Supported launch configurations

| Invocation                                                               | Nodes exercised                          | Phases run             |
| ------------------------------------------------------------------------ | ---------------------------------------- | ---------------------- |
| `/opt/tw/examples/bin/preflight` (bare, in the batch script)             | the single node the batch script runs on | matmul + intra         |
| `srun --ntasks-per-node=1 /opt/tw/examples/bin/preflight` (1-node alloc) | 1                                        | matmul + intra         |
| `srun --ntasks-per-node=1 /opt/tw/examples/bin/preflight` (N-node alloc) | all N                                    | matmul + intra + inter |

Key rules:

* **Multi-node RCCL runs only under `srun` across >1 node.** A bare `preflight` invocation (or a single-node allocation) stays single-node and never hangs waiting for peers that will not join.
* **Launch with `--ntasks-per-node=1`.** The launcher itself spawns one worker per GPU on each node; you only need one launcher task per node.
* **Multi-node runs require full 8-GPU nodes.** A partial node makes the inter-node comparison meaningless, so `preflight` hard-fails a multi-node run if any node has fewer than 8 GPUs. Partial-GPU checks (e.g. 2 or 4 GPUs) are supported **single-node only**.

#### Options and configuration

Any arguments after the launcher are forwarded to the check. Useful flags:

| Flag                                                                         | Effect                                                                                                                     |
| ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--warn-only`                                                                | Report failures but always exit `0` (does not abort the job). Does **not** bypass the 8-GPU-per-node topology requirement. |
| `--json`                                                                     | Emit a machine-readable JSON summary instead of the human tables.                                                          |
| `--skip-matmul` / `--skip-intra` / `--skip-inter`                            | Skip a phase.                                                                                                              |
| `--matmul-size` / `--matmul-iters` / `--coll-bytes` / `--coll-iters`         | Override workload sizing.                                                                                                  |
| `--min-tflops` / `--min-intra-busbw` / `--min-inter-busbw` / `--outlier-tol` | Override thresholds at runtime.                                                                                            |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.tensorwave.com/slurm/performance.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.