# Introduction to ROCm ROCm, otherwise known as Radeon Open Compute, is an open source GPU Compute Framework that enables developers to customize their GPU software while collaborating with other developers. It consists of a variety of drivers, development tools, and API's that enables GPU programming from the low-level kernel to end-user applications. ROCm can be deployed in many ways, including through the use of containers such as Docker, Spack, and your own build from source. {% hint style="info" %} Learn more about ROCm [here](https://rocm.docs.amd.com/en/latest/what-is-rocm.html). {% endhint %} ## ROCm Quickstart Guide * [Linux](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html) # FAQ ***
What types of GPUs do you currently offer? At this time, we exclusively offer **AMD GPUs** for our cloud rental services. Our selection includes high-performance AMD GPUs that are well-suited for AI and machine learning applications, providing robust computational power and excellent performance.
Why do you only offer AMD GPUs? We have chosen to offer AMD GPUs due to their strong performance, cost-effectiveness, and unique features that are highly beneficial for AI workloads. AMD GPUs provide excellent value, reliability, and are particularly effective in handling parallel processing tasks essential for AI and machine learning. AMD GPUs also offer several advantages, including: * **Cost-Effectiveness:** AMD GPUs typically offer a better price-to-performance ratio, making them a cost-effective choice for high-performance computing. * **High Memory Bandwidth:** AMD GPUs are equipped with high memory bandwidth, which is crucial for handling large datasets and complex neural networks. * **Open Software Ecosystem:** AMD supports an open software ecosystem, including ROCm (Radeon Open Compute), which provides robust tools and libraries for AI and machine learning development. * **Scalability:** AMD GPUs are highly scalable, allowing for efficient parallel processing and seamless integration into large-scale AI projects.
Can I run my existing AI frameworks and tools on AMD GPUs? **Yes**, AMD GPUs support a wide range of AI frameworks and tools. Through the ROCm platform, you can use popular frameworks such as TensorFlow, PyTorch, etc. We provide pre-configured environments optimized for AMD GPUs to ensure smooth and efficient development.
How do AMD GPUs handle AI inference and training tasks? AMD GPUs are highly efficient for both AI inference and training tasks. Their high memory bandwidth and robust computational power make them well-suited for handling large datasets and complex models. The open software ecosystem also ensures compatibility with major AI frameworks, facilitating smooth and efficient development and deployment.
Can I access the GPUs directly (bare metal access)? What are the benefits? **Yes**, we provide bare metal access to our GPUs via SSH. This means you have full control over the hardware without any virtualization overhead, allowing for full control over the hardware environment, and superior performance since there is no hypervisor layer. This is particularly beneficial for sensitive AI applications and large-scale training tasks.
What support options are available if I encounter issues? We offer full customer support via email, chat, and phone. Our support team comprises experienced professionals who can assist with technical issues, billing inquiries, and general questions about our services.
What operating systems are supported on the GPUs? Our systems are preloaded with **Linux (Ubuntu)**. However, you can choose whichever OS that best fits your development environment and project needs.
Do you offer pre-configured AI frameworks and tools? **Yes**, we provide pre-configured environments with popular AI frameworks and tools such as TensorFlow, PyTorch, and Jupyter Notebooks. These environments are optimized for our GPU hardware, allowing you to start your projects quickly without extensive setup.
Are there any discounts for long-term rentals or large-scale projects? Yes, we offer volume discounts and flexible pricing plans for long-term rentals and large-scale projects. Please contact our sales team to discuss your specific needs and to get a customized pricing plan.
How do I handle data storage and transfer to and from the GPU instances? We provide multiple data storage and transfer options, including high-speed SSDs and cloud storage integrations. You can also upload your datasets directly to the GPU instances.
Why should I choose a reserved GPU contract over on-demand rental? Reserved GPU contracts offer several advantages, including guaranteed availability, more predictable costs, and better pricing. This is particularly beneficial for long-term projects or ongoing AI development needs, where consistent access to high-performance GPUs is crucial. On-demand rentals are best suited for short-term or urgent needs but come with higher costs and potential availability issues.
Can I change or upgrade my reserved GPU contract? Yes, you can upgrade or modify your reserved GPU contract to better suit your needs. Our customer support team is available to assist you with any changes, ensuring you have the right resources for your projects. Changes are subject to contract terms and availability.
What happens if I need to cancel my reserved GPU contract early? Early cancellation of a reserved GPU contract may incur a fee, depending on the terms of the contract. We understand that project needs can change, so we aim to offer flexible solutions. Please contact our support team to discuss your specific situation and find the best resolution.
When will on-demand GPU rental be available? We are planning to launch our on-demand GPU rental service soon. Stay tuned for announcements on our website and through our communication channels. On-demand rental will offer additional flexibility for short-term or urgent project needs.
How do I get the best pricing for my GPU needs? The best pricing is achieved through longer-term reserved GPU contracts. The longer the commitment, the more competitive the pricing. Additionally, we offer volume discounts for large-scale projects. Contact our sales team to discuss your specific requirements and receive a customized pricing plan.
# Out of the Box After purchase, you will receive instructions for a preconfigured environment that will allow you to easily transition into computing with the MI300X. ## Out of the box, you will receive: * Initially configured system with Linux (Ubuntu) * Preinstalled libraries, system utilities, developmental tools, and packages * Installed YAML processor * Installed AMD GPU drivers and ROCm stack as well as ROCm system configurations * PyTorch with ROCm acceleration * Setup of the machine learning framework * Setup of the user environment * System hardening services for improved security * Network configurations # Bare Metal Quickstart *** Your node has Linux (Ubuntu) OS. It comes pre-loaded with some tools to simplify your set-up process. ## Connecting to Your Node When your bare metal node is ready, you will be provided its username and IP address. To connect to your node, use the following command on a device with one of the SSH keys: ```bash ssh [username]@[ip_address] ``` This command will be your primary method of accessing and managing your node. *** ## Node Basics Your node comes with Ubuntu 22.04 LTS and ROCm. All SSH keys you've initially provided will have root user access. ### Adding More SSH Keys In the event that you would like to provide more users SSH access to your node, follow these steps: 1. Copy your key, which should be structured like: ``` ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDxAZn... user@host ``` 2. SSH into your node using: ```bash ssh [username]@[ip_address] ``` 3. Open your authorized keys file using: ```bash nano /home/[username]/.ssh/authorized_keys ``` 4. Paste your key at the bottom of the file 5. Save and exit 6. Restart the sshd service using: ```bash sudo systemctl restart sshd ``` ### Monitoring Your GPUs To get more information on your GPUs, use `amd-smi` in your terminal. To continue to monitor this, you can run `watch -n 0.5 amd-smi`. This will provide information on both IDs and usage in intervals of 0.5 seconds, as shown below: ``` +------------------------------------------------------------------------------+ | AMD-SMI 26.2.0+021c61fc amdgpu version: 6.16.6 ROCm version: 7.1.1 | | VBIOS version: 00160474 | | Platform: Linux Baremetal | |-------------------------------------+----------------------------------------| | BDF GPU-Name | Mem-Uti Temp UEC Power-Usage | | GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage | |=====================================+========================================| | 0000:05:00.0 AMD Instinct MI325X | 0 % 26 °C 0 126/1000 W | | 0 1 6 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:15:00.0 AMD Instinct MI325X | 0 % 26 °C 0 126/1000 W | | 1 3 7 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:65:00.0 AMD Instinct MI325X | 0 % 27 °C 0 133/1000 W | | 2 2 5 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:75:00.0 AMD Instinct MI325X | 0 % 26 °C 0 132/1000 W | | 3 0 4 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:85:00.0 AMD Instinct MI325X | 0 % 26 °C 0 124/1000 W | | 4 5 2 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:95:00.0 AMD Instinct MI325X | 0 % 27 °C 0 126/1000 W | | 5 7 3 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:e5:00.0 AMD Instinct MI325X | 0 % 25 °C 0 126/1000 W | | 6 6 1 SPX/NPS1 | 0 % N/A 283/262128 MB | |-------------------------------------+----------------------------------------| | 0000:f5:00.0 AMD Instinct MI325X | 0 % 28 °C 0 120/1000 W | | 7 4 0 SPX/NPS1 | 0 % N/A 283/262128 MB | +-------------------------------------+----------------------------------------+ +------------------------------------------------------------------------------+ | Processes: | | GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % | |==============================================================================| | 0 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 0 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 1 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 1 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 2 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 2 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 3 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 3 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 4 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 4 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 5 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 5 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 6 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 6 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 7 8645 N/A 0.0 B 0.0 B 0.0 B 0.0 % | | 7 543182 N/A 0.0 B 0.0 B 0.0 B 0.0 % | +------------------------------------------------------------------------------+ Process Name may require elevated permissions. ``` ### Downloading and Uploading Files To download files from your server, use the `scp` command: ```bash // For individual files scp [username]@[ip_address]:/path/to/remote/file /path/to/local/destination // For subdirectories scp -r [username]@[ip_address]:/path/to/remote/directory /path/to/local/destination ``` To upload files to your server, you may also use the scp command: ```bash // For individual files scp /path/to/local/file [username]@[ip_address]:/path/to/remote/destination // For subdirectories scp -r /path/to/local/file [username]@[ip_address]:/path/to/remote/destination ``` ### Accessing Remote Services Locally Often times, you will find yourself needing to access a service being exposed on your remote server, locally. To do so, use the following command: ```bash ssh -L local_port:remote_host:remote_port [username]@[ip_address] ``` For example, let's assume you want to access a Jupyter Notebook that you've exposed on port 8888. From your **local** command line, you'll want to use the following: ```bash ssh -L 8888:localhost:8888 [username]@[ip_address] ``` Then, you can access this in your browser at [http://localhost:8888
](). *** # PyTorch Quickstart Estimated time: 2 minutes, 3 minutes with buffer *** ### Using ROCm Devices PyTorch is officially supported by AMD for ROCm, and should be plug-and-play once set up correctly. {% hint style="info" %} Learn more about installing PyTorch with ROCm [here](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/3rd-party/pytorch-install.html) {% endhint %} AMD GPU devices are configured and accessed the *exact* same way as NVIDIA GPU devices. This means that any workflow that sets the PyTorch device the following way will work out-of-the-box, assuming PyTorch can detect your GPUs: ```python torch.device("cuda") ``` *** ### Debugging In order to test whether your system is configured to use PyTorch with GPU acceleration, begin by starting a new file to run a couple of debugging commands: 
mkdir pytorch-hello-world
cd pytorch-hello-world
nano debug.py
The following code will return a boolean indicating whether your GPUs are being detected by PyTorch: 
import torch
print(torch.cuda.is_available())
Now, go ahead and run your file using: ```bash python3 debug.py ``` In the event that this does *not* return `True`, there are a couple things you must check. #### PyTorch Setup One reason the above command may not function properly is that the incorrect version of PyTorch is installed. To check, add the following line to your debugging file: ```python print(torch.__version__) ``` You should get an output similar to: ``` [torch_version]a0+git[hash] ``` Or: ``` [torch_version].dev[date]+rocm[rocm_version] ``` If this output is not a ROCm-enabled PyTorch build, you must reinstall PyTorch with the correct version. One way to do this would be: ```bash pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1/ ``` #### Checking ROCm Setup To ensure ROCm is properly configured, run the following command: ```bash rocm-smi ``` The output should be similar to (depending on your number of devices): ``` ========================================= ROCm System Management Interface ========================================= =================================================== Concise Info =================================================== Device [Model : Revision] Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% Name (20 chars) (Junction) (Socket) (Mem, Compute) ==================================================================================================================== 0 [0x74a1 : 0x00] 45.0°C 142.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 1 [0x74a1 : 0x00] 42.0°C 135.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 2 [0x74a1 : 0x00] 42.0°C 137.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 3 [0x74a1 : 0x00] 48.0°C 141.0W NPS1, SPX 138Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 4 [0x74a1 : 0x00] 46.0°C 142.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 5 [0x74a1 : 0x00] 40.0°C 137.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 6 [0x74a1 : 0x00] 47.0°C 142.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X 7 [0x74a1 : 0x00] 42.0°C 132.0W NPS1, SPX 132Mhz 900Mhz 0% auto 750.0W 0% 0% AMD Instinct MI300X ==================================================================================================================== =============================================== End of ROCm SMI Log ================================================ ``` If this is not the case, ROCm is not properly installed. You will more likely, however, have issues running the following command: ```bash rocminfo ``` The output should be of the format: 
ROCk module version 6.7.0 is loaded
=====================    
HSA System Attributes    
=====================    
....
If this command errors, it's most likely that devices are not properly mounted, or your user is not a part of the `render` group. *** ### Teardown Navigate back to your base directory and remove your `pytorch-hello-world` folder: ```bash cd ~ rm -rf pytorch-hello-world/ ``` # Docker Quickstart Estimated time: 5 minutes, 6 minutes with buffer. *** ### Pulling a Docker Image Your node comes with Docker Engine installed, so all Docker functionality should be available to you on your first connection. Begin by pulling the desired image: ```bash docker pull tensorwavehq/hello_world:latest ``` You can verify that your image was properly pulled by running the following command and checking for your desired image: ```bash docker images ``` If the pull was successful, your output should look similar to this: ``` REPOSITORY TAG IMAGE ID CREATED SIZE tensorwavehq/hello_world latest 359e600f7aac 2 minutes ago 61.2GB ``` {% hint style="info" %} TensorWave's officially supported images can be found [here](https://hub.docker.com/u/tensorwavehq). {% endhint %} *** ### Running a Docker Container In order to run your Docker containers with GPU acceleration, you must mount the devices. For certain applications, you must also add the container to a group to utilize your GPUs. #### Using the docker run Command Here's an example command to mount the devices and configure the correct permissions: ```bash docker run --device /dev/kfd --device /dev/dri --group-add video tensorwavehq/hello_world:latest ``` The usage of each option is as follows: * `--device /dev/kfd` * This command mounts the main compute interface to your container. * `--device /dev/dri` * This command mounts the Direct Rendering Interface for your GPU. To restrict access, append `/renderD`, where the node is the ID of the node you want to mount. * `--group-add video` (optional) * This command adds your container to the server's `video` group, which is necessary for certain applications (including PyTorch). #### Using docker-compose The following is an equivalent docker-compose to the command above: ```yaml version: '3' services: hello_world: image: tensorwavehq/hello_world:latest devices: - /dev/kfd - /dev/dri group_add: - video ``` To use it, create a `docker-compose.yml` file in any subdirectory, and within that subdirectory, run: ```bash docker compose up ``` #### Verifying Setup If done properly, the output of either the run or compose command should be similar to: ``` CUDA available: True Number of GPUs: 8 GPU 0: AMD Instinct MI300X GPU 1: AMD Instinct MI300X GPU 2: AMD Instinct MI300X GPU 3: AMD Instinct MI300X GPU 4: AMD Instinct MI300X GPU 5: AMD Instinct MI300X GPU 6: AMD Instinct MI300X GPU 7: AMD Instinct MI300X ``` For other containers, to verify that your Docker container has access to your GPUs, run **both** `rocm-smi` and `rocminfo`. These commands will reveal information about the GPUs mounted to your container. {% hint style="warning" %} If one or both of these commands fails to execute successfully, please double check your running commands. {% endhint %} *** # Kubernetes Quickstart Estimated time: 6 minutes, 8 minutes with buffer. *** ### Installing Tooling We'll start by installing kubectl and k3d. kubectl is a command-line tool for managing kubernetes interfaces, and k3d is a lightweight wrapper to run k3s in Docker. Download and install the lateset release of kubectl using the folllowing commands: ```bash curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl ``` Next, install the latest release of k3d using this command: ```bash wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash ``` {% hint style="info" %} Learn more about installing kubectl [here](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/), and installing k3d [here](https://k3d.io/v5.7.3/#installation). {% endhint %} *** ### Creating and Configuring a Cluster Now, you must create a cluster: ```bash k3d cluster create hello-world-cluster ``` Then go ahead and check your context: ```bash kubectl config current-context ``` This should list the cluster you just created, but if not, run the following command to switch to the needed context: ```bash kubectl config use-context k3d-hello-world-cluster ``` In order to operate with acceleration, Kubernetes must also be set up with the AMD GPU Operator and Labeler plugins. You can install these using the following commands: ```bash kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml ``` Then, go ahead and make your deployment manifest. Create a directory for your manifest and direct into it, then open up a `deployment.yaml` file. ```bash mkdir k8s-hello-world cd k8s-hello-world nano deployment.yaml ``` From there, paste in the following yaml: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: hello-world spec: replicas: 1 selector: matchLabels: app: hello-world template: metadata: labels: app: hello-world spec: containers: - name: hello-world image: tensorwavehq/hello_world:latest resources: limits: amd.com/gpu: 1 volumeMounts: - name: dev-kfd mountPath: /dev/kfd - name: dev-dri mountPath: /dev/dri securityContext: runAsGroup: 110 volumes: - name: dev-kfd hostPath: path: /dev/kfd - name: dev-dri hostPath: path: /dev/dri ``` You'll notice there are a few extra configurations we added. These are necessary to running the pod with GPU acceleration. * ```yaml resources: limits: amd.com/gpu: 1 ``` * This specifies that the container requires 1 AMD GPU. You must explicitly request GPU resources so that Kubernetes can schedule the pod on a node with an available AMD GPU. * ```yaml volumes: - name: dev-kfd hostPath: path: /dev/kfd - name: dev-dri hostPath: path: /dev/dri ``` * These definitions allow the cluster to use the necessary volumes from the host for utilizing the AMD GPUs. * ```yaml volumeMounts: - name: dev-kfd mountPath: /dev/kfd - name: dev-dri mountPath: /dev/dri ``` * These mounts correspond to the above volumes, allowing the container to access the GPU hardware. * ```yaml securityContext: runAsGroup: 110 ``` * This security context runs the containers in the pod as the group ID 110, the render group, which is necessary for PyTorch to detect the devices properly (PyTorch is used in the hello world container). Continue by applying the manifest using the following: ```bash kubectl apply -f deployment.yaml ``` This should take a few minutes to create the container. You can monitor the status here: ```bash kubectl get pods -l app=hello-world ``` Once this output displays that `STATUS` is `Completed`, you're ready to check output. Running: ```bash kubectl logs -l app=hello-world ``` Should give an output of: ``` CUDA available: True Number of GPUs: 1 GPU 0: AMD Instinct MI300X ``` You'll notice that you only have one GPU. That's because, as covered earlier, we specified a resource limit of one. You may raise or lower this number as necessary. *** ### Teardown Navigate back to your base directory and remove your `k8s-hello-world` folder: ```bash cd ~ rm -rf k8s-hello-world/ ``` # Hugging Face Quickstart Estimated time: 7 minutes, 9 minutes with buffer. *** Hugging Face is an AI/ML platform for the entire model pipeline. For this quickstart, we'll walk you through accelerated inference using a pretrained model. {% hint style="info" %} Learn more about Hugging Face [here](https://huggingface.co/). {% endhint %} *** ### Installing Dependencies Because PyTorch with ROCm comes preloaded on your device, you will not need to install this dependency. However, you will still need a couple of libraries in order to run our quickstart script. Begin by installing [transformers](https://huggingface.co/docs/transformers/en/index) using the following command: ```bash pip install transformers ``` This should take no more than a few minutes. *** ### Creating and Running Inference Script Next, go ahead and create and navigate to a new directory to create your script in: ```bash mkdir hf-hello-world cd hf-hello-world ``` Then, create a new script: ```bash nano hello-world.py ``` Within this script, paste the following code and exit: ```bash import torch from transformers import AutoModelForCausalLM, AutoTokenizer import time # Load model without quantization tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m") # Move model to GPU model = model.to("cuda") # Input text print("Warming up model...") input_text = "Hello, my name is" inputs = tokenizer(input_text, return_tensors="pt").to("cuda") warmup = model.generate(**inputs, max_new_tokens=20) print("Preparing text...") input_text = "According to all known laws of aviation, there is no way that a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course, flies anyway because" inputs = tokenizer(input_text, return_tensors="pt").to("cuda") print("Starting inference...") start = time.time() outputs = model.generate( **inputs, max_new_tokens=50, do_sample=True, temperature=0.7, top_k=50, top_p=0.95, no_repeat_ngram_size=2 ) t = time.time()-start print(f"inference time: {t}") print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` After doing so, you may run the script using the following: ```bash python3 hello-world.py ``` This runs a small model on one GPU, but feel free to swap out your model and prompts to your liking, then map to the proper devices. The output should be similar to: ``` Warming up model... Preparing text... Starting inference... inference time: 0.4770219326019287 According to all known laws of aviation, there is no way that a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course, flies anyway because it can. Well, if you fly with a little fat of your body, you can fly pretty damn well. You just have to be careful. ``` *** ### Teardown Navigate back to your base directory and remove your `hf-hello-world` folder: ```bash cd ~ rm -rf hf-hello-world/ ``` # Clusters ### Cluster Observability The Health tab gives you a single-pane view of your entire cluster's operational state, covering GPU nodes, networking interfaces, SLURM, storage, and logs. It's designed to surface problems immediately so your team can act fast without having to dig through multiple tools. At scale, visibility isn't optional. A single degraded node, a full storage volume, or a flapping network interface can silently stall a training run for hours if you don't catch it early. TensorWave's cluster observability is built around the idea that you should know about problems before they impact your workloads, not after. #### Nodes The Nodes section displays a grid of live status cards covering every category of node in your cluster and their current up/down state. **GPU Nodes** * **Up Cluster GPU Nodes** nodes that are online and available for workloads * **Down Cluster GPU Nodes** nodes that are currently unavailable. Expanding this card shows the hostname of each affected node * **Non-RMA GPU Nodes** nodes that have not been flagged for Return Merchandise Authorization, meaning they are healthy and not being serviced * **RMA GPU Nodes** nodes that have been flagged for RMA and are out of rotation for hardware servicing * **Nodes with 8 GPUs** nodes running a full complement of 8 GPUs * **Nodes with less than 8 GPUs** nodes that are online but have fewer than 8 GPUs available, which may indicate a GPU has failed or been taken offline * **Nodes Missing GPUs** nodes that are online but have no GPUs detected at all Tracking partial GPU availability matters. A node that appears healthy but is missing GPUs can silently reduce the compute available to your jobs, causing slower runs or unexpected failures without an obvious cause. **Head Nodes** Head nodes manage job scheduling and act as the primary entry point for SLURM workloads on your cluster. * **Up Cluster Head Nodes** head nodes that are online and operational * **Down Cluster Head Nodes** head nodes that are currently unavailable, with hostnames listed on expansion **Jump Nodes** Jump nodes serve as secure access points into your cluster environment. * **Up Cluster Jump Nodes** jump nodes that are online and reachable * **Down Cluster Jump Nodes** jump nodes that are currently unavailable, with hostnames listed on expansion **Frontend and Backend Interfaces** * **Up/Down Frontend Interfaces** frontend network interfaces handling inbound traffic to the cluster * **Up/Down Backend Interfaces** backend network interfaces handling internal cluster communication, including high-speed interconnects between nodes Network interface health is especially important for distributed workloads. Degraded or down backend interfaces can bottleneck inter-node communication and tank GPU utilization across your entire cluster, even when the nodes themselves appear healthy. Expanding any down card will list the hostnames of affected nodes, making it easy to pinpoint and isolate problems immediately. #### Kubernetes A summary of your Kubernetes environment at a glance, scoped to the selected namespace. **Node Health** shows the count of ready and not-ready nodes alongside an overall readiness percentage and total node count. **Pod Status** displays the total number of pods running across the cluster, broken down by state (Running, Succeeded, and others) and visualized as a live donut chart. **Deployments** shows how many deployments are available out of the total, with a clear indicator when all deployments are healthy. #### SLURM **Slurm Node States** gives you a breakdown of all SLURM nodes by their current state, including Idle, Allocated, Mixed, Down, and Drained/Draining. A progress bar shows how many nodes are active relative to the total. Keeping an eye on node states helps you spot scheduling bottlenecks early, particularly when nodes are unexpectedly Down or stuck in a Drained state that reduces the capacity available to your jobs. **Slurm Job Queue** shows the total number of jobs currently in the queue, broken down by Running, Pending, and Other states. A growing Pending count can be an early indicator of scheduler pressure, resource contention, or a node issue that is quietly reducing cluster capacity. #### Storage Keeping an eye on storage is critical for uninterrupted workloads. Full or near-full volumes can cause training jobs to fail, crash running pods, or corrupt checkpoints mid-run. The storage cards are designed to help you catch these issues before they become outages. **Shared Storage Used** shows current utilization of your shared storage volume as a percentage, with a live progress bar and the mount path for reference. Monitoring this regularly helps ensure your jobs always have the space they need to write outputs, logs, and model checkpoints. **Storage Volumes Used above 85%** flags any storage volumes approaching capacity. Catching volumes at this threshold gives you enough runway to free up space or expand capacity before a full volume takes down a workload. If all volumes are healthy, this card confirms that none have crossed the 85% threshold. #### Logs **Log Trend** shows a rolling view of log activity over the last 30 minutes. Spikes in log volume are often the earliest signal of something going wrong across the cluster, whether that's a failing job, a misbehaving pod, or an infrastructure event, giving you a chance to investigate before the impact is felt. # Nodes ### Node Insights Clicking into any node from the Nodes table opens a detailed view of that node's performance and health. The **Insights** tab provides deep telemetry across GPU and system-level metrics, giving you the data you need to understand exactly what's happening on a node at any point in time. TensorWave believes that the teams running the most demanding AI workloads deserve the same depth of observability that was previously only available to hyperscalers. Node Insights puts that data directly in your hands, so you can stop guessing why a job underperformed, catch hardware degradation before it causes a failure, and make informed decisions about your infrastructure without opening a support ticket first. Use the **insights category** dropdown to switch between GPU Insights and System Insights, and the **time range** dropdown to adjust the window of data displayed. *** #### GPU Insights GPU Insights surfaces per-GPU telemetry across compute, memory, power, and interconnect health. This is the primary view for understanding how your GPUs are performing under a workload and catching hardware issues before they cause job failures. At scale, even a single GPU behaving unexpectedly can silently degrade the performance of an entire training run, making this level of per-device visibility critical. **Summary cards** at the top of the view show: * **GPUs** total number of GPUs on the node * **Allocated GPUs** how many GPUs are currently allocated to a job * **Total ECC Counts** correctable and uncorrectable ECC memory errors detected across all GPUs. Uncorrectable ECC errors in particular can indicate failing GPU memory and should be investigated promptly **Compute and memory** * **Jobs by GPU Usage** shows utilization percentage per GPU, making it easy to spot underutilized or completely idle GPUs during an active workload. Low utilization on an allocated GPU is often the first sign of a bottleneck elsewhere in your pipeline * **GPUs VRAM Used** current VRAM consumption per GPU in MB * **Node GPU Usage** a time-series chart of GPU utilization across all GPUs on the node over the selected time range * **Used VRAM per GPU (%)** time-series view of VRAM consumption per GPU, useful for spotting memory pressure or unexpected growth during a run. Catching VRAM exhaustion early can save you from an out-of-memory crash mid-training **Power** * **Package Power Usage** current power draw per GPU in watts, with a time-series sparkline for each. Unexpected drops in power draw can indicate a GPU has gone idle or been taken offline mid-job * **GPU Power (W)** a combined time-series chart of power consumption across all GPUs, making it easy to correlate power behavior with workload activity * **Power-Activity Delta (%)** tracks the relationship between power draw and GPU activity. A large delta can indicate a GPU is drawing power without doing useful work, which may signal a hardware or driver issue. This metric is particularly useful for identifying silent GPU failures that don't surface as outright errors **Thermals** * **Memory Temperature (°C)** per-GPU memory temperature over time. Sustained high memory temperatures can throttle performance and shorten hardware lifespan * **GPU Sensor Temperatures (°C)** per-GPU sensor temperature readings over time, useful for identifying thermal outliers across the node. A GPU running consistently hotter than its peers is worth investigating before it causes a thermal throttle or hardware fault mid-run * **GPU Memory Clock (MHz)** memory clock frequency per GPU over time. Drops in memory clock speed can indicate thermal throttling that is silently reducing your effective compute throughput * **GPU System Clock (MHz)** core clock frequency per GPU over time **PCIe health** * **PCIe Counts** tracks Recovery, Replay, Replay Rollover, NACK Received, and NACK Sent events across the PCIe bus. Non-zero values here, especially Replay or NACK counts, can indicate PCIe instability that may affect GPU-to-CPU communication and overall node reliability * **PCIe Errors (ops/s)** rate of PCIe errors over time * **PCIe Bandwidth (MB/s)** per-GPU PCIe bandwidth over time, useful for identifying bottlenecks between GPUs and the host system **Interconnect** * **xGMI Transmission Rate (GB/s)** tracks data transmission rates across the GPU interconnect fabric. Degraded xGMI throughput can bottleneck multi-GPU communication and significantly reduce training efficiency on interconnect-heavy workloads. For large distributed runs, interconnect health is often the difference between hitting peak throughput and leaving performance on the table *** #### System Insights System Insights provides host-level telemetry covering storage, memory, CPU, and networking for the node. GPU problems don't always start with the GPU. Storage contention, CPU bottlenecks, memory pressure, and network instability can all silently degrade workload performance in ways that are difficult to diagnose without this level of system-level visibility. **Storage** * **Root FS Storage % Used** current utilization of the root filesystem as a percentage, with a live progress bar * **Filesystem Available Space Over Time** a time-series chart of available space across all mounted filesystems on the node. Watching this trend over time helps you catch storage being consumed faster than expected before a full filesystem crashes a job or corrupts a checkpoint **Memory** * **Memory Used** current system memory utilization as a percentage * **Memory Usage Over Time** time-series breakdown of RAM Cache and Buffer, RAM Free, RAM Used, and SWAP Used, giving you a full picture of how system memory is being consumed and whether the node is under memory pressure. Heavy SWAP usage in particular can be an early warning sign of a memory leak or an undersized allocation for your workload **CPU** * **Number of CPUs** total CPU count on the node * **Sys Load** current system load as a percentage * **CPU 1 min load avg** short-term CPU load average, useful for catching sudden spikes * **CPU 5 min load avg** medium-term load average, smoothing out short bursts to show sustained load * **CPU 15 min load avg** long-term load average, the best indicator of whether the node is consistently under pressure over time. A persistently high 15-minute average often points to a systemic issue rather than a transient spike **Thermals** * **CPU Temperatures (°C)** per-CPU temperature over time. Thermal throttling on the CPU can indirectly impact GPU workloads by creating bottlenecks in data preprocessing or job orchestration * **NIC Temperatures (°C)** per-NIC temperature over time. Overheating network interface cards can degrade throughput and contribute to link instability **Networking** * **Network Traffic Over Time** inbound and outbound traffic on the node over the selected time range * **Link Flaps Over Time** tracks network link flap events, where a network interface briefly goes down and comes back up. Frequent link flaps are a strong early indicator of a failing NIC or unstable network connection and should be investigated before they impact distributed workloads. For multi-node training jobs, a single flapping link can stall an entire run # Slurm # Kubernetes TensorWave's managed Kubernetes offering is designed to make it seamless to run and scale your clusters without the operational overhead. The Kubernetes tab gives you complete, real-time visibility into every layer of your cluster, from node health and pod status to deployments and scheduled workloads. Whether you're scaling a training run, debugging a degraded node, or auditing what's running across namespaces, everything you need is surfaced in one place without touching the terminal. This level of visibility means your team spends less time diagnosing and more time shipping. Problems are easier to catch early, incidents are faster to resolve, and you always have a clear picture of your cluster's state at any scale. #### Namespace filter The **Namespace** dropdown at the top of the page filters all resource views simultaneously. Select a specific namespace to narrow your focus, or choose **All Namespaces** for a full picture of everything running across your cluster. #### Overview The Kubernetes overview gives you an instant read on the health of your entire cluster across three summary cards: **Node Health** shows the number of ready and not-ready nodes, overall readiness percentage, and total node count at a glance. **Pod Status** provides a real-time breakdown of all pods running across the cluster, categorized by state (e.g. Running, Succeeded) and visualized as a live donut chart. **Deployments** shows total deployments and how many are available, with a clear healthy/degraded indicator so you can confirm your workloads are running as expected. #### Nodes Drill into the Nodes view for a granular per-node breakdown, no kubectl required. **Summary cards** at the top surface the metrics that matter most: * **Ready** nodes in a healthy, schedulable state * **Not Ready** nodes that are unavailable or degraded * **Control Plane** number of control plane nodes managing cluster state * **Workers** number of worker nodes available for workloads **Node table** breaks down every node in the cluster: * **Hostname** node identifier, with one-click copy * **Status** live node state (e.g. Ready, Unknown) * **Role** whether the node is a `worker` or `control-plane` * **Version** Kubernetes version running on the node * **GPU** number of GPUs attached, where applicable * **CPU** total CPU count * **Memory** total memory provisioned * **More Info** expands to show the node's internal IP, OS image, kernel version, and container runtime #### Pods, Deployments, and other resources The same namespace-scoped table interface is available for Pods, Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs, giving you consistent visibility across every layer of your Kubernetes workloads without leaving the console. #### Kubeconfig Download your cluster's kubeconfig directly from the cluster header by clicking **Kubeconfig**. Use it to connect kubectl or any Kubernetes-compatible tooling to your cluster instantly. *** # Notifications & Alerts TensorWave supports notifications via **Slack** and **Email** to help you stay on top of your infrastructure without having to watch a dashboard. Whether it's a storage threshold being crossed or an unexpected node reboot, alerts are designed to surface problems early — before they impact your workloads. Alerts are built around cluster and node observability, giving your team visibility into the health of your infrastructure in real time. Slack alerts are configured at the organization level by admins, while email alerts are configured individually by each user. *** ### Slack Alerts Slack alerts are organization-wide and can only be managed by org admins. When enabled, qualifying events are sent to your organization's connected Slack workspace. #### Enabling Slack Alerts Toggle **Enable Slack alerts** on to activate notifications. Use the workspace dropdown to select which Slack workspace should receive alerts. #### Alert types You can enable or disable individual alert types depending on what's relevant to your organization. These include events such as low storage on nodes, low shared storage, and node reboots, among others. *** ### Email Alerts Email alerts are per-user. Each team member independently controls which notifications are delivered to their account email address. #### Enabling Email Alerts Toggle **Enable email alerts** on to activate notifications. Alerts are sent to the email address on your profile. To update this address, go to **Profile** settings. #### Alert types The same set of alert types is available for email as for Slack. Each can be toggled on or off independently based on your preferences. ***
### General Notes * Only org admins can enable or modify Slack alerts. * Slack and email alerts are independent and can be enabled in any combination. * Individual alert types can be toggled without disabling alerts entirely. * Alert thresholds are set to TensorWave's recommended defaults. # Slurm Overview TensorWave Managed Slurm is a production Slurm cluster deployed on the same Kubernetes infrastructure as your other GPU workloads. TensorWave operates and supports the cluster; your teams continue to use the same commands they always have: `sbatch`, `squeue`, `salloc`, and `srun`. We manage the Slurm control plane, compute and login images, networking and storage integration, and upgrades, removing the need to run a separate Slurm fleet alongside your Kubernetes cluster. *** ### Slurm on Kubernetes Many organizations use **Kubernetes** for services, inference, CI, and other cloud-native workloads, and **Slurm** for large batch jobs, MPI, and distributed training. These are complementary scheduling models, not alternatives to each other. The traditional approach requires operating them as two separate systems: independent node imaging pipelines, separate monitoring stacks, and custom integration between a Kubernetes fleet and a standalone Slurm deployment running on the same physical hardware. **Slurm on Kubernetes** eliminates that separation. The Slurm cluster is deployed on the same Kubernetes cluster as your other workloads. Batch users continue to submit jobs through the standard Slurm interface. Kubernetes is not replacing Slurm for batch scheduling. It is the deployment and operations layer for the Slurm controller, login pods, and workers, using the same storage, secrets, and observability patterns applied to everything else on the cluster. | | Standalone Slurm | Slurm on Kubernetes | | --------------------- | ------------------------------------------------- | -------------------------------------------------------------- | | **Compute** | Separate machines or OS images outside Kubernetes | Slurm daemons running in a Kubernetes Pod on GPU nodes | | **Service workloads** | Separate cluster | Same cluster, via Kubernetes | | **Node lifecycle** | Bare-metal images and manual intervention | Workers as versioned container images, managed by the operator | | **Failure handling** | Custom scripts and manual drain procedures | Operator reconciliation integrated with Slurm drain and resume | | **Observability** | Separate tooling from the rest of the cluster | Shared metrics and logging pipelines | Batch and research users retain the Slurm interface they know. Platform teams retain Kubernetes for the workloads that belong there. *** ### How it works The Slurm cluster consists of five components, each running as a workload on Kubernetes: **Controller.** Runs `slurmctld`, which handles job submissions, scheduling, and node state. Configuration is distributed to clients automatically so login and compute pods always share a consistent `slurm.conf`. **Login pods.** The entry point for SSH sessions and interactive use. Users submit jobs, inspect the queue, and (where policy permits) connect to workers allocated to their jobs. **Worker pods.** Run `slurmd` on GPU and CPU hardware. Each worker is a container with the devices and network fabric the job requires, including RDMA NICs, local scratch, and access to shared storage. **Operator.** Reconciles a declarative cluster specification (Kubernetes custom resources) against the live state of the deployment. Changes to capacity, images, or partitions are applied by updating the specification; the operator handles the rest. **Storage.** Shared filesystems (including /home for users and /mnt/\* for system resources) are mounted on both login and worker pods so that paths are consistent at submission time and at runtime. Worker scratch storage is ephemeral and sized for container runtimes and job-local I/O. From a user perspective, `sinfo` and `scontrol` report standard Slurm node names. Operationally, workers are pods, running with host networking and the privileged device access that GPUs and RDMA require. *** ### What TensorWave manages You receive a fully operational Slurm on Kubernetes deployment without building or maintaining the integration yourself. TensorWave provides: * **Compute and login images.** Slurm 25.x, PMIx MPI, the GPU software stack, and (where enabled) Pyxis/Enroot and Apptainer, built and validated against your hardware profile. * **Access.** LDAP-backed SSH with optional per-user login pods, and documented procedures for worker access within your security policy. * **Accounting.** Job and resource usage records are stored via `slurmdbd` backed by a database. GPU usage is tracked per job, and associations, limits, and QOS policies are enabled. * **Storage.** `/home` and shared module paths mounted consistently across login and compute pods. * **Health checks.** Automated prolog and epilog checks, scheduled checks for deeper validation, and additional tools for node diagnostics. * **Observability.** Slurm metrics (jobs, nodes, partitions, scheduler) are pushed to the TensorWave platform, where they are available in a Slurm dashboard. This covers job queue depth, node states, GPU utilization per job, and scheduler activity without any additional setup on your end. * **Lifecycle management.** Coordinated upgrades, partition configuration, and change communication so maintenance does not catch users off guard. *** ### What stays the same If your team already uses Slurm, the job submission interface is unchanged: * Job submission: `sbatch`, `srun`, `salloc` * Cluster inspection: `sinfo`, `squeue`, `scontrol` * MPI workloads, container-based jobs, and environment modules where installed For standard Slurm behavior and command reference, the [official Slurm documentation](https://slurm.schedmd.com/documentation.html) remains authoritative. The documentation here covers what is specific to your TensorWave deployment: access setup, storage layout, container support, health checks, and support scope. *** ### Before you begin A few characteristics of this architecture are worth understanding before you start: * **Worker pods are containers.** They are not long-lived OS instances. Local scratch is ephemeral and will not persist across pod restarts. * **Privileged device access is required.** GPU, RDMA, and most container runtime support on worker pods requires elevated privileges. This is standard for GPU workloads on Kubernetes and should be accounted for in your security review. * **Network and topology configuration is site-specific.** NIC selection, NCCL/RCCL tuning, and multi-node topology settings depend on your hardware. Your deployment guide contains the values for your cluster. *** ### Where to go next | Topic | Description | | --------------------- | ------------------------------------------ | | Accessing the cluster | LDAP, SSH, login layout, worker access | | Storage | Shared home, scratch, shared project space | | Running Jobs | First `sinfo` / `sbatch`, validation jobs | | Health and monitoring | Checks, drains, metrics, `sdebug` | | Prolog and epilog | Custom hooks and customer scripts | | Common Issues | FAQ and patterns often used with Slurm | *** *Hostnames, filesystem names, and cluster-specific configuration are documented in your deployment guide.* # Access ### SSH Access The login URL for your cluster is available in the TensorWave dashboard or in the handoff documentation provided at cluster delivery. Depending on how your cluster is exposed, the SSH service listens on either **port 22** or **port 32222**. Check your handoff documentation if you are unsure which applies. ```bash ssh @ # or, if using the node port: ssh -p 32222 @ ``` Authentication is by SSH key. Password authentication is disabled. Your public key must be present in your LDAP user record before first login; contact your cluster administrator if you cannot connect. Successful login lands you on a **login pod** for your user. This is where you submit jobs, inspect the queue, and manage your files. Login pods are not intended for compute-intensive work. *** ### Slurm Accounting and User Setup The cluster enforces **Slurm accounting**: every user must have a Slurm account and association before they can submit jobs. This is required by the QOS and limits configuration. #### Initial setup (cluster administrator) A cluster administrator performs the following steps once per user using `sacctmgr` from a login pod or the Slurm controller. **1. Create an account** (if one does not already exist for the team or project): ```bash sacctmgr add account Description="" ``` **2. Add the user and associate them with the account:** ```bash sacctmgr add user Account= DefaultAccount= ``` **3. Verify the association is in place:** ```bash sacctmgr show assoc where user= ``` The user can now submit jobs. Without a valid association, job submissions will be rejected. #### Checking your own account Users can verify their associations at any time: ```bash sacctmgr show assoc where user=$USER ``` For full `sacctmgr` reference, see the [Slurm accounting documentation](https://slurm.schedmd.com/accounting.html). *** ### Accessing Worker Pods Worker pods are accessible by SSH from within a login pod. Access is restricted to users who have a job currently running or a resource allocation active on the target node. #### Connecting to a worker pod If you already have a job running, find its allocated nodes: ```bash squeue -u $USER ``` Then SSH to the worker pod by its Slurm node name directly from the login pod: ```bash ssh # example: ssh tus1-p14-g36 ``` > **Note:** SSH to a worker pod will be refused if you do not have an active allocation on that node. This is by design. Worker pods use port 2222 internally. This is configured automatically on login pods; no additional flags are required. *** ### IDP Configuration (Optional) The cluster supports identity provider (IDP) integration, which allows users to authenticate using your organization's existing SSO, such as Okta, Azure AD, or Google Workspace. IDP setup requires coordination during or after initial cluster deployment. To discuss options or initiate configuration, contact your TensorWave account manager. # Storage ### Shared Home Directory `/home` is a shared, persistent filesystem backed by a high-performance distributed storage system and mounted on every login pod and worker pod in the cluster. Files written to `/home` from a login pod are immediately visible on any worker pod running your jobs, and vice versa. This makes `/home` the right place for: * Source code, scripts, and configuration files * Job output you need to keep after the job finishes * Data that needs to be accessible from multiple nodes at once #### Storage quotas Your `/home` allocation is defined in your deployment. To check current usage: ```bash df -h /home/$USER ``` Live storage data can also be viewed via the TensorWave dashboard. If you need more space or your quota adjusted, contact your TensorWave account manager. If `/home` runs out of space, things will start to break. See Common Issues for advice on managing `/home`. #### Per-User Quotas It's possible to set per-user filesystem quotas using the `weka` CLI tool. This can be performed by a cluster administrator from a designated control node in the cluster. To set a default quota for any new user's home directory, run: ```bash weka fs quota set-default /home [--soft soft] [--hard hard] ``` Values can be specified as decimal or binary units (e.g. `1GB`, `1TB`, `1GiB`, `1TiB`). To modify a specific directory's quota, run: ```bash weka fs quota set [--soft soft] [--hard hard] ``` To view existing quotas, run: ```bash weka fs quota list --all ``` For more detailed documentation, see the [WekaFS quota docs](https://docs.weka.io/4.4/weka-filesystems-and-object-stores/quota-management/quota-management). *** ### Setting Up a Shared Directory For shared data that multiple users on the same team need to access, the recommended approach is a shared directory under `/home` with appropriate group permissions. **1. Create a shared directory:** ```bash mkdir /home/shared/ ``` **2. Set group ownership:** ```bash chgrp /home/shared/ ``` **3. Set permissions so group members can read and write:** ```bash chmod 2775 /home/shared/ ``` The `2` (setgid) bit ensures new files and subdirectories inherit the group, so members do not need to manually chown files they create there. **4. Verify:** ```bash ls -ld /home/shared/ ``` To create a globally available shared directory, use the `user` group. Groups are managed through LDAP. If you need a new group created or users added to an existing one, contact your cluster administrator. *** ### Worker Pod Storage Worker pods have several storage locations available during a job. Understanding which to use prevents both data loss and performance issues. #### Summary | Path | Type | Persists after job | Shared across nodes | Notes | | ------------- | ---------------------- | ------------------ | ------------------- | ---------------------------------------------------------------------------- | | `/home/$USER` | Distributed network FS | Yes | Yes | Durable and performant; use as primary storage | | `/tmp` | Memory-backed | No | No | Fast local scratch; useful for caching large files | | `/run/tmp` | Memory-backed | No | No | Fast local scratch; used by enroot for container-runtime | | `/dev/shm` | Memory-backed | No | No | Fast local scratch; commonly used by pytorch for inter-process communication | #### `/home`: primary storage #### `/tmp`: pod-local scratch Each worker pod has a memory-backed `/tmp`. It is fast relative to a network filesystem and suitable for intermediate files your job produces and consumes within the same pod. It is **not** shared between pods and is not guaranteed to be empty at job start (though it is cleaned between jobs by policy). Do not write job outputs here that you need after the job finishes. Keep in mind that this space is carved out of the node's RAM, writing large amounts of data here can lead up pages being swapped to disk, leading to reduced performance of your job. The filesystem is cleared when the pod is replaced. #### `/run/tmp` and `/dev/shm`: tmpfs for system resources `/run/tmp` and `/dev/shm` are memory-backed filesystems (`tmpfs`) mounted on each worker. These are similar to `/tmp`, but with less space and are reserved for application use. `/run/tmp` is used internally by Pyxis/Enroot for container image staging (`/run/tmp/enroot-data`, `/run/tmp/enroot-runtime`). `/dev/shm` is used by PyTorch for inter-process communication. *** > **General rule:** Write outputs you need to keep to `/home`. Use `/tmp` for scratch that only lives for the duration of the job. Treat all pod-local paths as ephemeral. # Jobs ### Overview Jobs on your cluster are scheduled and run using the Slurm workload manager. Resources can be allocated through two main mechanisms: * **Interactive** (`salloc`): allocate resources and run commands directly, useful for development and debugging. * **Batch** (`sbatch`): submit a script that runs when resources become available, the standard approach for production workloads. Within an allocation you can then use the `srun` command to execute job steps and further distribute work across your allocation. For command reference, see the [official Slurm documentation](https://slurm.schedmd.com/documentation.html). *** ### Inspecting the Cluster Before we submit jobs, it's useful to understand how we can inspect the state of the cluster and queue. #### [`sinfo`](https://slurm.schedmd.com/sinfo.html) `sinfo` is the basic mechanism for inspecting the state of the resources available ```bash sinfo ``` #### [`squeue`](https://slurm.schedmd.com/squeue.html) `squeue` is how we can inspect the state of the queue. You can use the `-u` flag to check the status of your current running jobs. ```bash squeue -u $USER ``` *** ### Resource Specification Though there are some differences in how resources are specified with the `salloc`, `sbatch`, and `srun` commands, the common directives are generally the same. | Flag | Description | | ------------------- | ------------------------------------ | | `--nodes` / `-N` | Number of nodes | | `--ntasks-per-node` | Number of tasks (processes) per node | | `--gpus-per-node` | GPUs per node | | `--cpus-per-task` | CPU cores per task | | `--time` | Wall-clock time limit (`HH:MM:SS`) | | `--nodelist` | Run on specific nodes | > This is not a comprehensive list of the available flags, please reference Slurm documentation for full man pages. All jobs must request GPU resources explicitly. Reasonable defaults are established to subdivide CPU cores and memory based on the GPU allocation, but these can be overridden per job. ### Job Steps with [`srun`](https://slurm.schedmd.com/srun.html) Within the following allocation methods, it is useful to understand how job steps work and how they can be used to maximize allocations. By default, `srun` commands will inherit the entire resource allocation for all subcommands. This is useful for submitting monolithic jobs, but can be tuned to instead subdivide resources within an allocation for multiple tasks. Consider the following examples in ways srun can be used from within an allocation: ```bash # Single node, single task. srun -n 1 -N 1 --gpus-per-node=8 python task.py # Two nodes, two running instances of a single task. srun -n 2 -N 2 --gpus-per-node=8 python task.py # Two nodes, two different tasks running in parallel. srun -n 1 -N 1 --gpus-per-node=8 --exclusive python task1.py & srun -n 1 -N 1 --gpus-per-node=8 --exclusive python task2.py & wait ``` ### Interactive Allocation with [`salloc`](https://slurm.schedmd.com/salloc.html) To launch an interactive allocation within Slurm, use the `salloc` command. ```bash # Allocate 1 node with 8 GPUs salloc -N 1 --gpus-per-node=8 ``` The `salloc` session holds the allocation open. When you exit the `salloc` shell, the allocation is released and the node becomes available to other jobs. For an interactive shell directly on the worker pod, you can add the following `srun`: ```bash salloc -N 1 --gpus-per-node=8 srun --interactive --pty bash -l ``` To drop into an interactive shell inside a container, use `apptainer shell` with `srun` (don't forget the `--pty` flag): ```bash srun -N 1 --gpus-per-node=8 --pty \ apptainer shell docker://rocm/pytorch:rocm7.2.2_ubuntu24.04_py3.12_pytorch_release_2.10.0 ``` This allocates a worker pod, pulls (or uses a cached) container image, and drops you into an interactive shell inside it with GPUs available. Use a local `.sif` file instead of a `docker://` URI if you have already pulled the image. For more on building and running Apptainer images, including multi-node jobs and networking, see Containers and Modules. ### Batch Allocation with [`sbatch`](https://slurm.schedmd.com/sbatch.html) Batch jobs are submitted with `sbatch` and run when the scheduler grants the allocation. Resource requests, environment setup, and the actual workload are all defined in the job script. Any `#SBATCH` directive in the script can also be overridden at submission time by passing the corresponding flag directly to `sbatch`. #### Basic structure ```bash #!/bin/bash #SBATCH --job-name=my-job #SBATCH --output=jid-%j.name-%x.log #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=96 #SBATCH --gpus-per-node=8 #SBATCH --time=01:00:00 srun my-program ``` Submit with: ```bash sbatch my-job.sh ``` View output once the job runs: ```bash tail -f jid-.name-my-job.log ``` #### Multi-node distributed training example The pattern below works for PyTorch `torch.distributed.run` (torchrun) across multiple nodes. The first node allocated by Slurm acts as the rendezvous host. ```bash #!/bin/bash #SBATCH --job-name=ddp-training #SBATCH --output=jid-%j.name-%x.log #SBATCH --nodes=4 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=48 #SBATCH --gpus-per-node=8 #SBATCH --time=04:00:00 GPUS_PER_NODE=8 MASTER_ADDR=$(hostname) MASTER_PORT=6000 srun bash < ``` *** ### MPI (PMIx) The cluster uses **PMIx** as the default MPI launch interface (`MpiDefault=pmix`). Open MPI and other PMIx-compatible MPI libraries work with `srun` without needing `mpirun`. The following environment is set cluster-wide and applied automatically to all login and compute sessions: ```bash OMPI_MCA_btl_tcp_if_include=eno0,eno1 # Route TCP over front-end to avoid back-end contention OMPI_MCA_btl=^openib # Disable legacy OpenIB transport PMIX_MCA_gds=hash # Required PMIx GDS backend ``` #### Running an MPI job Use `srun` directly. Slurm handles process launch and PMIx initialization: ```bash #!/bin/bash #SBATCH --job-name=mpi-job #SBATCH --nodes=2 #SBATCH --ntasks-per-node=8 #SBATCH --gpus-per-node=8 #SBATCH --cpus-per-task=16 #SBATCH --time=01:00:00 srun ./my-mpi-program ``` We recommend using `srun` rather than `mpirun` or `mpiexec`, as it integrates directly with Slurm's process placement and PMIx initialization. #### RCCL collective communication For GPU collective benchmarks and validation, RCCL tests are pre-installed on compute nodes. A minimal all-reduce test across 2 nodes: ```bash #!/bin/bash #SBATCH --job-name=rccl-test #SBATCH --nodes=2 #SBATCH --ntasks-per-node=8 #SBATCH --gpus-per-node=8 #SBATCH --cpus-per-task=16 #SBATCH --time=00:05:00 export NCCL_IB_QPS_PER_CONNECTION=2 export NCCL_BUFFSIZE=8388608 export UCX_NET_DEVICES=eno0 srun /opt/rccl-tests/all_reduce_perf -b 512M -e 8G -f 2 -g 1 ``` *** > For containerized jobs (Pyxis/Enroot, Apptainer) and environment modules, see Containers and Modules. # Containers ### Overview The cluster supports several ways to run containerized workloads including Apptainer, Pyxis/Enroot, and Docker. The recommended approach to run containers is **Apptainer**. Since it was designed for HPC/Slurm environments, it has several benefits and is significantly easier to use. It integrates cleanly with `srun` across multiple nodes, seamlessly uses the shared `/home` directory to cache images, and it runs containers as the submitting user (avoiding issues that Docker's privileged daemon introduces). For software that does not require a container, see Modules for Lmod and SHPC. Python virtual environments (`.venv`) installed under `/home` are also a straightforward option, since `/home` is mounted consistently across login and compute pods. *** ### Apptainer Apptainer (formerly Singularity) runs containers as the submitting user. Images can be pulled directly from a Docker registry at runtime or pre-built as `.sif` files for faster starts and multi-node use. #### Registry login Public images (such as `rocm/` on Docker Hub) can be pulled without authentication. Private registries require logging in first: ```bash apptainer registry login --username docker://docker.io ``` You will be prompted for your password or access token. To supply credentials non-interactively: ```bash echo '' | apptainer registry login --username --password-stdin docker://docker.io ``` Many cloud OCI registries use token-based authentication. In that case, pass the token as the password; a username is still required. Consult your provider's documentation for their specific login requirements. See the [Apptainer registry login documentation](https://apptainer.org/docs/user/main/cli/apptainer_registry_login.html) for all options. Credentials are stored under your home directory and apply to subsequent `apptainer pull`, `apptainer exec`, and `apptainer shell` calls that reference the registry. To remove stored credentials: ```bash apptainer registry logout docker://docker.io ``` #### Pulling an image Pull an image from a Docker registry and save it as a local `.sif` file. Running from a login pod is fine for this step since it does not require a GPU allocation: ```bash apptainer pull rocm-pytorch.sif docker://rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 ``` The resulting `.sif` file can be used in any subsequent `apptainer exec` or `apptainer shell` call and starts faster than pulling the `docker://` URI at runtime. Store it on `/home` so it is accessible from compute pods. #### Single-node interactive ```bash srun -N 1 --gpus-per-node=8 --pty apptainer shell rocm-pytorch.sif ``` You can also pass a `docker://` URI directly without pulling first: ```bash srun -N 1 --gpus-per-node=8 --pty \ apptainer shell docker://rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 ``` > **Note**: Apptainer will passthrough-mount the `/home/$USER` and `/tmp` directories. This can be disabled with `--contain` or `--no-home` flags. #### Batch job (single node) For single-node jobs, Apptainer can pull the image at runtime: ```bash #!/bin/bash #SBATCH --job-name=apptainer-job #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --gpus-per-node=8 #SBATCH --cpus-per-task=48 #SBATCH --time=02:00:00 srun apptainer exec \ docker://rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 \ python train.py ``` #### Multi-node jobs: building a BNXT-enabled SIF For multi-node jobs, the container image must include the correct network software for the cluster's NICs. This can either be built into the image, or passed through using Apptainer's cdi interface. See Installing Network Software in Container images for details. *** ### Pyxis Pyxis is a SPANK plugin that integrates OCI container execution directly into `srun` via flags. It uses Enroot under the hood to manage squashfs-format images (`.sqsh`). #### Pulling and caching a container Images can be pulled from public repos using the `--container-image` flag. ```bash # Pull the image and stash it as a named container (run once per image per node) srun \ --container-writable \ --container-name=my-pytorch \ --container-image=rocm/pytorch:latest \ true ``` The `--container-name` flag caches the image as a named Enroot container. Subsequent `srun` steps using the same name skip the pull and start much faster. Because the container is writable (`--container-writable`), any modifications made during one step are preserved across subsequent steps that reference the same named container. To persist the container to disk, the `--container-save=PATH` flag can be used, this saves the container state as a .sif file and can be reused in future jobs. #### Running a job with a named container ```bash #!/bin/bash #SBATCH --job-name=pyxis-job #SBATCH --nodes=2 #SBATCH --ntasks-per-node=8 #SBATCH --gpus-per-node=8 #SBATCH --cpus-per-task=16 #SBATCH --time=02:00:00 # Pull and cache on all nodes srun \ --no-container-remap-root \ --container-writable \ --container-name=my-pytorch \ --container-image=rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 \ true # Run the workload srun \ --no-container-remap-root \ --container-name=my-pytorch \ /opt/rccl-tests/all_reduce_perf -b 512M -e 8G -f 2 -g 1 ``` Example Pyxis scripts are available at `/opt/tw/examples/libexec/*pyxis*.sbatch`. #### Enroot as an Escape Hatch Under the hood, Pyxis uses Enroot as a containerization engine. Some operations (like pulling an image from a private repo) require using the `enroot` cli tool. ```bash # Login to Docker echo "YOUR_PASSWORD_OR_TOKEN" | docker login -u YOUR_USERNAME --password-stdin # Pull an image using with `dockerd://` (docker engine backend) # Image unpacking needs to run in a privileged environment, it can't be done on a login node srun enroot import dockerd://YOUR_REPO/YOUR_IMAGE:YOUR_TAG ``` This will leave a `*.sqsh` file in your current working directory, which can be passed to `--container-image` in future Slurm jobs. *** ### Docker > **Warning:** Docker on worker nodes runs as root via a privileged daemon. All Docker containers are re-parented and launched by the Docker-daemon's process tree, and this leads to several limitations. > > * **Docker containers ignore Slurm's job-isolation***.* A job requesting 1 GPU will still have access to all 8. Slurm jobs that use Docker should either be allocated as `--exclusive`, or launch jobs with appropriate resource-isolation flags. > * **Docker containers aren't deleted during job cleanup***.* Since Docker containers run outside the slurmd's process tree, they won't receive `SIGTERM`/`SIGKILL` signals on job-end. Any Slurm jobs using Docker should have a `trap` function to stop and clean up any containers. > > It is highly recommended to use Apptainer or Pyxis instead of Docker. Docker is available on worker nodes. You can use it to run containers, build images, or pull from a registry within a job allocation. #### Running a container ```bash srun -N 1 --gpus-per-node=8 --pty bash -l # Inside the allocation: docker run --rm --gpus all rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 \ python -c "import torch; print(torch.cuda.device_count())" ``` #### Building an image If you need to build a custom image during a job, allocate a node and run the build from there: ```bash srun -N 1 --pty bash # Inside the allocation: docker build -t my-org/my-image:latest -f Dockerfile . docker push my-org/my-image:latest ``` #### Using a Docker image with Apptainer Docker images can be consumed directly by Apptainer without running the Docker daemon at all, using the `docker://` URI: ```bash srun apptainer exec docker://rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 \ python train.py ``` This is the preferred pattern for job submission since it runs as the submitting user and integrates with Slurm resource accounting. *** ### Command Comparison | Operation | Apptainer | Pyxis (srun flags) | Docker | | ----------------------------- | ----------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | | **Launch a batch job** | `srun apptainer exec ` | `srun --container-image= ` | `srun docker run --rm ` | | **Get an interactive shell** | `srun --pty apptainer shell ` | `srun --pty --container-image= bash` | `srun --pty docker run -it --rm --entrypoint /bin/bash` | | **Download an image to Disk** | `apptainer pull .sif docker://` | `srun --container-image= --container-save=.sqsh true` | (closest equivalent) `docker pull ` | | **Volume mount** | `--bind [:]` | `--container-mounts=:` | `-v :` | # Network Drivers ### How to get docker images up-and-running on TensorWave clusters. To get multi-node containerized applications, two major steps need to happen: 1. make sure host-network passthrough is enabled, and 2. make sure the proper drivers exist in the image. Enabling host network passthrough is straightforward. When starting a Docker container, add --network host to the command. e.g.: `docker run --network host rocm/pytorch:latest`. When configuring Kubernetes, add hostNetwork: true to the pod spec. Here's an example of a pod with a host network configured: ``` Copy apiVersion: v1 kind: Pod metadata: name: host-network-example spec: hostNetwork: true # <=== this enables host network containers: - name: pytorch image: tensorwavehq/pytorch:latest ``` If you're using Apptainer or Pyxis on the TensorWave's Slurm platform, host-networks are configured by default. However, you may need to ensure the proper drivers are installed in your image. The sections below provide instructions for driver installation on our two platforms. > **Note** If you're using apptainer in Slurm, you can use TensorWave's provided container device interface (CDI) spec to mount the worker pod's network software into the host. See The CDI section below for details. ### AINIC (MI355X) The MI355X nodes use AMD Polara NICs for the backend network. AMD provides their NIC software as a binary. To install it, add the upstream apt source and apt-install the appropriate software. Below, we provide a sample Dockerfile that installs the AINIC drivers inside AMD's ROCM PyTorch image. Copy the file to your local machine, and run `docker build -f -t rocm-pytorch-ainic .`. You could also add this section as a build stage in your own Dockerfile build pipeline. Add it to the top of your Docker file, and update the appropriate `FROM AS ` lines to fold it into the build pipeline ```dockerfile FROM rocm/pytorch:latest AS rocm-pytorch-ainic ENV DEBIAN_FRONTEND=noninteractive WORKDIR /tmp ARG REPO_URL=https://repo.radeon.com ARG DRIVERS_VERSION=1.117.5-a-56 RUN << EOR set -eux UBUNTU_CODENAME=$(awk -F= '/^UBUNTU_CODENAME=/{gsub(/"/,"",$2); print $2}' /etc/os-release) mkdir --parents --mode=0755 /etc/apt/keyrings wget ${REPO_URL}/rocm/rocm.gpg.key -O - | gpg --dearmor | tee /etc/apt/keyrings/rocm.gpg > /dev/null echo "Types: deb" > /etc/apt/sources.list.d/amdainic.sources echo "URIs: ${REPO_URL}/amdainic/pensando/ubuntu/${DRIVERS_VERSION}" >> /etc/apt/sources.list.d/amdainic.sources echo "Suites: ${UBUNTU_CODENAME}" >> /etc/apt/sources.list.d/amdainic.sources echo "Components: main" >> /etc/apt/sources.list.d/amdainic.sources echo "Signed-By: /etc/apt/keyrings/rocm.gpg" >> /etc/apt/sources.list.d/amdainic.sources apt-get update && apt-get install -y bc jq libibverbs-dev rdma-core ibverbs-utils libionic-dev libionic1 perftest libfmt-dev EOR ``` ### Broadcom (MI325X & MI300X) Tensorwave MI300X and MI325X clusters use Broadcom Ethernet cards for the backend network. These require building Broadcom's driver from source code. The following Dockerfile builds the Broadcom drivers into AMD's ROCm PyTorch image. It can be used from a slurm-login node. Copy the file to your login environment and run `docker build -f -t rocm-pytorch-bnxt .`. If you don't have access to TensorWave's Slurm platform and need the libbnxt\_re tarfile, reach out to your cluster administrator. ```dockerfile Copy FROM rocm/pytorch:latest AS rocm-pytorch-bnxt ENV DEBIAN_FRONTEND=noninteractive WORKDIR /tmp ARG BNXT_VER=233.0.152.2 ARG BNXT_NAME=libbnxt_re-${BNXT_VER} COPY /opt/tw/drivers/${BNXT_NAME}.tar.gz /tmp/bnxt-drivers/ RUN << EOR set -eux export apt-get -qq update apt-get -qq -y install --no-install-recommends \ autoconf automake bison build-essential ethtool g++ hwloc \ ibverbs-utils infiniband-diags initramfs-tools iproute2 \ iputils-ping kmod libibverbs-dev libibumad-dev libncurses5-dev \ librdmacm-dev libsysfs-dev libtool make net-tools pciutils \ plocate strace sudo vim wget which dash &> /dev/null && (\ echo "dash dash/sh boolean false" | debconf-set-selections && \ dpkg-reconfigure dash) || \ echo "Skipping dash reconfigure (not applicable)" ls /tmp/bnxt-drivers tar -xzf /tmp/bnxt-drivers/${BNXT_NAME}.tar.gz -C /tmp/bnxt-drivers/ mv /usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so /usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so.hidden cd /tmp/bnxt-drivers/${BNXT_NAME}/ && sh ./autogen.sh && ./configure make -C /tmp/bnxt-drivers/${BNXT_NAME} clean all install echo '/usr/local/lib' > /etc/ld.so.conf.d/libbnxt_re.conf ldconfig EOR ``` ### Mounting Worker-Pod Network Software with CDI > **Note**: This feature is in beta. If you are using TensorWave's Managed Slurm, we provide a `/etc/cdi/tw.json` file to mount the userspace InfiniBand software into an image. This has been validated to work with apptainer using images built off [rocm/pytorch with Ubuntu22.04](https://hub.docker.com/r/rocm/pytorch/tags?name=ubuntu22.04). To enable it, pass the appropriate `--device` option when launching an apptainer image. Here's an example sbatch script running a pytorch RCCL benchmark. The `--device` value is either `tw.amd.com/bnxt=bnxt` or `tw.amd.com/ainic=ainic` depending on the cluster's NICs. There is also an example in the slurm login environment at `/opt/tw/examples/libexec/rccl-torch-apptainer.sbatch`. ```bash #!/bin/bash #SBATCH --job-name=torch_all_reduce_apptainer_cdi #SBATCH --output=jid-%j.name-%x.log #SBATCH --tasks-per-node=1 #SBATCH --gpus-per-node=8 #SBATCH --cpus-per-task=128 #SBATCH --nodes=4 if [[ -e "/opt/tw/drivers/libbnxt_re-233.0.152.2.tar.gz" ]]; then CDI_DEVICE="tw.amd.com/bnxt=bnxt" else CDI_DEVICE="tw.amd.com/ainic=ainic" fi MASTER_ADDR=$(cat /etc/hostname) srun apptainer exec \ --device $CDI_DEVICE \ --bind /opt/tw/examples/bin \ docker://rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0 \ bash -s <> $HOME/.bashrc ``` Admins can omit this step to install packages globally. The following install commands will target `/mnt/shpc`, and will be available to all users. **Install a module from the curated registry:** ```bash shpc show # list available recipes shpc install rocm/pytorch # pull SIF, write modulefile module avail # confirm it is visible module load rocm/pytorch ``` **Install an arbitrary OCI image:** ```bash shpc add docker:///: # Edit the generated recipe at /mnt/modulestore/shpc/registry/... shpc install /: ``` Note: **running module-installed apps only works on privileged pods** (worker-pods and VM-based login pods). If you encounter the error `ERROR : Failed to create mount namespace: mount namespace requires privileges, check Apptainer installation`, try running the same command with `srun`. Global modules and container images are stored on the shared `modulestore` volume (`/mnt/modulestore`), so an install on a login pod is immediately available on compute pods without any additional steps. Users are encouraged to install their modules in their `/home/$USER` directory. # Prolog / Epilog ### Overview Slurm supports prolog and epilog scripts that run automatically on each worker pod at the start and end of every job. Prologs run upon allocation; epilogs run after the job completes or is cancelled. Common uses include: * Verifying node health before a job starts * Cleaning up temporary files or resetting state after a job ends * Logging job metadata for monitoring or auditing * Enforcing site-specific policies around GPU, network, or filesystem state Prolog and epilog scripts run as **root** on the worker pod, which gives them broad access but also means a failing or buggy script can affect the pod and any jobs running on it. Specifically, **if a prolog or epilog script exits with a non-zero code, Slurm will place the node in DRAIN state**, taking it out of service. Test scripts carefully before deploying them. For full background on how Slurm handles prolog and epilog execution, see the [Slurm Prolog and Epilog Guide](https://slurm.schedmd.com/prolog_epilog.html). *** ### Built-in scripts TensorWave runs a set of managed prolog and epilog scripts on every job automatically. These handle node health checks (see Health Checks), GPU metrics collection for the dashboard, and dispatching your custom scripts. Your scripts always run after the built-in health checks. *** ### Adding custom scripts Custom prolog and epilog scripts go in the following directories on the shared storage volume: | Directory | Script Execution | | ------------------------- | ------------------------------------------------- | | `/mnt/customer/prolog.d/` | Upon allocation, on every allocated node | | `/mnt/customer/epilog.d/` | After each job completes, on every allocated node | Scripts are executed in **lexicographic order** by filename. Use numeric prefixes to control ordering, and use leading zeroes if necessary to ensure accurate sorting: ``` /mnt/customer/prolog.d/ 01-check.sh 10-check-something-else.sh 20-setup-environment.sh 99-final-step.sh ``` #### Requirements * Scripts must be **executable** (`chmod +x`). Non-executable files are skipped with a warning in the log. * Scripts must include a **shebang** on the first line (`#!/usr/bin/env bash`). * Scripts run as **root**. A non-zero exit code will drain the node. #### Example prolog script ```bash #!/usr/bin/env bash echo "Hello from job $SLURM_JOB_ID on node $SLURMD_NODENAME" ``` Install it: ```bash sudo cp my-prolog.sh /mnt/customer/prolog.d/10-my-prolog.sh sudo chmod +x /mnt/customer/prolog.d/10-my-prolog.sh ``` #### Example epilog script ```bash #!/usr/bin/env bash echo "Goodbye from job $SLURM_JOB_ID on node $SLURMD_NODENAME" # Clean up any job-specific scratch rm -rf /tmp/job-${SLURM_JOB_ID} ``` Install it: ```bash sudo cp my-epilog.sh /mnt/customer/epilog.d/10-my-epilog.sh sudo chmod +x /mnt/customer/epilog.d/10-my-epilog.sh ``` *** ### Viewing logs Per-node prolog and epilog logs are written to: | Path | Contents | | -------------------------------------- | ------------------------------------------- | | `/mnt/customer/logs/prolog/.log` | Output from all prolog scripts on that node | | `/mnt/customer/logs/epilog/.log` | Output from all epilog scripts on that node | Each entry includes a timestamp, script path, exit code, job ID, and user. Script output is only captured in the log when the script fails. **Viewing a node's prolog log:** ```bash cat /mnt/customer/logs/prolog/tus1-p2-g6.log ``` Example output for a successful prolog: ``` timestamp=2026-04-01T18:43:39Z script=/mnt/customer/prolog.d/10-my-prolog.sh exit_code=0 job_id=5995 job_user=user@example.com ``` Example output when an epilog script fails (output is included): ``` timestamp=2026-04-01T18:43:42Z script=/mnt/customer/epilog.d/10-my-epilog.sh exit_code=1 job_id=5995 job_user=user@example.com --- output --- Goodbye from job 5995 on node ``` If a script fails and the node drains, check the log for the affected node first, then inspect node state with `sinfo`: ```bash sinfo -n tus1-p2-g6 ``` Once the issue is resolved, contact your cluster administrator to resume the node. *** > Scripts in `/mnt/customer/prolog.d` and `/mnt/customer/epilog.d` are writable by administrators only (`chmod 1700`). Logs in `/mnt/customer/logs` are readable by all users. # Job Submission Lua Plugin ### Overview Slurm's job-plugin interface provides a facility for cluster admins to intercept, modify, and/or deny users' resource requests. This feature enables functionality like *ensuring jobs have comments for resource tracking,* or *setting a default 'timelimit' for interactive jobs*. The plugin behaviour is specified in a Lua file (`/mnt/customer/job_submit.lua`) which provides a flexible and powerful interface enabling broad functionality. For more background on Slurm's job plugin, see [Slurm's job submit plugin docs](https://slurm.schedmd.com/job_submit_plugins.html) or the [job\_submit\_lua.so source code](https://github.com/SchedMD/slurm/blob/slurm-25.11/src/plugins/job_submit/lua/job_submit_lua.c) for details on the Lua integration. *** ### Script Requirements The script must be placed in `/mnt/customer/job_submit.lua`, and return a module with functions for `slurm_job_submit` and `slurm_job_modify`. See the 'API' section for function specifications. **After modifying the `job_submit.lua`, run `scontrol reconfigure` to apply the changes**. `slurmctld` internally caches the most recent working version of the `job_submit.lua`. If a custom `job_submit.lua` script fails to load, this may result in either a no-op at job submission time, or the behaviour of the last successful `job_submit.lua`. Further, if the update `job_submit.lua` fails to load, errors may be logged at `/mnt/customer/job_submit_lua.log`. If the `job_submit.lua` file is not found, or if the `job_submit.lua` file is missing functions, errors are reported in `/mnt/customer/job_submit_lua.log`. The Lua script is run by the `slurmctld` on the controller pod. **The shared-`/home` directory is not accessible from the slurm-controller**. To generate persistent logs, write to `/mnt/customer`, see the `log()` function in the Example section below. *** ### API `function slurm_job_submit(job_desc, part_list, submit_uid)` This function is called when a job is submitted. It can be triggered by `salloc`, `sbatch`, or `srun`. Arguments: * `job_desc` : Requested job allocation * `part_list` : List of partitions the user is authorized to use * `submit_uid` : User ID of requesting user return values * `slurm.SUCCESS` on success * `slurm.ERROR` for generic errors * `slurm.ESLURM_*` for specific errors (see [slurm/slurm\_errno.h](https://github.com/SchedMD/slurm/blob/slurm-25.11/slurm/slurm_errno.h)) `function slurm_job_modify(job_desc, job_rec, part_list, uid)` This function is called when a job-modification request is made. Arguments: * `job_desc` : Specification of requested modifications * `job_ptr` : Pointer to the job to be modified * `part_list` : List of partitions the user is authorized to use * `modify_uid` : User ID of requesting user return values * `slurm.SUCCESS` on success * `slurm.ERROR` for generic errors * `slurm.ESLURM_*` for specific errors (see [slurm/slurm\_errno.h](https://github.com/SchedMD/slurm/blob/slurm-25.11/slurm/slurm_errno.h)) *** ### Example Plugin Here's a sample plugin to get up and running quickly. It enforces that all allocation requests have eight GPUs per node (i.e. `--gpus-per-node=8`). We also provide a simple test-harness which is useful for development/debugging. ```lua --- Example job_submit.lua for TensorWave's Managed Slurm --- Denies allocations that don't request `--gpus-per-node=8` --- Place this file at /mnt/customer/job_submit.lua and run `scontrol reconfigure` local function log(msg) -- logging helper, reads global var for dest, can be overridden in test-harness -- /home is not available in slurmctld context, need to log to /mnt/customer local log_dest = (g_log_dest == nil) and "/mnt/customer/job_submit_lua.log" or g_log_dest local file, err = io.open(log_dest, "a") if not file then slurm.log_error("Failed to open log file: "..tostring(err)) return end local timestamp = os.date("%Y-%m-%d %H:%M:%S") prefix = string.format("[%s] ", timestamp) file:write(prefix .. msg .. "\n") file:close() end local function eight_gpu_per_node(job_desc) -- Validates that --gpus-per-node=8 if not (job_desc["tres_per_node"] == "gres/gpu:8") then return slurm.ESLURM_INVALID_GRES end return slurm.SUCCESS end local function check_job_desc(job_desc, uid) local msg = string.format("uid: %d, job name: %s, tres_per_node: %s, gres: %s", uid, job_desc['name'], job_desc['tres_per_node'], job_desc['gres']) log("SKIP slurm_job_submit ".. msg) return eight_gpu_per_node(job_desc) end function slurm_job_submit(job_desc, part_list, uid) return check_job_desc(job_desc, uid) end function slurm_job_modify(job_desc, job_rec, part_list, uid) return check_job_desc(job_desc, uid) end -- Export the slurm_job_modiyf/slurm_job_submit functions so the plugin can consume them return { slurm_job_modify = slurm_job_modify, slurm_job_submit = slurm_job_submit } ``` ```lua --- Test harness for the example job_submit.lua --- Sets up a dummy Slurm-env, checks functions are exported properly --- Ensures that only allocations with "gres/gpu:8" are allowed -- Set globals to emulate the slurm environment _G.slurm = { log_error = print, SUCCESS = "SUCCESS", ERROR = "ERROR", ESLURM_INVALID_GRES = "ESLURM_INVALID_GRES", } -- Log to local directory _G.g_log_dest = "./job_submit_lua.test.log" -- Load the job script to test -- package.path = "/mnt/customer/job_submit.lua;" .. package.path package.path = "./job_submit.lua;" .. package.path local jsm = require("job_submit.lua") assert(jsm.slurm_job_submit) assert(jsm.slurm_job_modify) assert(type(jsm.slurm_job_submit == 'function')) assert(type(jsm.slurm_job_modify == 'function')) tests = { gpus_per_node_8 = {"gres/gpu:8", slurm.SUCCESS}, gpus_per_node_6 = {"gres/gpu:6" , slurm.ESLURM_INVALID_GRES}, no_gpu_req = {nil , slurm.ESLURM_INVALID_GRES}, } for test_name, test_payload in pairs(tests) do tg, expected = table.unpack(test_payload) test_job_desc = { name = "test-name", tres_per_node = tg, gres = tg, } ret = jsm.slurm_job_submit(test_job_desc, nil, 9002) if ret == expected then print(string.format("SUCCESS, gres: %s, returned: %s", tg, ret)) else print(string.format("FAIL, gres: %s, returned: %s", tg, ret)) end end ``` # Healthchecks ### Overview TensorWave runs four layers of health checks to assure reliability. Passive checks are run continuously via a 5-minute 'heartbeat', as well as during job prolog + epilog. Active checks are scheduled every few days. Passive checks act as health gates: a failed check places the node in DRAIN state and it will not accept new jobs until the issue is resolved. Once the underlying problem is fixed, a passing check returns the node to service automatically. Nodes drained for reasons unrelated to health checks (for example, manual operator action) are not automatically resumed. Healthcheck results are continuously logged, use `sdebug info` to read and summarize results for all checks across all nodes. *** ### Passive checks Passive checks are quick and non-invasive checks that validate hardware is properly configured and in a known-good state. #### Prolog and Epilog Passive checks run on every job start and end via Slurm prolog and epilog. They are fast, non-disruptive, and cover hardware and configuration state: * GPU presence, ECC error counts, and reset state * RDMA link status, GID tables, and InfiniBand device presence * Network interface state and recent link flap events * Filesystem mounts and system resource limits * Required daemons (SSSD, SSHD, LLDP) #### Continuous checks Slurm's `HealthCheckProgram` runs a health check on every node at a fixed wall-clock interval, independent of job activity. Continuous checks run the full passive check suite plus an **RDC health check**, which verifies that the ROCm Data Center daemon and underlying GPU hardware are in a known-good state. A failure drains the node the same way a prolog failure does. Once the issue is resolved, a subsequent passing continuous check will automatically return the node to service. *** ### Active checks Active checks are heavier per-node tests submitted as Slurm jobs by a cron process on the controller. They run on a schedule against idle nodes and do not require a user job to be present: * Single-node RCCL collective communication performance * Multi-node training convergence against a reference baseline * GPU stress, memory, and PCIe bandwidth and error tests *** ### Multinode checks Multinode checks are initiated by an operator or user and do not produce a drain signal on their own. They are used for cluster validation and troubleshooting. Available multinode checks include: * **RCCL** — collective communication tests across node pairs to identify nodes with degraded interconnect performance * **IB perf** — InfiniBand bandwidth tests between node pairs ```bash # Run a shakedown across all nodes in a nodelist sdebug run rccl --nodelist tus1-p14-g[1-64] ``` *** ### sdebug `sdebug` is the CLI for running manual checks and reviewing persisted results from all check layers. **Manually run checks against a set of nodes:** ```bash sdebug run -N 2 --nodelist tus1-p14-g[1-2] passive ``` **Run a specific set of checks in an `salloc` instance:** ```bash salloc -N 2 --gpus-per-node=8 sdebug run --testlist ecc,rdma-links,gpu-count ``` **Run the active suite in an sbatch:** ```bash #!/bin/bash #SBATCH --job-name=my-job #SBATCH --output=jid-%j.name-%x.log #SBATCH --nodes=4 #SBATCH --gpus-per-node=8 # Validate node health sdebug run passive SDEBUG_ECODE=$? if [[ "$SDEBUG_ECODE" != "0" ]]; then echo "Node failed healthcheck!" return $SDEBUG_ECODE fi srun pytorch train.py ``` #### sdebug info `sdebug info` reads persisted results from disk and prints a summary across nodes, similar to `sinfo`. No allocation is required. ```bash sdebug info ``` Filter by event type or hardware category: ```bash sdebug info prolog # prolog check results sdebug info epilog # epilog check results sdebug info gpu # GPU-related checks sdebug info net # network-related checks sdebug info scheduled # scheduled active check results sdebug info timestamps # last check time per node ``` Results are read from `/mnt/twhc/` and cover all event types: prolog, epilog, scheduled, and healthcheck runs. To drill into a specific node: ```bash sdebug info -n tus1-p14-g36 ``` This shows the full health check history for that node across all event types, which is useful when investigating a drain or verifying a node after repair. # Performance ### Network Topology TensorWave GPU clusters are designed around two complementary networking optimizations that directly affect multi-GPU workload performance. **Rail-optimized networking.** Each GPU has a dedicated RDMA NIC, giving it an exclusive high-bandwidth path for collective communications. With 8 GPUs per node, there are 8 independent rails (`rdma0`-`rdma7`), one per GPU. This means RCCL ring and tree algorithms can saturate all available bandwidth simultaneously without GPUs competing for shared NIC resources. **Topology-aware scheduling.** Nodes are organized into physical pods, each sharing a top-of-rack network fabric. Slurm's tree topology plugin uses a `topology.conf` that maps nodes to pods and pods to a spine, allowing the scheduler to preferentially allocate nodes within the same pod for a given job. For multi-node jobs, this reduces cross-switch hops and keeps the majority of collective traffic on the lower-latency intra-pod fabric. *** ### RCCL All-Reduce Test RCCL (ROCm Collective Communications Library) tests are pre-installed on compute nodes at `/opt/rccl-tests/`. The `all_reduce_perf` benchmark measures collective communication bandwidth across GPUs and is useful for validating interconnect performance and identifying nodes with degraded network throughput. A sample sbatch job driving `all_reduce_perf` is provided at `/opt/tw/examples/libexec/rccl.sbatch`. #### Running the test Create an output directory and submit the job: ```bash sbatch /opt/tw/examples/libexec/rccl.sbatch ``` `**rccl.sbatch`:\*\* ```bash #!/bin/bash #SBATCH --job-name=rccl_tests #SBATCH --output=jid-%j.name-%x.log #SBATCH --ntasks-per-node=8 #SBATCH --cpus-per-task=16 #SBATCH --gpus-per-node=8 #SBATCH --time=01:00:00 #SBATCH --nodes=2 set -euxo pipefail # Use 2 InfiniBand queue pairs per connection between ranks export NCCL_IB_QPS_PER_CONNECTION=2 # Double buffer size for NCCL communications export NCCL_BUFFSIZE=8388608 # Prevent MPI from using InfiniBand export UCX_NET_DEVICES=eno0 srun /opt/rccl-tests/all_reduce_perf -b 512M -e 8G -f 2 -g 1 ``` To run on more nodes, override the `--nodes` value at submission time: ```bash sbatch --nodes /opt/tw/examples/libexec/rccl.sbatch ``` #### Script parameters **Environment variables** | Variable | Value | Purpose | | ---------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- | | `NCCL_IB_QPS_PER_CONNECTION` | `2` | Increases InfiniBand queue pairs per connection, improving routing entropy and throughput. | | `NCCL_BUFFSIZE` | `8388608` | Sets the RCCL communication buffer to 8 MB. Larger buffers can improve performance at high message sizes. | | `UCX_NET_DEVICES` | `eno0` | Directs UCX control traffic over Ethernet, leaving InfiniBand dedicated to RCCL data traffic. | | `NCCL_IB_GID_INDEX` | `1` or `3` | Specifies which GID index RCCL should use, values are dependent on the NIC vendor of your cluster. | **RCCL test arguments** | Argument | Value | Description | | -------- | ------ | ------------------------------------------------------ | | `-b` | `512M` | Minimum message size | | `-e` | `8G` | Maximum message size | | `-f` | `2` | Step factor (doubles each step: 512M, 1G, 2G, ..., 8G) | | `-g` | `8` | GPUs per process | #### Reading the output A successful run completes without errors and shows increasing bus bandwidth as message size grows. Key fields in the output: #### Results on an 4-node MI355X Cluster ``` # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 536870912 134217728 float sum -1 2945.0 182.30 353.20 0 2947.0 182.17 352.96 0 1073741824 268435456 float sum -1 5452.6 196.92 381.54 0 5446.2 197.16 381.99 0 2147483648 536870912 float sum -1 10806 198.72 385.03 0 10817 198.52 384.63 0 4294967296 1073741824 float sum -1 21843 196.63 380.97 0 21846 196.60 380.91 0 8589934592 2147483648 float sum -1 44986 190.95 369.96 0 44930 191.19 370.42 0 # Errors with asterisks indicate errors that have exceeded the maximum threshold. # Out of bounds values : 0 OK # Avg bus bandwidth : 374.161 # # Collective test concluded: all_reduce_perf ``` * `**algbw**` — algorithm bandwidth: message size divided by time. Reflects how quickly one collective operation completes. * `**busbw**` — bus bandwidth: `algbw` corrected for the number of ranks. Better reflects peak hardware utilization. * `**#wrong**` — should be `0`. Any non-zero value indicates a data correctness error. * `**Avg bus bandwidth**` — average `busbw` across all message sizes. Useful as a single summary figure for comparison. A healthy cluster shows `busbw` increasing steadily with message size and leveling off at a stable peak at larger sizes. Nodes with degraded interconnect will show lower `busbw` or fail to complete. Target `busbw` values will be dependent on your cluster architecture. *** ### Training Benchmarks (MFU / TFLOPS) Beyond raw interconnect bandwidth, it's often useful to measure end-to-end training performance — throughput (tokens/sec), compute utilization (TFLOPS), and Model FLOPs Utilization (MFU). For this, TensorWave recommends the [Primus Auto Benchmark tool](https://github.com/AMD-AGI/Primus/blob/main/tools/auto_benchmark/Primus_Auto_Benchmark_README.md) from AMD, which runs standardized model configs on the Megatron or TorchTitan backend and automatically generates a metrics summary table. The tool runs inside the `rocm/primus` container on a single GPU node. It is interactive: after launching, it prompts you to choose a backend, auto-detects the GPU (MI300X / MI355X), and lets you select which model config(s) to run. Selected configs run sequentially, with logs and metrics written to the `results/` directory. > **Note:** MFU is reported by the **TorchTitan** backend. The **Megatron** backend reports TPS, TFLOPS, memory utilization, and step time. Choose the backend based on the metric you need. #### Running the benchmark Because the tool runs interactively in Docker, first allocate a GPU node and open a shell on it. From a login pod, allocate a node and find its Slurm node name: ```bash salloc -N 1 --gpus-per-node=8 squeue -u $USER ``` Then SSH to the allocated worker pod by its node name: ```bash ssh # example: ssh tus1-p1-g1 ``` Pull the Primus container: ```bash docker pull rocm/primus:v26.5 ``` Launch the container: ```bash docker run -it --rm \ --gpus all \ --network host \ --ipc host \ -v $HOME/.ssh:/root/.ssh \ --name primus \ rocm/primus:v26.5 ``` Change into the auto-benchmark directory and launch the interactive tool: ```bash cd /workspace/Primus/tools/auto_benchmark/ bash run_primus_autobenchmark.sh ``` Follow the on-screen prompts to select a backend, model config(s), and any parameter overrides. You will be asked for a HuggingFace token to pull model and tokenizer assets. #### Reading the results When all runs finish, the tool automatically generates a metrics table. Logs and the exact configs used are preserved for reproducibility under `/workspace/Primus/tools/auto_benchmark/results/logs_/`. | Metric | Megatron | TorchTitan | Meaning | | -------------- | -------- | ---------- | -------------------------------------------------------- | | **TPS** | ✓ | ✓ | Tokens processed per second. | | **TFLOPS** | ✓ | ✓ | Achieved compute throughput. | | **MFU** | — | ✓ | Model FLOPs Utilization — achieved vs. theoretical peak. | | **Memory (%)** | ✓ | ✓ | Peak GPU memory utilization. | | **Time (ms)** | ✓ | — | Per-step elapsed time. | *** ### Preflight Smoke Test `preflight` is a fast, self-contained health check meant to be dropped into the **top of a job** to efficiently validate hardware health before your real workload starts. It runs the pre-installed PyTorch environment at `/opt/venv` and exercises three things in under a minute: 1. **matmul** — every GPU runs a `bf16` GEMM; validates the result is finite and measures throughput (TFLOPs). 2. **intra-node RCCL** — an `all_reduce` within each node's local GPUs; measures intra-node bus bandwidth. 3. **inter-node RCCL** — an `all_reduce` across the full allocation; measures cross-node bus bandwidth. **Only runs when launched across more than one node** (see below). Each measurement must clear two bars: a per-SKU **absolute floor** and a **peer-relative** check (no GPU/link may sit more than a configurable tolerance below the fleet median), which catches a single throttled GPU or degraded link regardless of hardware SKU. On any failure `preflight` exits non-zero, so a `set -e` job **aborts before wasting the allocation**. The launcher lives at `/opt/tw/examples/bin/preflight` and a ready-to-copy job is provided at `/opt/tw/examples/libexec/preflight.sbatch`. #### Injecting it into your job Add a single line near the top of your sbatch script, before your workload. Because the script uses `set -e`, a failed preflight stops the job before your workload runs: ```bash #!/bin/bash #SBATCH --nodes=2 #SBATCH --gpus-per-node=8 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=48 set -euo pipefail # Preflight: validates GPUs + RCCL across the whole allocation, then exits. srun --ntasks-per-node=1 /opt/tw/examples/bin/preflight # --- your real workload runs only if preflight passed ----------------------- srun ... my_training_job ``` You do not need to configure `torchrun`, master addresses, or GPU counts — the launcher detects the allocation from the SLURM environment and bootstraps `torch.distributed.run` for you. #### Supported launch configurations | Invocation | Nodes exercised | Phases run | | ------------------------------------------------------------------------ | ---------------------------------------- | ---------------------- | | `/opt/tw/examples/bin/preflight` (bare, in the batch script) | the single node the batch script runs on | matmul + intra | | `srun --ntasks-per-node=1 /opt/tw/examples/bin/preflight` (1-node alloc) | 1 | matmul + intra | | `srun --ntasks-per-node=1 /opt/tw/examples/bin/preflight` (N-node alloc) | all N | matmul + intra + inter | Key rules: * **Multi-node RCCL runs only under `srun` across >1 node.** A bare `preflight` invocation (or a single-node allocation) stays single-node and never hangs waiting for peers that will not join. * **Launch with `--ntasks-per-node=1`.** The launcher itself spawns one worker per GPU on each node; you only need one launcher task per node. * **Multi-node runs require full 8-GPU nodes.** A partial node makes the inter-node comparison meaningless, so `preflight` hard-fails a multi-node run if any node has fewer than 8 GPUs. Partial-GPU checks (e.g. 2 or 4 GPUs) are supported **single-node only**. #### Options and configuration Any arguments after the launcher are forwarded to the check. Useful flags: | Flag | Effect | | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `--warn-only` | Report failures but always exit `0` (does not abort the job). Does **not** bypass the 8-GPU-per-node topology requirement. | | `--json` | Emit a machine-readable JSON summary instead of the human tables. | | `--skip-matmul` / `--skip-intra` / `--skip-inter` | Skip a phase. | | `--matmul-size` / `--matmul-iters` / `--coll-bytes` / `--coll-iters` | Override workload sizing. | | `--min-tflops` / `--min-intra-busbw` / `--min-inter-busbw` / `--outlier-tol` | Override thresholds at runtime. | # Common Issues #### Full `/home` directory As `/home` approaches full-capacity, healthchecks will start to drain worker pods to avoid breaking in-flight jobs with 'no space left on device' errors. To clean up files and free disk space, use `du`: `du -h -d1 /home/$USER | sort -h` This lists the size of all files+directories in the current user's home directory (including hidden files). Users can then recursively descend from there, replacing `/home/$USER` with diretories further down the tree to explore and clean-up files. Admins can explore the entire weka tree with `sudo du -sh $(ls -A /home)`. # Health Overview The Kubernetes section of the **Health** tab gives you a glance summary of your cluster's Kubernetes workload health. Unlike the Kubernetes tab, which lets you drill into individual resources, this view is designed for quick status checks and early problem detection. To access it, navigate to your cluster and select the **Health** tab. The Kubernetes section will appear on the page alongside other cluster health information. *** ### Namespace Filter A **Namespace** dropdown at the top of the section lets you filter all cards by a specific Kubernetes namespace. By default it is set to **All Namespaces**, which aggregates metrics across the entire cluster. Select a specific namespace to scope the view down to the workloads running within it. *** ### Node Health The **Node Health** card shows the readiness state of every node in your cluster. | Metric | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------ | | **Ready** | The number of nodes that are healthy and available to schedule workloads. Displayed in green. | | **Not Ready** | The number of nodes that are unhealthy or unavailable. Displayed in red. Any non-zero value here warrants investigation. | | **Readiness** | A percentage representing the proportion of nodes currently in a Ready state, shown as a progress bar. | | **Total Nodes** | The total number of nodes across the cluster, shown below the progress bar. |
*** ### Pod Status The **Pod Status** card displays a donut chart summarizing the state of all pods in the cluster (or selected namespace). The total pod count is shown at the center of the chart. The legend below it breaks pods down by phase: | Phase | Description | | ------------- | ----------------------------------------------------------------------- | | **Running** | Pods that are actively running and have at least one container started. | | **Succeeded** | Pods that have completed successfully and exited cleanly. | > **Note:** If pods appear in unexpected states such as `Pending`, `Failed`, or `Unknown`, they will also appear in the chart. Investigate any pods not in a `Running` or `Succeeded` state.
*** ### Deployments The **Deployments** card shows a summary of the health of all Kubernetes Deployments in the cluster. The card displays the number of available Deployments out of the total (e.g. `54 / 54 available`). When all Deployments are healthy, the message **All deployments healthy** is shown. If any Deployments are degraded or unavailable, the count will reflect this and the healthy message will not appear.
*** ### Slurm Node States The **Slurm Node States** card summarizes the state of nodes managed by the Slurm workload manager running alongside Kubernetes (if applicable) on your cluster. The header shows the number of **active** nodes out of the **total** Slurm managed nodes, along with a progress bar. Below that, each state is broken down individually: | State | Indicator | Description | | ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Idle** | Green | Nodes that are available and waiting to be assigned a job. | | **Allocated** | Blue | Nodes currently assigned to one or more running jobs. | | **Mixed** | Green | Nodes that are partially allocated, with some resources in use and some available. | | **Down** | Red | Nodes that Slurm has marked as unavailable. A non-zero value here may indicate a hardware or configuration issue. | | **Drained / Draining** | Orange | Nodes that have been manually taken out of service. A draining node is finishing existing jobs before going offline, a drained node is fully offline and accepting no new work. | *** ### Slurm Job Queue The **Slurm Job Queue** card shows the number of jobs currently in the Slurm queue, broken down by state. The total job count is displayed prominently at the top of the card. | State | Indicator | Description | | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Running** | Green | Jobs that are actively executing on one or more nodes. | | **Pending** | Orange | Jobs that are queued and waiting for resources to become available. A high pending count may indicate resource contention or that nodes are down. | | **Other** | Gray | Jobs in any other state, such as completing, suspended, or held. | ### Note: Downloading Your Kubeconfig Each cluster page includes a **Kubeconfig** download button in the top header, next to the cluster name. Clicking this button downloads a kubeconfig file for your cluster, which you can use to authenticate and interact with it via `kubectl` from your local machine. > **Note:** Keep your kubeconfig file secure. It contains credentials that grant access to your cluster. # Nodes The Nodes section within the Kubernetes tab gives you a detailed look at every node in your cluster, including its current status, role, Kubernetes version, and hardware resource specifications such as GPU count, CPU cores, and available memory. To access this view, navigate to your cluster, select the **Kubernetes** tab, then choose **Nodes** from the left sidebar. *** ### Summary Cards Four summary cards at the top of the Nodes view give you a quick breakdown of node states across the cluster: | Card | Description | | ----------------- | ------------------------------------------------------------------------------------------------ | | **Ready** | The number of nodes in a `Ready` state and available to schedule workloads. | | **Not Ready** | The number of nodes that are not ready. This may indicate a problem that requires investigation. | | **Control Plane** | The number of nodes serving as control plane nodes, responsible for managing the cluster state. | | **Workers** | The number of worker nodes available to run your workloads. | *** ### Node Table Below the summary cards, a table lists every node in the cluster with the following columns: | Column | Description | | ------------- | ----------------------------------------------------------------------------------------------- | | **Hostname** | The node's hostname. Click the copy icon alongside it to copy the hostname to your clipboard. | | **Status** | The current Kubernetes node status. See Node Statuses below for possible values. | | **Role** | Whether the node is a `worker` or a control plane node. | | **Version** | The Kubernetes version running on the node (e.g. `v1.34.7+rke2r1`). | | **GPU** | The number of GPUs detected on the node. A `-` indicates a CPU-only node with no GPU resources. | | **CPU** | The total number of CPU cores on the node. | | **Memory** | The total memory available on the node, displayed in kibibytes (Ki). | | **More Info** | Click the info icon to view additional details for that node. | *** ### Node Statuses The **Status** column reflects the Kubernetes node condition reported by the control plane: | Status | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | **Ready** | The node is healthy and ready to accept workloads. | | **Not Ready** | The node is reachable but is failing one or more health checks. Workloads will not be scheduled to this node. | | **Unknown** | The control plane has lost contact with the node. This typically means the node has gone offline or the kubelet has stopped reporting. | # Workloads The Workloads section of the Kubernetes tab gives you visibility into every workload resource running on your cluster. Use the left sidebar to navigate between resource types: Pods, Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs. All views share a **Namespace** dropdown at the top of the page, which lets you filter results to a specific namespace. By default it is set to **All Namespaces**. *** ### Pods A Pod is the smallest deployable unit in Kubernetes. Each Pod represents one or more containers running together on a node. The Pods view lists every Pod across your cluster.
ColumnDescription
Pod NameThe name of the Pod. Click the copy icon alongside it to copy the name to your clipboard.
NamespaceThe namespace the Pod belongs to.
StatusThe current lifecycle phase of the Pod. A green dot indicates the Pod is running normally.
NodeThe node the Pod has been scheduled and is running on.
CreatedThe date and time the Pod was created, including timezone.
#### Pod Status Values
StatusDescription
RunningThe Pod has been bound to a node and all containers have been started.
PendingThe Pod has been accepted by the cluster but is not yet running. It may be waiting for resources or for its container image to be pulled.
SucceededAll containers in the Pod have completed successfully and will not be restarted.
FailedOne or more containers in the Pod terminated with an error.
UnknownThe state of the Pod could not be determined, usually due to a communication error with the node.
*** ### Deployments A Deployment manages a set of identical, stateless Pods and ensures the desired number of replicas are running at all times. If a Pod fails, the Deployment controller automatically replaces it.
ColumnDescription
NameThe name of the Deployment. Click the copy icon alongside it to copy the name to your clipboard.
NamespaceThe namespace the Deployment belongs to.
ReadyThe number of replicas currently ready out of the total desired, displayed as ready/desired (e.g. 1/1).
AvailableThe number of replicas that are available to serve traffic.
UpdatedThe number of replicas that have been updated to the latest desired state.
CreatedThe date and time the Deployment was created, including timezone.
*** ### StatefulSets A StatefulSet manages Pods that require stable identities and persistent storage, such as databases. Unlike Deployments, each Pod in a StatefulSet has a unique, stable network identity and storage that persists across restarts.
ColumnDescription
NameThe name of the StatefulSet. Click the copy icon alongside it to copy the name to your clipboard.
NamespaceThe namespace the StatefulSet belongs to.
ReadyThe number of replicas currently ready out of the total desired, displayed as ready/desired (e.g. 3/3). A value like 0/1 indicates replicas are desired but none are currently ready.
CurrentThe number of Pods that currently exist for this StatefulSet.
UpdatedThe number of Pods that have been updated to the latest version.
CreatedThe date and time the StatefulSet was created, including timezone.
*** ### DaemonSets A DaemonSet ensures that a copy of a specific Pod runs on every node in the cluster (or a subset of nodes). DaemonSets are commonly used for cluster-wide services such as log collectors, monitoring agents, and GPU device plugins.
ColumnDescription
NameThe name of the DaemonSet. Click the copy icon alongside it to copy the name to your clipboard.
NamespaceThe namespace the DaemonSet belongs to.
DesiredThe number of nodes that should be running the DaemonSet Pod.
CurrentThe number of nodes that are currently running the DaemonSet Pod.
ReadyThe number of nodes running the Pod in a Ready state.
AvailableThe number of nodes where the Pod is available and able to serve its intended function.
CreatedThe date and time the DaemonSet was created, including timezone.
> **Note:** If the **Ready** or **Available** count is lower than **Desired**, it may indicate that some nodes are unable to run the Pod. This could be caused by resource constraints, node taints, or scheduling issues. *** ### Jobs A Job creates one or more Pods to carry out a specific task and tracks whether those Pods complete successfully. Once all Pods finish, the Job is considered complete. Jobs are commonly used for one-off tasks such as installing modules or running database migrations.
ColumnDescription
Job NameThe name of the Job. Click the copy icon alongside it to copy the name to your clipboard.
Job StatusThe current status of the Job. A green dot alongside Complete indicates the Job finished successfully.
NamespaceThe namespace the Job belongs to.
KindThe resource kind associated with the Job, where applicable.
CreatedThe date and time the Job was created, including timezone.
*** ### CronJobs A CronJob creates Jobs on a repeating schedule, similar to a cron task in Linux. CronJobs are useful for recurring automated tasks such as backups, report generation, or cleanup operations. If no CronJobs have been configured on your cluster, the view will display **No Results Found**. This is expected for clusters that do not have any scheduled recurring workloads defined. # Filing Support Tickets # Syncing Public Keys # Managing team member access # Self-serve inviting users # Slurm- Old Slurm provides a multi-tenant framework for managing compute resources and jobs that span large clusters. ### Overview TensorWave Slurm combines the power of **Slurm**, the industry-standard workload manager for HPC and AI, with the flexibility of a **Kubernetes-native orchestration layer**.\ This integration delivers a modern, multi-tenant environment that scales seamlessly across AMD Instinct GPU clusters—enabling teams to run distributed training, fine-tuning, and simulation workloads without managing the underlying infrastructure. With TensorWave, you get the familiar Slurm interface running on top of a cloud-native control plane that provides automated scheduling, easy scaling, and container-based execution. *** ### Why Slurm on Kubernetes? Traditional Slurm deployments were designed for static on-prem clusters.\ TensorWave modernizes that model by running Slurm **inside Kubernetes**, unlocking: * **Scalable compute pools** — resize your slurm cluster within your K8s environment. * **Container-native workflows** — integrate directly with your existing Docker or Enroot environments. * **Multi-tenant isolation** — each user or team runs in a secure namespace with defined resource limits. The result is a unified, cloud-native scheduling experience that bridges HPC scalability with Kubernetes reliability. *** ### Quickstart Example #### **1. Connect to Your Login Node** Each Slurm environment provides a **login node**, your interactive entry point for running Slurm commands. Behind the scenes, this login node runs as a managed **Kubernetes pod**, with the same Slurm interface (`srun`, `sinfo`, `sbatch`), and the benefits of cloud-native orchestration. ```bash ssh @ ``` Once connected, you’ll have access to all standard Slurm utilities and your team’s partitions.\ From here, you can submit jobs, monitor queues, and launch multi-node workloads just as you would on a traditional HPC cluster. *** #### 2. Inspect Available Resources List available partitions and node states: ```bash sinfo ``` Example: ``` PARTITION AVAIL TIMELIMIT NODES STATE NODELIST gpuworker* up infinite 1024 idle compute-[1-1024] ``` Even though these nodes are dynamically managed by Kubernetes, the Slurm CLI remains identical to traditional HPC clusters. *** #### 3. Launch a Multi-Node Job To verify connectivity and RDMA functionality, run a distributed RCCL test across four nodes (32 GPUs total): ```bash srun -N4 \ --gpus-per-node=8 \ /opt/rccl-tests/all_reduce_perf -g 8 -b 1g -e 16G -f 2 ``` Slurm automatically handles: * GPU and node allocation * Network interface binding * MPI coordination ### Containers and More Info Often, users will want to run their HPC payloads in containers. You can learn more about this in the Pyxis Quickstart section. # Running Containerized Jobs Containers provide environment isolation, ensure consistent environments across nodes, simplify dependency management, and let you reproduce results reliably. In short, they are a fantastic way to manage dependencies for Slurm jobs. TensorWave's Slurm provides several methods for running jobs in containerized environments. * [pyxis/enroot](/slurm-old/running-containerized-jobs/running-containerized-jobs-in-pyxis) * [apptainer](/slurm-old/running-containerized-jobs/running-containerized-jobs-with-apptainer) * docker **We recommend using Pyxis** as it's the most performant. Docker is provided due to popular demand; however, we **highly discourage using Docker** due to issues with performance and how the Docker daemon interacts with Slurm. Apptainer provides a similar interface/user experience to Docker, while avoiding a lot of the problems the Docker daemon causes. # Running Containerized Jobs in Pyxis TensorWave Slurm integrates Pyxis, a container runtime plugin for Slurm that enables users to run containerized workloads directly within their jobs. Pyxis is a Slurm plugin that uses the Enroot container runtime under the hood. This integration lets you launch distributed AI or HPC jobs inside optimized ROCm containers while maintaining full GPU, RDMA, and filesystem performance. *** #### **Running Your First Containerized Job** In this example, you’ll run a **PyTorch matmul test** using Pyxis.\ This will demonstrate how to pull images from Docker Hub, show how to mount your code into the container, and verify that your containerized environment can access full GPU performance. 1. Create a Python script to measure matmul performance. Copy the following code block into a file named `torch_matmul.py`. 
import torch

   device = torch.device("cuda:0")
   dtype = torch.float16
   torch.set_default_device(device)

   print(f"Device: {torch.cuda.get_device_name(device)}")

   sizes = [1024, 2048, 4096, 8192]
   iters = 50

   for n in sizes:
       a = torch.randn((n, n), dtype=dtype)
       b = torch.randn((n, n), dtype=dtype)

       start = torch.cuda.Event(enable_timing=True)
       end = torch.cuda.Event(enable_timing=True)

       # warmup
       for _ in range(2):
           torch.matmul(a, b)
       torch.cuda.synchronize()

       start.record()
       for _ in range(iters):
           c = torch.matmul(a, b)
       end.record()
       torch.cuda.synchronize()

       elapsed_ms = start.elapsed_time(end)
       elapsed_s = elapsed_ms / 1e3

       # FLOPs for matmul ≈ 2 * n^3
       total_flops = 2 * n**3 * iters
       tflops = total_flops / elapsed_s / 1e12

       print(f"n={n:5d}  {tflops:6.2f} TFLOPs")
2. **Create a new job script** named `torch-matmul-pyxis.sbatch`: 
#!/bin/bash
   #SBATCH --job-name=torch_matmul-pyxis
   #SBATCH --output=jid-%j.name-%x.log
   #SBATCH --gpus-per-node=8
   #SBATCH -N1

   # Script created in step 1.
   MATMUL_PY="$PWD/torch_matmul.py" 
   # pytorch-rocm image from Docker Hub, published by AMD
   CONTAINER_IMAGE='rocm/pytorch:rocm7.1.1_ubuntu22.04_py3.10_pytorch_release_2.9.1'
   CONTAINER_NAME="pytorch_matmul_test"

   # Download the image and instantiate the container
   srun --container-name=$CONTAINER_NAME --container-image=$CONTAINER_IMAGE true

   # Mount torch_matmul.py into the container and run the benchmark
   srun --container-writable \
     --container-name=$CONTAINER_NAME \
     --container-mounts="$MATMUL_PY:/root/torch_matmul.py" \
     /opt/venv/bin/python /root/torch_matmul.py

   # Save the image to disk for use later
   srun --container-name=$CONTAINER_NAME \
     --container-save=$PWD/torch-matmul.sqsh \
     true
3. **Submit the job.** Here's an example run: ```shellscript $ sbatch torch-matmul-pyxis.sbatch Submitted batch job 90 $ tail -f jid-90.name-torch_matmul.log pyxis: importing docker image: rocm/pytorch:rocm7.1.1_ubuntu22.04_py3.10_pytorch_release_2.9.1 pyxis: imported docker image: rocm/pytorch:rocm7.1.1_ubuntu22.04_py3.10_pytorch_release_2.9.1 Device: AMD Instinct MI325X n= 1024 144.42 TFLOPs n= 2048 466.27 TFLOPs n= 4096 640.01 TFLOPs n= 8192 763.36 TFLOPs pyxis: exported container pyxis_90_pytorch_matmul_test to /home/bkitor@tensorwave.com/snpyxis/torch-matmul.sqsh ^C $ ls jid-90.name-torch_matmul.log torch-matmu-pyxis.sbatch torch-matmul.sqsh torch_matmul.py ``` To highlight some of the key features of the `torch-matmul-pyxis.sbatch` file: * Container images are automatically pulled from Docker Hub * Named containers are persistent across `srun` invocations. * On line 14, the `"pytorch_matmul_test"` container is created. * On line 18, the container is invoked to run the benchmark. * On line 23, the container is saved to disk for reuse later. * On line 19: Use the `--container-mounts` flags to bridge data into and out of the container * On line 24: The `--container-save` flag writes the `"pytorch_matmul_test"` container as a squashfs file. *** #### **Using a Pre-Staged SquashFS Image** Instead of pulling a container from a registry, you can point Slurm directly to a **pre-staged SquashFS (`.sqsh`) image**.\ This is often faster and preferred for large models or shared environments, as using a local `.sqsh` file avoids repeated network pulls and ensures consistent environments across jobs. To modify the `torch-matmul-pyxis.sbatch` example to use a squashfs file, modify the following line: ```bash CONTAINER_IMAGE='.sqsh' ``` Squashfs files can be generated by the `--container-save` flag, or generated via `enroot`. See the [Managing Container Images with Enroot](/slurm-old/managing-container-images-with-enroot)page for more info on generating `.sqsh` images. *** #### **Pyxis Flags** Pyxis extends Slurm with several container-related flags that control how your job interacts with the container environment. Below are the most commonly used options: | Flag | Description | | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `--container-image` | Specifies the container to run. Accepts Docker/OCI URLs or local `.sqsh` images. | | `--container-writable` | Makes the container filesystem writable during execution. Useful for logs, checkpoints, or temporary files. | | `--container-mounts=/src:/dst[,/src2:/dst2]` | Binds local or shared directories into the container. Multiple mounts can be separated by commas. | | `--container-workdir=/path` | Sets the working directory inside the container (defaults to `/`). | | `--container-name=` | Assigns a name to the running container instance, useful for debugging or monitoring. | | `--container-save=PATH` | Save the container state to a squashfs file on the remote host filesystem. | | `--container-remap-root` | By default, containers are launched in userland. This flag launches the container as root. It's useful for images that are expected to run as root. | *** #### **Learn More** For advanced configuration options and the full list of supported flags, see the official containers documentation from SchedMD:\ # Running Containerized Jobs with Apptainer * `apptainer pull docker://:` to pull and cache an image from Docker Hub * Images are cached under `$HOME/.apptainer`, so images pulled on a login pod are available on the worker nodes * Due to login pod security settings, Apptainer build/exec/shell can only be run on the worker pods #### **Running Your First Containerized Job** In this example, you’ll run a **PyTorch matmul test** using Apptainer.\ This will demonstrate how to pull images from Docker Hub, show how to mount your code into the container, and verify that your containerized environment can access full GPU performance. 1. Create a Python script to measure PyTorch's matrix multiplication performance. Copy the following code block into a file named `torch_matmul.py`. 
import torch

   device = torch.device("cuda:0")
   dtype = torch.float16
   torch.set_default_device(device)

   print(f"Device: {torch.cuda.get_device_name(device)}")

   sizes = [1024, 2048, 4096, 8192]
   iters = 50

   for n in sizes:
       a = torch.randn((n, n), dtype=dtype)
       b = torch.randn((n, n), dtype=dtype)

       start = torch.cuda.Event(enable_timing=True)
       end = torch.cuda.Event(enable_timing=True)

       # warmup
       for _ in range(2):
           torch.matmul(a, b)
       torch.cuda.synchronize()

       start.record()
       for _ in range(iters):
           c = torch.matmul(a, b)
       end.record()
       torch.cuda.synchronize()

       elapsed_ms = start.elapsed_time(end)
       elapsed_s = elapsed_ms / 1e3

       # FLOPs for matmul ≈ 2 * n^3
       total_flops = 2 * n**3 * iters
       tflops = total_flops / elapsed_s / 1e12

       print(f"n={n:5d}  {tflops:6.2f} TFLOPs")
2. **Create a new job script** named `torch-matmul-apptainer.sbatch`: 
#!/bin/bash
   #SBATCH --job-name=torch_matmul-apptainer
   #SBATCH --output=jid-%j.name-%x.log
   #SBATCH --gpus-per-node=8
   #SBATCH -N1

   # Script created in step 1.
   MATMUL_PY="$PWD/torch_matmul.py" 
   # pytorch-rocm image from Docker Hub, published by AMD
   CONTAINER_IMAGE='rocm/pytorch:rocm7.1.1_ubuntu22.04_py3.10_pytorch_release_2.9.1'
   CONTAINER_SAVE="./rocm+pytorch+rocm7.1.1_ubuntu22.04_py3.10_pytorch_release_2.9.1.sif"

   # pull the image from Docker Hub and save to disk
   apptainer pull "$CONTAINER_SAVE" "docker://$CONTAINER_IMAGE"

   # Mount torch_matmul.py into the container and run the benchmark
   srun apptainer exec "$CONTAINER_SAVE" \
     /opt/venv/bin/python "$MATMUL_PY"
3. **Submit the job.** Here's an example run: ```shellscript $ sbatch torch-matmul-apptainer.sbatch Submitted batch job 4241 $ tail -f jid-4241.name-torch_matmul-apptainer.log INFO: Converting OCI blobs to SIF format INFO: Starting build... INFO: Fetching OCI image... INFO: Extracting OCI image... 2026/05/05 21:06:48 warn rootless{usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper} ignoring (usually) harmless EPERM on setxattr "security.capability" INFO: Inserting Apptainer configuration... INFO: Creating SIF file... Device: AMD Instinct MI325X n= 1024 145.11 TFLOPs n= 2048 448.87 TFLOPs n= 4096 630.48 TFLOPs n= 8192 753.88 TFLOPs ``` To highlight some of the key features of the `torch-matmul-apptainer.sbatch` file: * On line 14, the rocm-pytorch image is pulled from Docker Hub and saved to disk as a `.sif` file. * `docker://` is prepended to the image to tell Apptainer to pull the image from Docker Hub. Other container repositories can be configured and run `apptainer pull --help` for details. * Pulling images before running parallel jobs is highly recommended to avoid a [thundering herd problem](https://en.wikipedia.org/wiki/Thundering_herd_problem) pulling remote images. * The saved container is launched with `apptainer exec` on line 17. * By default, the `/home` directory is mounted into the container. So if `torch_matmul.py` is saved somewhere in the home directory, it'll be available in the container. #### Useful apptainer commands | Command | Description | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | `apptainer exec ` | Execute 'command' in the container. | | `apptainer shell ` | Open a shell in the container. This is useful for interactive sessions, `srun --gpus=8 --pty apptainer shell ` | | `apptainer pull [output file] ` | Save an image locally as an .sif file. | | `apptainer build ` | Build an apptainer image from a .spec file. Useful for injecting modifications into a docker image. | | | | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `--bind src[:dst[:opts]]` | Mount files into the container. | | `--fakeroot` | By default, containers are launched in userland. This flag launches the container as root. It's useful for images that are expected to run as root. | # Managing Container Images with Enroot TensorWave Slurm uses Enroot as its lightweight, high-performance container runtime for HPC and AI workloads. Unlike traditional container engines, Enroot runs entirely in user space with no privileged daemons or root access required, making it ideal for multi-tenant and secure compute environments. Enroot executes standard **Docker or OCI images** as unprivileged user processes, unpacking each image into an isolated filesystem that can be shared across nodes.\ It preserves direct access to GPUs, high-speed interconnects, and local storage, ensuring your jobs run efficiently within containers. You don’t need to run Enroot commands directly; TensorWave Slurm handles that automatically through **Pyxis**, which integrates Enroot with familiar Slurm tools like `srun` and `sbatch`. See the [Running Containerized Jobs in Pyxis](/slurm-old/running-containerized-jobs/running-containerized-jobs-in-pyxis) page for info on how to run containerized workloads.\ Together, they allow you to launch containerized jobs using the same workflow you already know, with the added benefits of portability and reproducibility. *** #### **Optional: Importing an Image with Enroot** Although Pyxis automatically handles Enroot under the hood, you can manually import container images for debugging or pre-caching. Due to Kubernetes security-permissioning constraints on the login pods, container image management has to happen on the worker pods. This means operations including `docker build`, `docker pull`, and `enroot import`, must happen in a Slurm-job context. So, to build a .sqsh image locally, run `docker build` and `enroot import` with `srun`: ```bash IMG=":" # srun command to 'build' and 'import' $IMG srun --pty -N1 bash -c "docker build -f -t $IMG . && enroot import dockerd://$IMG" ``` To pull pre-built images, you can authenticate via Docker and pull/import the container image: ```bash # Login into Docker docker login -u -p # Pull and squash Docker image srun --pty -N1 bash -c "enroot import dockerd://:" ``` This workflow downloads the image, converts it into an Enroot container bundle, and runs it as an unprivileged user process. You’ll typically never need to do this when submitting jobs through Pyxis, but it’s a useful way to verify container contents or pre-stage larger images. #### Resources Enroot usage guide: # Running Jobs With Modules Software modules are provided for commonly used software packages. #### Modules Quickstart As an example, we provide a module for [Huggingface's Transformer Reinforcement Learning](https://github.com/huggingface/trl/tree/v0.28.0) package. So getting a working TRL environment is as easy as `module load trl`. We also provide a sample sbatch script that uses the TRL module `/opt/examples/libexec/trl-module.sbatch` : {% code title="/opt/examples/libexec/trl-module.sbatch" lineNumbers="true" expandable="true" %} ```bash #!/usr/bin/bash #SBATCH --job-name=trl-finetuner #SBATCH --output=jid-%j.name-%x.log #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=48 #SBATCH --gpus-per-node=8 #SBATCH --time=01:00:00 #SBATCH --nodes=4 set -exuo pipefail module load trl TRLFT_PY="/opt/examples/scripts/trl_tune/trl_tune.py" GPUS_PER_NODE=8 MASTER_ADDR=$(hostname) MASTER_PORT=6000 if [[ -d $HOME/.cache/huggingface/datasets/mlech26l___shell-helper ]]; then HF_OFFLINE=1 else HF_OFFLINE=0 fi srun bash < /path/to/script.sbatch`. {% code expandable="true" %} ``` $ sbatch /opt/examples/libexec/trl-module.sbatch Submitted batch job 268 $ tail -f jid-268.name-trl-finetuner.log ++ hostname + MASTER_ADDR=tus1-p13-g2 + MASTER_PORT=6000 + [[ -d /home/tensorwave/.cache/huggingface/datasets/mlech26l___shell-helper ]] + HF_OFFLINE=1 + srun bash MY IP is tus1-p13-g2 and am I host? 1 and what is master addr tus1-p13-g2 MY IP is tus1-p14-g24 and am I host? 0 and what is master addr tus1-p13-g2 MY IP is tus1-p14-g37 and am I host? 0 and what is master addr tus1-p13-g2 MY IP is tus1-p16-g17 and am I host? 0 and what is master addr tus1-p13-g2 Rank: 0 out of 32 Number of gpus available: 8 GPU 0: AMD Instinct MI325X GPU 1: AMD Instinct MI325X GPU 2: AMD Instinct MI325X GPU 3: AMD Instinct MI325X GPU 4: AMD Instinct MI325X GPU 5: AMD Instinct MI325X GPU 6: AMD Instinct MI325X GPU 7: AMD Instinct MI325X Loading model: LiquidAi/LFM2.5-1.2B-Instruct ... Found the latest cached dataset configuration 'default' at /home/tensorwave/.cache/huggingface/datasets/mlech26l___shell-helper/default/0.0.0/bf4e04b465240544350f49c89cd108c35698f588 (last modified on Sat Feb 14 08:59:42 2026). Launching training {'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'} {'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'} {'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'} {'loss': '1.294', 'grad_norm': '0.9996', 'learning_rate': '1.85e-05', 'entropy': '1.288', 'num_tokens': '8.023e+06', 'mean_token_accuracy': '0.7068', 'epoch': '0.3067'} {'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'} {'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'} {'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'} {'loss': '0.9138', 'grad_norm': '0.837', 'learning_rate': '1.696e-05', 'entropy': '0.9295', 'num_tokens': '1.604e+07', 'mean_token_accuracy': '0.7676', 'epoch': '0.6135'}k ``` {% endcode %} #### Module Management To explore available modules, run `module avail` or `module spider`. Avail gives a simplified output, while spider is more detailed and useful for sorting out module dependencies. If you need a specific piece of software, contact us and we can provide a module that fits your needs. #### Resources Lmod user guide: # Debugging Workloads and Interactive Jobs Often dropping into a shell is the easiest way to hammer out development / debugging work. To get a shell access to a node, use a combination of `salloc` and `srun` commands. ``` tensorwave@tensorwave.com@slurm-login-skip-8566547b9c-zdznk:~$ salloc -N 1 --gpus-per-node=8 salloc: Granted job allocation 269 salloc: Waiting for resource configuration salloc: Nodes tus1-p13-g41 are ready for job tensorwave@tensorwave.com@slurm-login-skip-8566547b9c-zdznk:~$ srun --pty bash tensorwave@tensorwave.com@tus1-p13-g41:~$ # <== bash shell on a worker node ``` `salloc` reserves a resource allocation. It creates a sub-shell tied to a SLRUM allocation, and allows users to call `srun` multiple times on the same set of resrouces. This provides much faster iteration time than calling `srun`/`sbatch` from a login node and having to wait for a resource allocation each time, especially for debugging multi-node jobs. `srun --pty bash` launches the bash shell on the worker pods. The `--pty` flag sets up a psudoshell, forwarding stdin, stdout, and stderr to the terminal. The `--pty` flag is also useful for `srun` commands with complex outputs like `docker build` or python tools with `tqdm` progress bars. # Custom Prolog and Epilog scripts Slurm's prolog and epilog features allow users to specify scripts to run on each node before the first job step/at job termination. This enables admins to perform tasks like system monitoring or cleanup in a non-disruptive way at fairly frequent intervals. TensorWave's managed Slurm solution provides `/mnt/customer/prolog.d` and `/mnt/customer/epilog.d` for cluster admins to add their custom prolog/epilog scripts, respectively. Per-node summaries of custom prolog/epilog script runs are logged in `/mnt/customer/logs`. Prolog/Epilog scripts are a powerful tool for managing a cluster, but there are also a few easy ways to 'shoot yourself in the foot' with them. If a Prolog/Epilog script returns a non-zero exit code, the node will be placed in DRAIN state, so if a buggy script is deployed, it can bring down the entire cluster. Prolog/Epilog scripts are run as the root user, this provides broad access for system monitoring, but can also enable disrupting running jobs if performing cleanup tasks. #### Test example of a custom prolog In this example, we have a prolog and an epilog script. Both scripts print an output to stdout. The epilog script returns 1 to simulate a failure event. ```bash tensorwave@tensorwave.com@slurm-login-skip-849dbcf5c-q7ffr:~$ sudo cat /mnt/customer/prolog.d/test-prolog-1.sh #!/usr/bin/env bash echo "Hello from job id $SLURM_JOB_ID on node $SLURM_NODENAME" tensorwave@tensorwave.com@slurm-login-skip-849dbcf5c-q7ffr:~$ sudo cat /mnt/customer/epilog.d/test-epilog-1.sh #!/usr/bin/env bash echo "Goodbye from job id $SLURM_JOB_ID on node $SLURM_NODENAME" exit 1 ``` To trigger the prolog/epilog, we submit an `srun` job. Since the `test-epilog-1.sh` 'fails', the node our job ran on drains. ```bash tensorwave@tensorwave.com@slurm-login-skip-849dbcf5c-q7ffr:~$ srun -N 1 --gpus-per-node=8 hostname tus1-p2-g6 tensorwave@tensorwave.com@slurm-login-skip-849dbcf5c-q7ffr:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST gpuworker* up infinite 1 drain tus1-p2-g6 gpuworker* up infinite 1 idle tus1-p2-g5 ``` Investigating logs, the prolog runs successfully, but since the epilog failed, the full output is saved in the logs. ```bash tensorwave@tensorwave.com@slurm-login-skip-849dbcf5c-q7ffr:~$ cat /mnt/customer/logs/prolog/tus1-p2-g6.log timestamp=2026-04-01T18:43:39Z script=/mnt/customer/prolog.d/test-prolog-1.sh exit_code=0 job_id=5995 job_user=tensorwave@tensorwave.com tensorwave@tensorwave.com@slurm-login-skip-849dbcf5c-q7ffr:~$ cat /mnt/customer/logs/epilog/tus1-p2-g6.log timestamp=2026-04-01T18:43:42Z script=/mnt/customer/epilog.d/test-epilog-1.sh exit_code=1 job_id=5995 job_user=tensorwave@tensorwave.com --- output --- Goodbye from job id 5995 on node ``` #### Refrences Slurm Prolog and Epilog Guide: # Healthchecks and SDebug Overview of TensorWave's health checks and SDebug hardware debugging tool. This tool is designed to make cluster admins life easier w\.r.t managing (un)healthy nodes. * NHC and TWHC health checks are frequently run by SLURM prolog + epilog scripts. * Use `sdebug info` to get a summary of the most recent health checks * Use `sdebug info --node ` to get in-depth health check details for a specific node *** #### Healthchecks TensorWave Slurm regularly runs health checks to ensure nodes are operating at peak performance. We've deployed [LBNL's Node Health Checks (NHC)](https://github.com/mej/nhc), as well as our own in-house Tensor Wave Health Check (TWHC) suite. Checks are performed before and after any job is run using SLURM prolog and epilog, and any nodes that don't pass muster are automatically drained to avoid disrupting operations. All health check results are saved to disk and can be investigated through the accompanying `sdebug` tool. #### Investigating Node Health with SDebug Get a high-level overview of cluster health with `sdebug info`. This summarizes the results of the most recent health checks. ``` tensorwave@tensorwave.com@slurm-login-skip-8566547b9c-qsnbh:~$ sdebug info check-ecc check-gpu-reset check-link-flap check-lldpd check-nhc check-rdma-links check-ulimits NODES ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ PASS PASS PASS PASS PASS PASS PASS tus1-p13-g[1-6,8,9,11-23,25-26,28-30,32-43,45-50,52-54,56-59,62,64],tus1-p14-g[1-31,33-49,51-52,54-61,63-64],tus1-p16-g[1-6,9-17,19-20,24-30,33-36,39,42-50,52,54-62,64] PASS PASS FAIL PASS PASS PASS PASS tus1-p13-g[10,24,55],tus1-p16-g[21-22,37-38,53] PASS PASS PASS PASS PASS FAIL PASS tus1-p13-g[7,61],tus1-p16-g[7,23,41,51] PASS FAIL PASS PASS PASS PASS PASS tus1-p14-g50 PASS PASS PASS PASS PASS PASS FAIL tus1-p14-g62,tus1-p16-g8 ``` You can use the `--node` flag to dive deeper into a node's specific healthcheck report. This gives more detailed logs for the specific health checks that failed. Here's an example report from `tus1-p14-g62` when its limits were set incorrectly: ``` tensorwave@tensorwave.com@slurm-login-skip-8566547b9c-qsnbh:~$ sdebug info --node tus1-p14-g62 Health Check Report for Node: tus1-p14-g62 ======================================================= TEST STATUS RETCODE TIMESTAMP ------------------------------------------------------- check-nhc PASS 0 2026-02-16T23:47:06Z check-ecc PASS 0 2026-02-16T23:47:07Z check-rdma-links PASS 0 2026-02-16T23:47:07Z check-link-flap PASS 0 2026-02-16T23:47:07Z check-ulimits FAIL 1 2026-02-16T23:47:07Z check-gpu-reset PASS 0 2026-02-16T23:47:07Z check-lldpd PASS 0 2026-02-16T23:47:07Z ======================================================= Failed Test Logs: ================================================================================ Test: check-ulimits Return Code: 1 Timestamp: 2026-02-16T23:47:07Z Log: -------------------------------------------------------------------------------- FAIL: ulimit check(s) failed: - max_locked_memory: expected 'unlimited', got '8192' max_locked_memory=8192 ================================================================================ ``` # Easy Porting: NVIDIA to AMD Guide ### Introduction With both AMD and NVIDIA establishing themselves as top offerings for AI compute, questions have arisen over the differences in software required to run on each. Real-world workloads can run on both types of hardware with little to no code changes, and we're excited to demonstrate this further today. We'll start by training an image classifier on the CIFAR-10 dataset in PyTorch on both NVIDIA and AMD. {% hint style="info" %} Learn more about the CIFAR-10 dataset [here](https://www.cs.toronto.edu/~kriz/cifar.html). {% endhint %} We'll then move on to a more practical use-case: fine-tuning Llama 3.1 8B on a corpus of SQL data. {% hint style="info" %} Learn more about Llama 3.1 [here](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct). {% endhint %} *** ### Training an Image Classification Model To start, you're going to need to install PyTorch locally. Install the appropriate version depending on your hardware. {% tabs %} {% tab title="ROCm" %} ```bash pip install requests pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.2/ ``` {% endtab %} {% tab title="CUDA" %} 
pip install requests
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
{% endtab %} {% endtabs %} {% hint style="info" %} This will be the only difference in process for this tutorial. {% endhint %} Next, navigate to the directory you'd like to set this tutorial up in. From there, create the following Python script: 
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import os

device = torch.device('cuda')

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super(SimpleCNN, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),
            nn.Conv2d(64, 128, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),
            nn.Conv2d(128, 256, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2)
        )
        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(256 * 4 * 4, 512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

def train_and_evaluate(model, train_loader, test_loader, num_epochs=10, learning_rate=0.01):
    model = model.to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9)

    for epoch in range(num_epochs):
        model.train()
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

        model.eval()
        correct = 0
        total = 0
        with torch.no_grad():
            for images, labels in test_loader:
                images, labels = images.to(device), labels.to(device)
                outputs = model(images)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
        
        accuracy = 100 * correct / total
        print(f'Epoch [{epoch+1}/{num_epochs}], Accuracy: {accuracy:.2f}%')

    return model

def save_model(model, path):
    torch.save(model.state_dict(), path)
    print(f"Model saved to {path}")

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)

model = SimpleCNN()
trained_model = train_and_evaluate(model, train_loader, test_loader)

model_save_path = 'cifar10_cnn_model.pth'
save_model(trained_model, model_save_path)

trained_model.eval()
correct = 0
total = 0
with torch.no_grad():
    for images, labels in test_loader:
        images, labels = images.to(device), labels.to(device)
        outputs = trained_model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print(f'Final Test Accuracy: {100 * correct / total:.2f}%')
This script loads the dataset, transforms it, then trains and evaluates a CNN model that can classify at around 80% accuracy. This model gets saved at the `model_save_path`, which can be configured on your own. You'll notice that at the top, we set our computation device via `device = torch.device('cuda')`. In PyTorch's ROCm installation, `'cuda'` actually points to AMD GPUs, leaving no need to make any changes to any of your desired scripts. Next, create the following inference script: ```python import torch import torch.nn as nn from torchvision import transforms from PIL import Image import requests from io import BytesIO device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model_save_path = 'cifar10_cnn_model.pth' def load_model(model, path): model.load_state_dict(torch.load(path, map_location=device)) model.eval() print(f"Model loaded from {path}") return model class SimpleCNN(nn.Module): def __init__(self, num_classes=10): super(SimpleCNN, self).__init__() self.features = nn.Sequential( nn.Conv2d(3, 64, 3, padding=1), nn.ReLU(inplace=True), nn.MaxPool2d(2, 2), nn.Conv2d(64, 128, 3, padding=1), nn.ReLU(inplace=True), nn.MaxPool2d(2, 2), nn.Conv2d(128, 256, 3, padding=1), nn.ReLU(inplace=True), nn.MaxPool2d(2, 2) ) self.classifier = nn.Sequential( nn.Dropout(0.5), nn.Linear(256 * 4 * 4, 512), nn.ReLU(inplace=True), nn.Dropout(0.5), nn.Linear(512, num_classes) ) def forward(self, x): x = self.features(x) x = x.view(x.size(0), -1) x = self.classifier(x) return x def predict_image_from_url(model, image_url): transform = transforms.Compose([ transforms.Resize((32, 32)), # CIFAR10 images are 32x32 transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) ]) # Download the image response = requests.get(image_url) image = Image.open(BytesIO(response.content)).convert('RGB') image = transform(image).unsqueeze(0).to(device) model.eval() with torch.no_grad(): output = model(image) _, predicted = torch.max(output, 1) classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck') return classes[predicted.item()] # Initialize and load the model model = SimpleCNN() model = load_model(model, model_save_path) model = model.to(device) # Predict from URL image_url = 'https://images.twinkl.co.uk/tw1n/image/private/t_630/u/ux/frog-2_ver_1.jpg' predicted_class = predict_image_from_url(model, image_url) print(f"The image is predicted to be: {predicted_class}") ``` This script loads the model generated by the previous script, then classifies the specified image in `image_url` into one of the 10 categories: ```python 'plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck' ``` That's it! *** ### Fine-Tuning LLMs For the purposes of this tutorial, we'll be fine-tuning Facebook's OPT-350m model. We'll begin by setting up our dependencies for significantly speeding up LLM training. The following tutorial assumes the following prerequisites. If you're using different versions, please adjust your commands accordingly. * Linux (Ubuntu) * CUDA 12.1 or ROCm 6.2 Begin by installing the needed dependencies. {% tabs %} {% tab title="ROCm" %} 
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.1
pip install packaging ninja accelerate wandb
export GPU_ARCHS="gfx942"
export ROCM_HOME="/opt/rocm"
pip install --no-deps --force-reinstall 'https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.44.1.dev0-py3-none-manylinux_2_24_x86_64.whl' 
pip install trl
pip install --no-deps peft
{% endtab %} {% tab title="CUDA" %} 
pip install torch torchvision torchaudio xformers --index-url https://download.pytorch.org/whl/cu121
pip install packaging ninja accelerate wandb bitsandbytes trl
pip install --no-deps peft
{% endtab %} {% endtabs %} {% hint style="info" %} Notice that, as above, this will be the only difference between the two training processes {% endhint %} From there, make the following script in a subfolder you'd like to do your work in. ```python # imports from datasets import load_dataset from trl import SFTTrainer # get dataset dataset = load_dataset("imdb", split="train") # get trainer trainer = SFTTrainer( "facebook/opt-350m", train_dataset=dataset, dataset_text_field="text", max_seq_length=512, ) # train trainer.train() trainer.save_model("imdb_saved") ``` This script trains Facebook's OPT-350m model on an imdb review dataset, and saves the model for later inference. To conduct inference, use the following script: ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch # Load the model and tokenizer model_path = "imdb_saved" model = AutoModelForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path) # Move the model to GPU if available device = "cuda" if torch.cuda.is_available() else "cpu" model = model.to(device) def generate_text(prompt, max_length=150): inputs = tokenizer(prompt, return_tensors="pt").to(device) # Generate with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_length, num_return_sequences=1, no_repeat_ngram_size=2 ) # Decode and return the generated text generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) return generated_text # Test with a positive prompt positive_prompt = "This movie was amazing! The plot" print("Model loading...") positive_response = generate_text(positive_prompt) print("Positive prompt:") print(positive_response ) # Test with a negative prompt negative_prompt = "I hated this film. The acting" print("\nNegative prompt:") print(generate_text(negative_prompt)) # Test with a neutral prompt neutral_prompt = "This movie was okay. It had" print("\nNeutral prompt:") print(generate_text(neutral_prompt)) ``` *** ### Accelerated Inference for Llama 3.1 (and other HF Models) For this section of the tutorial, we're going to use vLLM, a framework for accelerated LLM inference and serving. {% hint style="info" %} More information on vLLM [here](https://docs.vllm.ai/en/latest/). {% endhint %} We're going to serve Llama 3.1 8B Instruct through Docker containers. We'll start by pulling the images and serving the endpoints from there. Note that since the Llama models are gated, we'll have to log in through huggingface-cli to use them. {% tabs %} {% tab title="ROCm" %} ```bash docker pull rocm/vllm:rocm6.2_mi300_ubuntu22.04_py3.9_vllm_7c5fd50 docker run -it --network=host --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device /dev/kfd --device /dev/dri vllm-rocm huggingface-cli login #paste your token as needed vllm serve meta-llama/Llama-3.1-8B-Instruct ``` {% endtab %} {% tab title="CUDA" %} ```bash docker pull vllm/vllm-openai:latest docker run --runtime nvidia --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=" \ -p 8000:8000 \ --ipc=host \ vllm/vllm-openai:latest \ --model meta-llama/Llama-3.1-8B-Instruct ``` {% endtab %} {% endtabs %} {% hint style="info" %} Note that you will have to note your HuggingFace API Token for both methods. {% endhint %} In a separate terminal, you can now query the endpoints! ```bash curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "What is the meaning of life?", "max_tokens": 128, "top_p": 0.95, "top_k": 20, "temperature": 0.8 }' ``` # Policies # Terms of Service REVISED JAN 2024 ### Website Terms of Use The TensorWave website located at is a copyrighted work belonging to TensorWave Inc.. Certain features of the Site may be subject to additional guidelines, terms, or rules, which will be posted on the Site in connection with such features. All such additional terms, guidelines, and rules are incorporated by reference into these Terms. These Terms of Use described the legally binding terms and conditions that oversee your use of the Site. BY LOGGING INTO THE SITE, YOU ARE BEING COMPLIANT THAT THESE TERMS and you represent that you have the authority and capacity to enter into these Terms. YOU SHOULD BE AT LEAST 18 YEARS OF AGE TO ACCESS THE SITE. IF YOU DISAGREE WITH ALL OF THE PROVISION OF THESE TERMS, DO NOT LOG INTO AND/OR USE THE SITE. These terms require the use of arbitration Section 10.2 on an individual basis to resolve disputes and also limit the remedies available to you in the event of a dispute. These Terms of Use were created with the help of the [Terms Of Use Generator](https://www.termsofusegenerator.net/). ### Access to the Site **Subject to these Terms.** Company grants you a non-transferable, non-exclusive, revocable, limited license to access the Site solely for your own personal, noncommercial use. **Certain Restrictions.** The rights approved to you in these Terms are subject to the following restrictions: (a) you shall not sell, rent, lease, transfer, assign, distribute, host, or otherwise commercially exploit the Site; (b) you shall not change, make derivative works of, disassemble, reverse compile or reverse engineer any part of the Site; (c) you shall not access the Site in order to build a similar or competitive website; and (d) except as expressly stated herein, no part of the Site may be copied, reproduced, distributed, republished, downloaded, displayed, posted or transmitted in any form or by any means unless otherwise indicated, any future release, update, or other addition to functionality of the Site shall be subject to these Terms. All copyright and other proprietary notices on the Site must be retained on all copies thereof. Company reserves the right to change, suspend, or cease the Site with or without notice to you. You approved that Company will not be held liable to you or any third-party for any change, interruption, or termination of the Site or any part. **No Support or Maintenance.** You agree that Company will have no obligation to provide you with any support in connection with the Site. Excluding any User Content that you may provide, you are aware that all the intellectual property rights, including copyrights, patents, trademarks, and trade secrets, in the Site and its content are owned by Company or Company’s suppliers. Note that these Terms and access to the Site do not give you any rights, title or interest in or to any intellectual property rights, except for the limited access rights expressed in Section 2.1. Company and its suppliers reserve all rights not granted in these Terms. ### User Content **User Content.** "User Content" means any and all information and content that a user submits to the Site. You are exclusively responsible for your User Content. You bear all risks associated with use of your User Content. You hereby certify that your User Content does not violate our Acceptable Use Policy. You may not represent or imply to others that your User Content is in any way provided, sponsored or endorsed by Company. Because you alone are responsible for your User Content, you may expose yourself to liability. Company is not obliged to backup any User Content that you post; also, your User Content may be deleted at any time without prior notice to you. You are solely responsible for making your own backup copies of your User Content if you desire. You hereby grant to Company an irreversible, nonexclusive, royalty-free and fully paid, worldwide license to reproduce, distribute, publicly display and perform, prepare derivative works of, incorporate into other works, and otherwise use and exploit your User Content, and to grant sublicenses of the foregoing rights, solely for the purposes of including your User Content in the Site. You hereby irreversibly waive any claims and assertions of moral rights or attribution with respect to your User Content. **Acceptable Use Policy.** The following terms constitute our "Acceptable Use Policy": You agree not to use the Site to collect, upload, transmit, display, or distribute any User Content (i) that violates any third-party right or any intellectual property or proprietary right; (ii) that is unlawful, harassing, abusive, tortious, threatening, harmful, invasive of another’s privacy, vulgar, defamatory, false, intentionally misleading, trade libelous, pornographic, obscene, patently offensive, promotes racism, bigotry, hatred, or physical harm of any kind against any group or individual; (iii) that is harmful to minors in any way; or (iv) that is in violation of any law, regulation, or obligations or restrictions imposed by any third party. In addition, you agree not to: (i) upload, transmit, or distribute to or through the Site any software intended to damage or alter a computer system or data; (ii) send through the Site unsolicited or unauthorized advertising, promotional materials, junk mail, spam, chain letters, pyramid schemes, or any other form of duplicative or unsolicited messages; (iii) use the Site to harvest, collect, gather or assemble information or data regarding other users without their consent; (iv) interfere with, disrupt, or create an undue burden on servers or networks connected to the Site, or violate the regulations, policies or procedures of such networks; (v) attempt to gain unauthorized access to the Site, whether through password mining or any other means; (vi) harass or interfere with any other user’s use and enjoyment of the Site; or (vi) use software or automated agents or scripts to produce multiple accounts on the Site, or to generate automated searches, requests, or queries to the Site. We reserve the right to review any User Content, and to investigate and/or take appropriate action against you in our sole discretion if you violate the Acceptable Use Policy or any other provision of these Terms or otherwise create liability for us or any other person. Such action may include removing or modifying your User Content, terminating your Account in accordance with Section 8, and/or reporting you to law enforcement authorities. If you provide Company with any feedback or suggestions regarding the Site, you hereby assign to Company all rights in such Feedback and agree that Company shall have the right to use and fully exploit such Feedback and related information in any manner it believes appropriate. Company will treat any Feedback you provide to Company as non-confidential and non-proprietary. You agree to indemnify and hold Company and its officers, employees, and agents harmless, including costs and attorneys’ fees, from any claim or demand made by any third-party due to or arising out of (a) your use of the Site, (b) your violation of these Terms, (c) your violation of applicable laws or regulations or (d) your User Content. Company reserves the right to assume the exclusive defense and control of any matter for which you are required to indemnify us, and you agree to cooperate with our defense of these claims. You agree not to settle any matter without the prior written consent of Company. Company will use reasonable efforts to notify you of any such claim, action or proceeding upon becoming aware of it. ### Third-Party Links & Ads; Other Users **Third-Party Links & Ads.** The Site may contain links to third-party websites and services, and/or display advertisements for third-parties. Such Third-Party Links & Ads are not under the control of Company, and Company is not responsible for any Third-Party Links & Ads. Company provides access to these Third-Party Links & Ads only as a convenience to you, and does not review, approve, monitor, endorse, warrant, or make any representations with respect to Third-Party Links & Ads. You use all Third-Party Links & Ads at your own risk, and should apply a suitable level of caution and discretion in doing so. When you click on any of the Third-Party Links & Ads, the applicable third party’s terms and policies apply, including the third party’s privacy and data gathering practices. **Other Users.** Each Site user is solely responsible for any and all of its own User Content. Because we do not control User Content, you acknowledge and agree that we are not responsible for any User Content, whether provided by you or by others. You agree that Company will not be responsible for any loss or damage incurred as the result of any such interactions. If there is a dispute between you and any Site user, we are under no obligation to become involved. You hereby release and forever discharge the Company and our officers, employees, agents, successors, and assigns from, and hereby waive and relinquish, each and every past, present and future dispute, claim, controversy, demand, right, obligation, liability, action and cause of action of every kind and nature, that has arisen or arises directly or indirectly out of, or that relates directly or indirectly to, the Site. If you are a California resident, you hereby waive California civil code section 1542 in connection with the foregoing, which states: "a general release does not extend to claims which the creditor does not know or suspect to exist in his or her favor at the time of executing the release, which if known by him or her must have materially affected his or her settlement with the debtor." **Cookies and Web Beacons.** Like any other website, TensorWave uses ‘cookies’. These cookies are used to store information including visitors’ preferences, and the pages on the website that the visitor accessed or visited. The information is used to optimize the users’ experience by customizing our web page content based on visitors’ browser type and/or other information. ### Disclaimers The site is provided on an "as-is" and "as available" basis, and company and our suppliers expressly disclaim any and all warranties and conditions of any kind, whether express, implied, or statutory, including all warranties or conditions of merchantability, fitness for a particular purpose, title, quiet enjoyment, accuracy, or non-infringement. We and our suppliers make not guarantee that the site will meet your requirements, will be available on an uninterrupted, timely, secure, or error-free basis, or will be accurate, reliable, free of viruses or other harmful code, complete, legal, or safe. If applicable law requires any warranties with respect to the site, all such warranties are limited in duration to ninety (90) days from the date of first use. Some jurisdictions do not allow the exclusion of implied warranties, so the above exclusion may not apply to you. Some jurisdictions do not allow limitations on how long an implied warranty lasts, so the above limitation may not apply to you. ### Limitation on Liability To the maximum extent permitted by law, in no event shall company or our suppliers be liable to you or any third-party for any lost profits, lost data, costs of procurement of substitute products, or any indirect, consequential, exemplary, incidental, special or punitive damages arising from or relating to these terms or your use of, or incapability to use the site even if company has been advised of the possibility of such damages. Access to and use of the site is at your own discretion and risk, and you will be solely responsible for any damage to your device or computer system, or loss of data resulting therefrom. To the maximum extent permitted by law, notwithstanding anything to the contrary contained herein, our liability to you for any damages arising from or related to this agreement, will at all times be limited to a maximum of fifty U.S. dollars (u.s. $50). The existence of more than one claim will not enlarge this limit. You agree that our suppliers will have no liability of any kind arising from or relating to this agreement. Some jurisdictions do not allow the limitation or exclusion of liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. **Term and Termination.** Subject to this Section, these Terms will remain in full force and effect while you use the Site. We may suspend or terminate your rights to use the Site at any time for any reason at our sole discretion, including for any use of the Site in violation of these Terms. Upon termination of your rights under these Terms, your Account and right to access and use the Site will terminate immediately. You understand that any termination of your Account may involve deletion of your User Content associated with your Account from our live databases. Company will not have any liability whatsoever to you for any termination of your rights under these Terms. Even after your rights under these Terms are terminated, the following provisions of these Terms will remain in effect: Sections 2 through 2.5, Section 3 and Sections 4 through 10. ### Copyright Policy. Company respects the intellectual property of others and asks that users of our Site do the same. In connection with our Site, we have adopted and implemented a policy respecting copyright law that provides for the removal of any infringing materials and for the termination of users of our online Site who are repeated infringers of intellectual property rights, including copyrights. If you believe that one of our users is, through the use of our Site, unlawfully infringing the copyright(s) in a work, and wish to have the allegedly infringing material removed, the following information in the form of a written notification (pursuant to 17 U.S.C. § 512(c)) must be provided to our designated Copyright Agent: * your physical or electronic signature; * identification of the copyrighted work(s) that you claim to have been infringed; * identification of the material on our services that you claim is infringing and that you request us to remove; * sufficient information to permit us to locate such material; * your address, telephone number, and e-mail address; * a statement that you have a good faith belief that use of the objectionable material is not authorized by the copyright owner, its agent, or under the law; and * a statement that the information in the notification is accurate, and under penalty of perjury, that you are either the owner of the copyright that has allegedly been infringed or that you are authorized to act on behalf of the copyright owner. Please note that, pursuant to 17 U.S.C. § 512(f), any misrepresentation of material fact in a written notification automatically subjects the complaining party to liability for any damages, costs and attorney’s fees incurred by us in connection with the written notification and allegation of copyright infringement. ### General These Terms are subject to occasional revision, and if we make any substantial changes, we may notify you by sending you an e-mail to the last e-mail address you provided to us and/or by prominently posting notice of the changes on our Site. You are responsible for providing us with your most current e-mail address. In the event that the last e-mail address that you have provided us is not valid our dispatch of the e-mail containing such notice will nonetheless constitute effective notice of the changes described in the notice. Any changes to these Terms will be effective upon the earliest of thirty (30) calendar days following our dispatch of an e-mail notice to you or thirty (30) calendar days following our posting of notice of the changes on our Site. These changes will be effective immediately for new users of our Site. Continued use of our Site following notice of such changes shall indicate your acknowledgement of such changes and agreement to be bound by the terms and conditions of such changes. Dispute Resolution. Please read this Arbitration Agreement carefully. It is part of your contract with Company and affects your rights. It contains procedures for MANDATORY BINDING ARBITRATION AND A CLASS ACTION WAIVER. **Applicability of Arbitration Agreement.** All claims and disputes in connection with the Terms or the use of any product or service provided by the Company that cannot be resolved informally or in small claims court shall be resolved by binding arbitration on an individual basis under the terms of this Arbitration Agreement. Unless otherwise agreed to, all arbitration proceedings shall be held in English. This Arbitration Agreement applies to you and the Company, and to any subsidiaries, affiliates, agents, employees, predecessors in interest, successors, and assigns, as well as all authorized or unauthorized users or beneficiaries of services or goods provided under the Terms. **Notice Requirement and Informal Dispute Resolution.** Before either party may seek arbitration, the party must first send to the other party a written Notice of Dispute describing the nature and basis of the claim or dispute, and the requested relief. A Notice to the Company should be sent to: 300 S 4th St Ste 180 Las Vegas, NV 89101. After the Notice is received, you and the Company may attempt to resolve the claim or dispute informally. If you and the Company do not resolve the claim or dispute within thirty (30) days after the Notice is received, either party may begin an arbitration proceeding. The amount of any settlement offer made by any party may not be disclosed to the arbitrator until after the arbitrator has determined the amount of the award to which either party is entitled. **Arbitration Rules.** Arbitration shall be initiated through the American Arbitration Association, an established alternative dispute resolution provider that offers arbitration as set forth in this section. If AAA is not available to arbitrate, the parties shall agree to select an alternative ADR Provider. The rules of the ADR Provider shall govern all aspects of the arbitration except to the extent such rules are in conflict with the Terms. The AAA Consumer Arbitration Rules governing the arbitration are available online at adr.org or by calling the AAA at 1-800-778-7879. The arbitration shall be conducted by a single, neutral arbitrator. Any claims or disputes where the total amount of the award sought is less than Ten Thousand U.S. Dollars (US $10,000.00) may be resolved through binding non-appearance-based arbitration, at the option of the party seeking relief. For claims or disputes where the total amount of the award sought is Ten Thousand U.S. Dollars (US $10,000.00) or more, the right to a hearing will be determined by the Arbitration Rules. Any hearing will be held in a location within 100 miles of your residence, unless you reside outside of the United States, and unless the parties agree otherwise. If you reside outside of the U.S., the arbitrator shall give the parties reasonable notice of the date, time and place of any oral hearings. Any judgment on the award rendered by the arbitrator may be entered in any court of competent jurisdiction. If the arbitrator grants you an award that is greater than the last settlement offer that the Company made to you prior to the initiation of arbitration, the Company will pay you the greater of the award or $2,500.00. Each party shall bear its own costs and disbursements arising out of the arbitration and shall pay an equal share of the fees and costs of the ADR Provider. **Additional Rules for Non-Appearance Based Arbitration.** If non-appearance based arbitration is elected, the arbitration shall be conducted by telephone, online and/or based solely on written submissions; the specific manner shall be chosen by the party initiating the arbitration. The arbitration shall not involve any personal appearance by the parties or witnesses unless otherwise agreed by the parties. **Time Limits.** If you or the Company pursues arbitration, the arbitration action must be initiated and/or demanded within the statute of limitations and within any deadline imposed under the AAA Rules for the pertinent claim. **Authority of Arbitrator.** If arbitration is initiated, the arbitrator will decide the rights and liabilities of you and the Company, and the dispute will not be consolidated with any other matters or joined with any other cases or parties. The arbitrator shall have the authority to grant motions dispositive of all or part of any claim. The arbitrator shall have the authority to award monetary damages, and to grant any non-monetary remedy or relief available to an individual under applicable law, the AAA Rules, and the Terms. The arbitrator shall issue a written award and statement of decision describing the essential findings and conclusions on which the award is based. The arbitrator has the same authority to award relief on an individual basis that a judge in a court of law would have. The award of the arbitrator is final and binding upon you and the Company. **Waiver of Jury Trial.** THE PARTIES HEREBY WAIVE THEIR CONSTITUTIONAL AND STATUTORY RIGHTS TO GO TO COURT AND HAVE A TRIAL IN FRONT OF A JUDGE OR A JURY, instead electing that all claims and disputes shall be resolved by arbitration under this Arbitration Agreement. Arbitration procedures are typically more limited, more efficient and less expensive than rules applicable in a court and are subject to very limited review by a court. In the event any litigation should arise between you and the Company in any state or federal court in a suit to vacate or enforce an arbitration award or otherwise, YOU AND THE COMPANY WAIVE ALL RIGHTS TO A JURY TRIAL, instead electing that the dispute be resolved by a judge. **Waiver of Class or Consolidated Actions.** All claims and disputes within the scope of this arbitration agreement must be arbitrated or litigated on an individual basis and not on a class basis, and claims of more than one customer or user cannot be arbitrated or litigated jointly or consolidated with those of any other customer or user. **Confidentiality.** All aspects of the arbitration proceeding shall be strictly confidential. The parties agree to maintain confidentiality unless otherwise required by law. This paragraph shall not prevent a party from submitting to a court of law any information necessary to enforce this Agreement, to enforce an arbitration award, or to seek injunctive or equitable relief. **Severability.** If any part or parts of this Arbitration Agreement are found under the law to be invalid or unenforceable by a court of competent jurisdiction, then such specific part or parts shall be of no force and effect and shall be severed and the remainder of the Agreement shall continue in full force and effect. **Right to Waive.** Any or all of the rights and limitations set forth in this Arbitration Agreement may be waived by the party against whom the claim is asserted. Such waiver shall not waive or affect any other portion of this Arbitration Agreement. **Survival of Agreement.** This Arbitration Agreement will survive the termination of your relationship with Company. **Small Claims Court.** Nonetheless the foregoing, either you or the Company may bring an individual action in small claims court. **Emergency Equitable Relief.** Anyhow the foregoing, either party may seek emergency equitable relief before a state or federal court in order to maintain the status quo pending arbitration. A request for interim measures shall not be deemed a waiver of any other rights or obligations under this Arbitration Agreement. **Claims Not Subject to Arbitration.** Notwithstanding the foregoing, claims of defamation, violation of the Computer Fraud and Abuse Act, and infringement or misappropriation of the other party’s patent, copyright, trademark or trade secrets shall not be subject to this Arbitration Agreement. In any circumstances where the foregoing Arbitration Agreement permits the parties to litigate in court, the parties hereby agree to submit to the personal jurisdiction of the courts located within Netherlands County, California, for such purposes. The Site may be subject to U.S. export control laws and may be subject to export or import regulations in other countries. You agree not to export, re-export, or transfer, directly or indirectly, any U.S. technical data acquired from Company, or any products utilizing such data, in violation of the United States export laws or regulations. Company is located at the address in Section 10.8. If you are a California resident, you may report complaints to the Complaint Assistance Unit of the Division of Consumer Product of the California Department of Consumer Affairs by contacting them in writing at 400 R Street, Sacramento, CA 95814, or by telephone at (800) 952-5210. **Electronic Communications.** The communications between you and Company use electronic means, whether you use the Site or send us emails, or whether Company posts notices on the Site or communicates with you via email. For contractual purposes, you (a) consent to receive communications from Company in an electronic form; and (b) agree that all terms and conditions, agreements, notices, disclosures, and other communications that Company provides to you electronically satisfy any legal obligation that such communications would satisfy if it were be in a hard copy writing. **Entire Terms.** These Terms constitute the entire agreement between you and us regarding the use of the Site. Our failure to exercise or enforce any right or provision of these Terms shall not operate as a waiver of such right or provision. The section titles in these Terms are for convenience only and have no legal or contractual effect. The word "including" means "including without limitation". If any provision of these Terms is held to be invalid or unenforceable, the other provisions of these Terms will be unimpaired and the invalid or unenforceable provision will be deemed modified so that it is valid and enforceable to the maximum extent permitted by law. Your relationship to Company is that of an independent contractor, and neither party is an agent or partner of the other. These Terms, and your rights and obligations herein, may not be assigned, subcontracted, delegated, or otherwise transferred by you without Company’s prior written consent, and any attempted assignment, subcontract, delegation, or transfer in violation of the foregoing will be null and void. Company may freely assign these Terms. The terms and conditions set forth in these Terms shall be binding upon assignees. **Your Privacy.** Please read our [Privacy Policy](/policies/privacy-policy). **Copyright/Trademark Information.** Copyright ©. All rights reserved. All trademarks, logos and service marks displayed on the Site are our property or the property of other third-parties. You are not permitted to use these Marks without our prior written consent or the consent of such third party which may own the Marks. ### Contact Information Email: **©2024 TensorWave. All Rights Reserved** # Privacy Policy REVISED JAN 2024 This Privacy Policy describes Our policies and procedures on the collection, use and disclosure of Your information when You use the Service and tells You about Your privacy rights and how the law protects You. We use Your Personal data to provide and improve the Service. By using the Service, You agree to the collection and use of information in accordance with this Privacy Policy. ### Interpretation and Definitions #### Interpretation The words of which the initial letter is capitalized have meanings defined under the following conditions. The following definitions shall have the same meaning regardless of whether they appear in singular or in plural. #### Definitions For the purposes of this Privacy Policy: * **Account** means a unique account created for You to access our Service or parts of our Service. * **Company** (referred to as either "the Company", "We", "Us" or "Our" in this Agreement) refers to TensorWave Inc. * **Cookies** are small files that are placed on Your computer, mobile device or any other device by a website, containing the details of Your browsing history on that website among its many uses. * **Country** refers to: Nevada, United States * **Device** means any device that can access the Service such as a computer, a cellphone or a digital tablet. * **Personal Data** is any information that relates to an identified or identifiable individual. * **Service** refers to the Website. * **Service Provider** means any natural or legal person who processes the data on behalf of the Company. It refers to third-party companies or individuals employed by the Company to facilitate the Service, to provide the Service on behalf of the Company, to perform services related to the Service or to assist the Company in analyzing how the Service is used. * **Third-party Social Media Service** refers to any website or any social network website through which a User can log in or create an account to use the Service. * **Usage Data** refers to data collected automatically, either generated by the use of the Service or from the Service infrastructure itself (for example, the duration of a page visit). * **Website** refers to TensorWave, accessible from * **You** means the individual accessing or using the Service, or the company, or other legal entity on behalf of which such individual is accessing or using the Service, as applicable. ### Collecting and Using Your Personal Data #### Types of Data Collected **Personal Data** While using Our Service, We may ask You to provide Us with certain personally identifiable information that can be used to contact or identify You. Personally identifiable information may include, but is not limited to: * Email address * First name and last name * Phone number * Usage Data **Usage Data** Usage Data is collected automatically when using the Service. Usage Data may include information such as Your Device's Internet Protocol address (e.g. IP address), browser type, browser version, the pages of our Service that You visit, the time and date of Your visit, the time spent on those pages, unique device identifiers and other diagnostic data. When You access the Service by or through a mobile device, We may collect certain information automatically, including, but not limited to, the type of mobile device You use, Your mobile device unique ID, the IP address of Your mobile device, Your mobile operating system, the type of mobile Internet browser You use, unique device identifiers and other diagnostic data. We may also collect information that Your browser sends whenever You visit our Service or when You access the Service by or through a mobile device. **Information from Third-Party Social Media Services** The Company allows You to create an account and log in to use the Service through the following Third-party Social Media Services: * Google * Facebook * Twitter If You decide to register through or otherwise grant us access to a Third-Party Social Media Service, We may collect Personal data that is already associated with Your Third-Party Social Media Service's account, such as Your name, Your email address, Your activities or Your contact list associated with that account. You may also have the option of sharing additional information with the Company through Your Third-Party Social Media Service's account. If You choose to provide such information and Personal Data, during registration or otherwise, You are giving the Company permission to use, share, and store it in a manner consistent with this Privacy Policy. **Tracking Technologies and Cookies** We use Cookies and similar tracking technologies to track the activity on Our Service and store certain information. Tracking technologies used are beacons, tags, and scripts to collect and track information and to improve and analyze Our Service. The technologies We use may include: * **Cookies or Browser Cookies.** A cookie is a small file placed on Your Device. You can instruct Your browser to refuse all Cookies or to indicate when a Cookie is being sent. However, if You do not accept Cookies, You may not be able to use some parts of our Service. Unless you have adjusted Your browser setting so that it will refuse Cookies, our Service may use Cookies. * **Flash Cookies.** Certain features of our Service may use local stored objects (or Flash Cookies) to collect and store information about Your preferences or Your activity on our Service. Flash Cookies are not managed by the same browser settings as those used for Browser Cookies. For more information on how You can delete Flash Cookies, please read "Where can I change the settings for disabling, or deleting local shared objects?" available at * **Web Beacons.** Certain sections of our Service and our emails may contain small electronic files known as web beacons (also referred to as clear gifs, pixel tags, and single-pixel gifs) that permit the Company, for example, to count users who have visited those pages or opened an email and for other related website statistics (for example, recording the popularity of a certain section and verifying system and server integrity). Cookies can be "Persistent" or "Session" Cookies. Persistent Cookies remain on Your personal computer or mobile device when You go offline, while Session Cookies are deleted as soon as You close Your web browser. You can learn more about cookies here: [All About Cookies by TermsFeed](https://www.termsfeed.com/blog/cookies/). We use both Session and Persistent Cookies for the purposes set out below: * **Necessary / Essential Cookies** Type: Session Cookies Administered by: Us Purpose: These Cookies are essential to provide You with services available through the Website and to enable You to use some of its features. They help to authenticate users and prevent fraudulent use of user accounts. Without these Cookies, the services that You have asked for cannot be provided, and We only use these Cookies to provide You with those services. * **Cookies Policy / Notice Acceptance Cookies** Type: Persistent Cookies Administered by: Us Purpose: These Cookies identify if users have accepted the use of cookies on the Website. * **Functionality Cookies** Type: Persistent Cookies Administered by: Us Purpose: These Cookies allow us to remember choices You make when You use the Website, such as remembering your login details or language preference. The purpose of these Cookies is to provide You with a more personal experience and to avoid You having to re-enter your preferences every time You use the Website. For more information about the cookies we use and your choices regarding cookies, please visit our Cookies Policy or the Cookies section of our Privacy Policy. ### Use of Your Personal Data The Company may use Personal Data for the following purposes: * **To provide and maintain our Service**, including to monitor the usage of our Service. * **To manage Your Account:** to manage Your registration as a user of the Service. The Personal Data You provide can give You access to different functionalities of the Service that are available to You as a registered user. * **For the performance of a contract:** the development, compliance and undertaking of the purchase contract for the products, items or services You have purchased or of any other contract with Us through the Service. * **To contact You:** To contact You by email, telephone calls, SMS, or other equivalent forms of electronic communication, such as a mobile application's push notifications regarding updates or informative communications related to the functionalities, products or contracted services, including the security updates, when necessary or reasonable for their implementation. * **To provide You** with news, special offers and general information about other goods, services and events which we offer that are similar to those that you have already purchased or enquired about unless You have opted not to receive such information. * **To manage Your requests:** To attend and manage Your requests to Us. * **For business transfers:** We may use Your information to evaluate or conduct a merger, divestiture, restructuring, reorganization, dissolution, or other sale or transfer of some or all of Our assets, whether as a going concern or as part of bankruptcy, liquidation, or similar proceeding, in which Personal Data held by Us about our Service users is among the assets transferred. * **For other purposes**: We may use Your information for other purposes, such as data analysis, identifying usage trends, determining the effectiveness of our promotional campaigns and to evaluate and improve our Service, products, services, marketing and your experience. We may share Your personal information in the following situations: * **With Service Providers:** We may share Your personal information with Service Providers to monitor and analyze the use of our Service, to contact You. * **For business transfers:** We may share or transfer Your personal information in connection with, or during negotiations of, any merger, sale of Company assets, financing, or acquisition of all or a portion of Our business to another company. * **With Affiliates:** We may share Your information with Our affiliates, in which case we will require those affiliates to honor this Privacy Policy. Affiliates include Our parent company and any other subsidiaries, joint venture partners or other companies that We control or that are under common control with Us. * **With business partners:** We may share Your information with Our business partners to offer You certain products, services or promotions. * **With other users:** when You share personal information or otherwise interact in the public areas with other users, such information may be viewed by all users and may be publicly distributed outside. If You interact with other users or register through a Third-Party Social Media Service, Your contacts on the Third-Party Social Media Service may see Your name, profile, pictures and description of Your activity. Similarly, other users will be able to view descriptions of Your activity, communicate with You and view Your profile. * **With Your consent**: We may disclose Your personal information for any other purpose with Your consent. #### Retention of Your Personal Data The Company will retain Your Personal Data only for as long as is necessary for the purposes set out in this Privacy Policy. We will retain and use Your Personal Data to the extent necessary to comply with our legal obligations (for example, if we are required to retain your data to comply with applicable laws), resolve disputes, and enforce our legal agreements and policies. The Company will also retain Usage Data for internal analysis purposes. Usage Data is generally retained for a shorter period of time, except when this data is used to strengthen the security or to improve the functionality of Our Service, or We are legally obligated to retain this data for longer time periods. #### Transfer of Your Personal Data Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ than those from Your jurisdiction. Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer. The Company will take all steps reasonably necessary to ensure that Your data is treated securely and in accordance with this Privacy Policy and no transfer of Your Personal Data will take place to an organization or a country unless there are adequate controls in place including the security of Your data and other personal information. #### Disclosure of Your Personal Data **Business Transactions** If the Company is involved in a merger, acquisition or asset sale, Your Personal Data may be transferred. We will provide notice before Your Personal Data is transferred and becomes subject to a different Privacy Policy. **Law enforcement** Under certain circumstances, the Company may be required to disclose Your Personal Data if required to do so by law or in response to valid requests by public authorities (e.g. a court or a government agency). **Other legal requirements** The Company may disclose Your Personal Data in the good faith belief that such action is necessary to: * Comply with a legal obligation * Protect and defend the rights or property of the Company * Prevent or investigate possible wrongdoing in connection with the Service * Protect the personal safety of Users of the Service or the public * Protect against legal liability #### Security of Your Personal Data The security of Your Personal Data is important to Us, but remember that no method of transmission over the Internet, or method of electronic storage is 100% secure. While We strive to use commercially acceptable means to protect Your Personal Data, We cannot guarantee its absolute security. ### Children's Privacy Our Service does not address anyone under the age of 13. We do not knowingly collect personally identifiable information from anyone under the age of 13. If You are a parent or guardian and You are aware that Your child has provided Us with Personal Data, please contact Us. If We become aware that We have collected Personal Data from anyone under the age of 13 without verification of parental consent, We take steps to remove that information from Our servers. If We need to rely on consent as a legal basis for processing Your information and Your country requires consent from a parent, We may require Your parent's consent before We collect and use that information. ### Links to Other Websites Our Service may contain links to other websites that are not operated by Us. If You click on a third party link, You will be directed to that third party's site. We strongly advise You to review the Privacy Policy of every site You visit. We have no control over and assume no responsibility for the content, privacy policies or practices of any third party sites or services. ### Changes to this Privacy Policy We may update Our Privacy Policy from time to time. We will notify You of any changes by posting the new Privacy Policy on this page. We will let You know via email and/or a prominent notice on Our Service, prior to the change becoming effective and update the "Last updated" date at the top of this Privacy Policy. You are advised to review this Privacy Policy periodically for any changes. Changes to this Privacy Policy are effective when they are posted on this page. ### Contact Us If you have any questions about this Privacy Policy, You can contact us: * By email: **©2024 TensorWave. All Rights Reserved** # AMD vs NVIDIA # MI300X vs H100 *** ## Raw Performance Comparison

Raw performance comparison of AMD's MI300x vs NVIDIA's H100

The current go-to provider of GPUs, NVIDIA, has a long history in developing graphics accelerators and related hardware, so adding a line of AI-focused GPUs was not a great leap for them. Their flagship AI GPU, the H100, is in such high demand that customers must wait for a year or more for their orders to be filled. Meanwhile, Advanced Micro Devices (AMD), better known as a competitor to Intel in the PC and server CPU market, has introduced its own GPU product line, called Instinct. The Instinct MI300X, introduced in late 2023, is causing a stir in the AI development community. *Lets break down their individual capabilities to determine which best fits your use case.* *** ## Technical Specifications Comparison ### Architecture The H100 and MI300X have quite different architectures. The H100 is implemented on a single large (814 square millimeters) chip of silicon, with all the components in the same plane. This architecture is the same tried-and-true approach used in almost all integrated circuits. The advantage is that the manufacturing process is mature, although the large size pushes the limits of what can be manufactured using standard processes. The MI300X, in contrast, is assembled as a three-dimensional stack. The MI300X has eight separate GPU integrated circuits surrounded by high-bandwidth memory in one layer, which is placed on top of a layer of input-output circuitry. This approach packs more transistors in a smaller area with shorter distances between the computing modules and memory. However, the manufacturing process is entirely new and more complex: The layers must line up perfectly with nanometer precision for the device to work. ### Memory The H100 comes with 80 GB of GPU memory, whereas the MI300X has 192 GB. The memory bandwidth—the speed at which the chip can move data between memory and the computing modules, and an important contributor to overall performance—is also greater for the MI300X (5.2 TB/s vs. 3.35 TB/s). *** ## Performance Benchmarks {% hint style="warning" %} At this writing, independent comparisons are not yet available, so all we have are published performance claims by each side, without knowing the exact environments from which these claims were generated. {% endhint %} ### Inference Performance AMD claims a 20% advantage over the H100 in inference performance (that is, using a trained AI model to perform tasks) on the Llama 2 LLM with 13 billion parameters. ### Floating Point Operations For eight-bit floating-point precision (known as FP8), AMD claims 2,614.9 trillion FLOPS (TFLOPS) vs. 1,978.9 TFLOPS for the H100. ### Latency AMD claims a 40% advantage over the H100 in inference latency on Llama 2 with 70 billion parameters. The higher memory bandwidth of the MI300X has a strong influence on this performance metric. {% hint style="info" %} For more information or to discuss your specific requirements, [contact TensorWave today](https://www.tensorwave.com/contact). {% endhint %} *** # Architecture ## AMD MI300X The MI300X combines eight MI300X accelerators into a single system using the infinity architecture technology. It boasts 192 GB of HBM memory, 5.3TB/s of peak memory bandwidth, and a unified memory architecture. > The AMD MI300X provides lower latency and better consistency at larger batch sizes. With the MI300 series, AMD is introducing the Accelerator Complex Die (XCD), which contains the GPU computational elements of the processor along with the lower levels of the cache hierarchy. {% hint style="info" %} Learn more on AMD's MI300X architecture [here](https://rocm.docs.amd.com/en/latest/conceptual/gpu-arch/mi300.html). {% endhint %} ## Nvidia H100 The H100 uses Hopper architecture and Tensor Core GPUs as well as fourth-generation Tensor Cores that can speed up inference by up to 30X and reduce memory usage as well as supporting a maximum of 120GB of memory. While the H100 may outperform the MI300X in smaller quantities, in larger batch sizes the MI300X outperforms where its larger VRAM helps it handle more workloads efficiently. > The NVIDIA H100 SXM offers higher throughput at smaller to medium batch sizes {% hint style="info" %} Learn more about NVIDIA's H100 architecture [here](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/). {% endhint %} # Why TensorWave? ***

We believe that every business should have access to AI computing tools to their fullest potential.

### Immediate Availability Skip long queues and empty promises with other providers. Simply [contact us](https://tensorwave.com/book-a-call) and have your node up and running in just a few days. ### Easy Porting AMD fully supports PyTorch, TensorFlow, Jax, Hugging Face and more standard ML libraries seamlessly. Many workflows include **0 code changes** to run on our infrastructure. ### Higher Performance / $ than NVIDIA Compared to Nvidia's H100, the M1300X offers a better cost to performance ratio for all of your AI computing needs. MI300X offers 192GB of VRAM compared to 80GB so you can store larger models on a single GPU.
{% hint style="info" %} **Still not convinced?** Contact us on our [site](https://tensorwave.com/book-a-call) to test your workload for free today! {% endhint %} ***