Managing Container Images with Enroot

TensorWave Slurm uses Enroot as its lightweight, high-performance container runtime for HPC and AI workloads.

Unlike traditional container engines, Enroot runs entirely in user space with no privileged daemons or root access required, making it ideal for multi-tenant and secure compute environments.

Enroot executes standard Docker or OCI images as unprivileged user processes, unpacking each image into an isolated filesystem that can be shared across nodes. It preserves direct access to GPUs, high-speed interconnects, and local storage, ensuring your jobs run efficiently within containers.

You don’t need to run Enroot commands directly; TensorWave Slurm handles that automatically through Pyxis, which integrates Enroot with familiar Slurm tools like srun and sbatch. See the Running Containerized Jobs in Pyxis page for info on how to run containerized workloads. Together, they allow you to launch containerized jobs using the same workflow you already know, with the added benefits of portability and reproducibility.

---

Optional: Importing an Image with Enroot

Although Pyxis automatically handles Enroot under the hood, you can manually import container images for debugging or pre-caching.

Due to Kubernetes security-permissioning constraints on the login pods, container image management has to happen on the worker pods. This means operations including docker build, docker pull, and enroot import, must happen in a Slurm-job context.

So, to build a .sqsh image locally, run docker build and enroot import with srun:

IMG="<repo>:<tag>"
# srun command to 'build' and 'import' $IMG
srun --pty -N1 bash -c "docker build -f <dockerfile> -t $IMG . && enroot import dockerd://$IMG"

To pull pre-built images, you can authenticate via Docker and pull/import the container image:

# Login into Docker
docker login -u <docker-username> -p <dockerhub-oat>

# Pull and squash Docker image
srun --pty -N1 bash -c "enroot import dockerd://<repo>:<tag>"

This workflow downloads the image, converts it into an Enroot container bundle, and runs it as an unprivileged user process.

You’ll typically never need to do this when submitting jobs through Pyxis, but it’s a useful way to verify container contents or pre-stage larger images.

Resources

Enroot usage guide: https://github.com/NVIDIA/enroot/blob/main/doc/usage.md