Architecture

AMD MI300X

The MI300X combines eight MI300X accelerators into a single system using the infinity architecture technology. It boasts 192 GB of HBM memory, 5.3TB/s of peak memory bandwidth, and a unified memory architecture.

The AMD MI300X provides lower latency and better consistency at larger batch sizes.

With the MI300 series, AMD is introducing the Accelerator Complex Die (XCD), which contains the GPU computational elements of the processor along with the lower levels of the cache hierarchy.

Learn more on AMD's MI300X architecture here.

Nvidia H100

The H100 uses Hopper architecture and Tensor Core GPUs as well as fourth-generation Tensor Cores that can speed up inference by up to 30X and reduce memory usage as well as supporting a maximum of 120GB of memory. While the H100 may outperform the MI300X in smaller quantities, in larger batch sizes the MI300X outperforms where its larger VRAM helps it handle more workloads efficiently.

The NVIDIA H100 SXM offers higher throughput at smaller to medium batch sizes

Learn more about NVIDIA's H100 architecture here.