A detailed guide to building enterprise‑grade GPU clusters with Hopper and Ada GPUs, covering power, cooling, networking and workload scheduling.
Executive summary
Enterprise AI workloads place divergent demands on compute, memory, bandwidth and orchestration. The Hopper‑based H100 and H200 GPUs deliver high‑density memory and an NVLink fabric that excels for large‑scale model training, while the Ada‑based L40S targets inference, 3D rendering and video pipelines with a PCIe‑only, lower‑power form factor. Designing a cluster therefore begins with a clear understanding of each accelerator’s thermal design power, memory capacity, interconnect bandwidth and multi‑instance GPU (MIG) capability. This article walks through the technical fundamentals, component selection, step‑by‑step implementation, security considerations, enterprise use cases, benefits, limitations, operational practices and best‑practice guidance needed to build robust H100, H200 and L40S clusters that can serve the full AI lifecycle from training to serving.
Introduction
Artificial‑intelligence models have exploded from millions to billions of parameters in just a few years, stretching the limits of traditional CPU farms. GPUs accelerate matrix multiplication, tensor operations and transformer kernels, but the practical limits of a cluster are governed by the physical characteristics of the accelerator cards themselves. The H100 and H200 belong to NVIDIA’s Hopper generation, offering fourth‑generation Tensor Cores, a Transformer Engine with FP8 support, and configurable MIG partitions that enable multiple isolated workloads on a single physical device. The L40S belongs to the Ada Lovelace generation, providing a universal GPU for generative AI inference, 3D rendering and video workloads. Understanding each product’s memory type, bandwidth, power envelope and host interface is essential before any rack‑level design can proceed, because those parameters dictate server chassis selection, power distribution, cooling strategy and network topology.
Technical fundamentals
Hopper H100
Form factors – Available as an SXM module (up to 700 W TDP, configurable) and as an NVL PCIe card (350‑400 W). The SXM version plugs into a specialized HGX board that supplies high‑density power and cooling, while the NVL version fits into a standard PCIe slot.
Memory – 80 GB of HBM3 on the SXM version and 94 GB on the NVL version, delivering peak bandwidth of roughly 3.35 TB/s for SXM and 3.9 TB/s for NVL.
Interconnect – NVLink provides 900 GB/s of bidirectional bandwidth on SXM and 600 GB/s on NVL; both also expose PCIe Gen 5 at 128 GB/s.
MIG – Up to seven partitions can be created, giving 10 GB per instance on SXM or 12 GB on NVL. Each partition appears as an independent GPU to the operating system.
Features – Built‑in confidential computing, a Transformer Engine that supports FP8 arithmetic, and fourth‑generation Tensor Cores.
A deep dive into the memory‑scheduling, quantisation, and continuous batching techniques that make large language model serving efficient on modern GPU clusters.
How Ray’s core primitives, Train, and Serve enable teams to orchestrate large‑scale training and low‑latency inference on GPU clusters without replacing existing infrastructure.
Kingson Jebaraj·18 Nov 2026
Read
Hopper H200
Form factors – SXM (up to 700 W) and NVL (up to 600 W) versions, mirroring the H100 physical options but with a higher power ceiling for the NVL card.
Memory – 141 GB of HBM3E, a next‑generation high‑bandwidth memory that pushes raw bandwidth to about 4.8 TB/s for both SXM and NVL.
Interconnect – NVLink 900 GB/s and PCIe Gen 5 at 128 GB/s, identical to the H100 in link speed but with greater memory depth.
MIG – Up to seven partitions, each offering 18 GB on SXM or 16.5 GB on NVL, enabling larger per‑partition workloads.
Features – Same power profile as the H100, confidential computing enclave, and the same fourth‑generation Tensor Core architecture.
Ada L40S
Form factor – Dual‑slot passive PCIe Gen 4 x16 card with a 350 W TDP, designed for standard rack‑mount servers without specialized cooling.
Memory – 48 GB of GDDR6 equipped with ECC, delivering about 864 GB/s of bandwidth.
Interconnect – PCIe Gen 4 at 64 GB/s bidirectional; the card does not support NVLink, so inter‑GPU communication relies on the host PCIe bus.
MIG – Not supported; the L40S presents a single logical device.
Features – Secure boot with a root of trust, NEBS Level 3 compliance, virtual‑GPU (vGPU) support, and dedicated video encode/decode engines for media workloads.
Architecture and components
The cluster architecture can be visualized as three logical layers that must be engineered in concert:
Compute nodes – Servers that house the GPU modules. Hopper SXM cards require HGX‑style chassis that provide board‑level cooling, power distribution and high‑density NVLink bridges. Hopper NVL and L40S cards fit into conventional rack‑mount servers, but those servers still need sufficient airflow and power headroom to sustain the cards' TDP.
Fabric – High‑speed links that move tensors between GPUs inside a node and between nodes across the rack. For Hopper SXM, intra‑node bandwidth is provided by NVLink, while inter‑node traffic uses InfiniBand (typically NDR Quantum‑2). L40S relies exclusively on PCIe Gen 4; the fabric for L40S clusters therefore consists of Ethernet or standard InfiniBand switches that aggregate the PCIe traffic.
Management stack – The software ecosystem that schedules work, provisions drivers, and orchestrates MIG. Common choices include Slurm, PBS or Kubernetes as the scheduler, the NVIDIA driver suite, the NVIDIA Container Toolkit, and the MIG manager utilities (nvidia-smi mig). The stack must be aware of the partition topology so that jobs can request either whole GPUs or specific MIG instances.
Detailed implementation approach
Power and cooling planning
SXM modules – Design for up to 700 W per GPU. A four‑GPU HGX node therefore draws roughly 2.8 kW for the accelerators alone; CPU, memory, storage and chassis overhead add another 1 kW or more. The rack power distribution unit (PDU) and the cooling plant must be sized for this peak load, with headroom for transient spikes.
NVL cards – At 350‑400 W per card, eight‑card servers remain within a typical 2 kW rack envelope, but you must still verify that the server’s internal airflow can keep GPU temperatures below the thermal throttling point.
L40S cards – Also rated at 350 W, these dual‑slot cards fit comfortably into standard air‑cooled servers. However, when densely populating a rack with many L40S cards, the cumulative heat load can still become significant, so hot‑aisle/cold‑aisle containment is recommended.
Hopper SXM nodes – Connect each server to an InfiniBand leaf switch using NDR Quantum‑2. This provides sub‑microsecond latency and tens of terabits per second of aggregate bandwidth, which is essential for gradient‑reduction traffic during large‑scale training.
L40S nodes – Aggregate the PCIe bandwidth of each card through a top‑of‑rack Ethernet or InfiniBand switch. Because each GPU is limited to 64 GB/s of PCIe bandwidth, the network design should avoid oversubscription that could become a bottleneck for inference request traffic.
Storage and memory hierarchy
Training workloads typically ingest terabytes of raw data. Locating high‑performance NVMe SSDs close to the GPU—either on the same motherboard or via a direct‑attach PCIe backplane—reduces host‑to‑GPU data transfer latency. Inference clusters, by contrast, often cache model weights in system RAM and rely on the GPU’s high‑bandwidth memory for rapid tensor access; the storage layer can therefore be slower, such as SATA or lower‑cost NVMe drives, provided the inference latency budget is met.
Software stack
Driver installation – Use the NVIDIA driver version that explicitly lists support for Hopper and Ada GPUs. Verify driver compatibility with the operating system and the chosen container runtime.
Container runtime – Deploy the NVIDIA Container Toolkit so that Docker or OCI‑compatible runtimes can expose GPUs to containers without manual device binding.
MIG configuration – Enable MIG on H100/H200 via nvidia-smi mig commands. For example, to create seven 10 GB instances on an H100 SXM:
nvidia-smi mig -cgi 1,1,1,1,1,1,1 -C
Scheduler integration – Choose a scheduler that understands MIG, such as Slurm with the gres plugin. Define MIG profiles in the scheduler configuration so that users can request a specific partition size (e.g., gres=gpu:10gb).
Monitoring – Install NVIDIA Data Center GPU Manager (DCGM) to collect telemetry on utilization, temperature, power draw and error counters. Enable export to a time‑series database for long‑term capacity planning.
Architecture and data‑flow
Training pipelines generally adopt a data‑parallel approach. Each GPU processes a mini‑batch of training examples, computes gradients locally, and then participates in an all‑reduce operation to synchronize gradients across the node and the cluster. In Hopper SXM nodes the NVLink fabric provides a high‑speed, low‑latency path for the all‑reduce, dramatically reducing the time spent on gradient aggregation. For inter‑node reduction, the InfiniBand NDR Quantum‑2 network carries the reduced gradients to other nodes.
Inference pipelines often use model‑parallel or pipeline‑parallel strategies, especially when serving very large language models that exceed the memory of a single GPU. The H200’s 141 GB of HBM3E enables a single GPU to hold a model that would otherwise need to be split across two H100 devices. When using L40S for inference, the host memory serves as the staging area; the model is streamed over PCIe into the GPU’s GDDR6 as needed. This design places greater emphasis on PCIe throughput and on the efficiency of the inference engine (such as TensorRT) to minimize data movement.
Security and governance
Confidential computing – Hopper GPUs embed a hardware enclave that encrypts data while it resides in GPU memory. This protects proprietary model weights and sensitive training data from privileged software on the host, meeting regulatory requirements for data‑in‑use protection.
Secure boot – The L40S includes a root‑of‑trust mechanism that verifies firmware signatures before execution, ensuring that only authorized code runs on the card.
Access control – Implement role‑based access control (RBAC) in the cluster manager to restrict who can create, modify or destroy MIG instances. Because each MIG partition presents an isolated address space, improper handling could expose other tenants’ workloads.
Audit logging – Enable DCGM telemetry and forward logs to a central SIEM system. Capture GPU utilization, temperature, power usage and error events to satisfy compliance audits and to provide forensic evidence if a breach is suspected.
Enterprise use cases
GPU
Primary use case
Reasoning
H100 SXM
Large‑scale model training
80 GB HBM3, 900 GB/s NVLink, up to seven MIG instances enable dense tensor computation and multi‑tenant training on a single node.
H200 SXM
Generative AI with very large context windows
141 GB HBM3E and 4.8 TB/s bandwidth allow single‑GPU models that exceed H100 memory limits, reducing the need for complex model‑parallel orchestration.
L40S
Real‑time inference, rendering, video transcoding
48 GB GDDR6, secure boot, virtual‑GPU support and lower power fit into existing GPU‑accelerated inference servers, providing a cost‑effective solution for latency‑sensitive workloads.
Benefits and limitations
Benefits
Hopper GPUs deliver massive memory bandwidth and NVLink, which reduces cross‑GPU communication latency during training and enables efficient data‑parallel scaling.
MIG partitions allow multiple isolated workloads on the same physical card, improving overall utilization and supporting multi‑tenant SaaS offerings.
The L40S provides a versatile, lower‑power option for inference, graphics and video pipelines, and it includes security features such as secure boot and vGPU licensing.
Limitations
SXM modules require specialized chassis; they cannot be retrofitted into standard rack servers, which can increase capital expenditure and limit flexibility.
MIG adds scheduling complexity; incorrect partition sizing can lead to memory fragmentation and underutilization.
L40S lacks NVLink and MIG, making it unsuitable for multi‑GPU training workloads that depend on high‑speed intra‑node communication.
Operational considerations
Thermal monitoring – Deploy temperature sensors at the rack level and configure alert thresholds for GPUs approaching their TDP limits. Automatic throttling can protect hardware but may impact workload performance.
Power budgeting – Compute per‑rack power draw using the TDP values (for example, eight × 700 W = 5.6 kW for an eight‑GPU SXM node) and provision UPS capacity with sufficient headroom for burst loads.
Firmware updates – Follow NVIDIA’s release notes carefully; apply firmware and driver updates in a staged manner to avoid mismatched versions across the cluster, which can cause instability.
Capacity planning – Use DCGM to collect baseline utilization metrics over a representative period. Adjust MIG partition sizes based on observed workload patterns to maximize efficiency while avoiding memory starvation.
Backup and disaster recovery – Snapshot container images and configuration files regularly. Store model checkpoints on redundant storage arrays that are separate from the compute tier.
Best practices
Match form factor to rack design – Deploy SXM GPUs only in servers engineered for their cooling and power needs; use standard servers for NVL and L40S cards.
Isolate high‑power nodes – Place Hopper SXM servers on dedicated power circuits to avoid tripping breakers and to simplify load‑balancing.
Leverage NVLink for intra‑node communication – Configure NCCL to prioritize NVLink before falling back to PCIe, ensuring the fastest possible gradient reduction.
Standardize on a single PCIe generation – Mixing Gen 5 and Gen 4 can cause sub‑optimal bandwidth allocation and complicate firmware compatibility.
Document MIG profiles – Maintain a registry that maps jobs to specific MIG instances; this simplifies troubleshooting and helps audit resource consumption.
Enable confidential computing for sensitive data – Activate the enclave mode on Hopper GPUs when processing regulated datasets to meet data‑in‑use encryption requirements.
Validate network performance – Run synthetic bandwidth and latency tests (e.g., using NCCL benchmarks) after fabric installation to confirm that NVLink and InfiniBand meet the expected throughput.
Implement health checks – Integrate GPU health probes into the cluster’s monitoring stack so that a failing GPU can be gracefully drained without interrupting jobs.
Analysis
The three GPUs address distinct segments of the enterprise AI market. Hopper’s high‑bandwidth memory and NVLink make it the logical choice for training massive transformer models, while the newer H200’s larger HBM3E memory pushes the envelope for single‑GPU generative workloads that would otherwise require multi‑GPU model‑parallel setups. The L40S, lacking NVLink and MIG, excels in inference, graphics and video pipelines where lower power, built‑in security and virtual‑GPU capabilities are paramount. A mixed‑environment cluster—combining Hopper nodes for training and L40S nodes for serving—allows enterprises to optimize cost and performance across the AI lifecycle, allocating the most appropriate hardware to each phase of model development and deployment.
What the sources do not settle
The exact power consumption of a fully loaded H200 SXM node under sustained training loads remains a preliminary figure; the specifications note that values are subject to change based on firmware and workload characteristics.
Real‑world latency differences between NVLink‑based gradient reduction and PCIe‑based reduction for the L40S are not quantified on the product pages, leaving a gap in performance prediction for inference‑heavy clusters.
The impact of confidential computing on training throughput is described as a feature but without concrete performance measurements, making it difficult to assess the trade‑off between security and raw speed.
Conclusion
Designing an enterprise GPU cluster is a systems‑engineering problem anchored in the concrete specifications of the chosen accelerators. By aligning power budgets, cooling capacity, interconnect topology and MIG partitioning with the H100, H200 and L40S specifications, platform teams can build reliable, secure clusters that meet the divergent demands of training and inference. Careful attention to thermal design, network fabric selection and scheduler configuration will ensure that the theoretical capabilities of Hopper and Ada GPUs translate into practical productivity for enterprise AI initiatives.
References
NVIDIA H100 Tensor Core GPU documentation
NVIDIA H200 Tensor Core GPU documentation
NVIDIA L40S GPU documentation
NVIDIA L40 GPU documentation
Related reading on CloudExchangers
These published CloudExchangers articles are further desk reading. They are not sources for the technical claims above.