Enterprise AI infrastructure rarely fails because “the GPU wasn’t fast enough.” It fails because the system around the GPU (CPU, host RAM, networking, and storage) is under-provisioned, creating invisible bottlenecks that show up as low GPU utilization, unstable throughput, or unpredictable latency.
This guide explains the sizing logic behind AceCloud’s H200 NVL IaaS flavors and provides practical, workload-aligned configurations for 1, 2, 4, and 8× H200 GPUs.
Why H200 NVL changes the sizing conversation
NVIDIA’s H200 platform is defined by two numbers that matter to every exec tracking cost-per-token and time-to-train: 141 GB of HBM3e memory per GPU and up to 4.8 TB/s of memory bandwidth.
More memory per GPU increases the “model-per-GPU” envelope (especially for inference), while higher bandwidth improves the real-world efficiency of attention-heavy workloads where data movement is the bottleneck.
On the NVL platform side, NVIDIA’s enterprise guidance emphasizes validated server and networking configurations (for example, a PCIe-optimized 2-socket / 8-GPU / multi-NIC pattern) with NVLink + RDMA fabrics to reduce CPU overhead and improve multi-GPU scaling.
The sizing principles we apply (and why they matter)
1) Start with memory reality, not GPU count
For inference, GPU memory isn’t just model weights. It’s also KV cache (which grows with context length and concurrency), plus framework/runtime overhead. Even when a model “fits,” throughput can collapse as KV cache expands.
A helpful baseline is:
Model memory ≈ parameters × bytes per parameter
Example: a 70B model in FP16/BF16 is ~70B × 2 bytes ≈ 140 GB just for weights, leaving very little headroom for KV cache and overhead on a 141 GB GPU.
That’s why single-GPU inference sizing is often limited by operational headroom (context + concurrency), not just “weights fit.”
2) Host RAM should scale with total GPU VRAM
NVIDIA’s GPU-ready guidance recommends system memory of at least 2× total GPU memory, with 4× being optimal for deep learning training.
This matters because the CPU side still stages data, pins memory for transfers, and supports preprocessing, dataloaders, and system services. Under-sizing host RAM is one of the fastest ways to create non-obvious stalls.
3) CPU cores should scale with GPUs (and with your workload’s CPU share)
Two anchor points we use:
- NVIDIA’s reference guidance commonly pairs 2 high-end CPUs for 8 GPUs in balanced designs.
- A practical industry rule of thumb is at least 4 CPU cores per GPU accelerator, increasing when CPU work is significant.
In practice, training/fine-tuning generally needs more CPU per GPU than pure inference (dataloader + augmentation + distributed runtime overhead), so we offer separate CPU tiers per scenario.
4) Don’t ignore networking and storage when you scale beyond one server
For multi-node training, NVIDIA recommends at least one 100 Gb NIC (with RDMA) per two GPUs, and emphasizes topology/alignment to reduce bottlenecks.
Storage locality (NVMe/SSD close to GPU PCIe domains) matters for dataset-heavy pipelines.
AceCloud H200 NVL flavor philosophy
We provide two tiers per GPU count (1, 2, 4, 8 GPUs), tuned by workload type:
- Training / Full fine-tuning: host RAM targets around 2× GPU VRAM as a baseline, with a higher tier for headroom.
- Inference: host RAM targets are lower (because the dominant working set is on-GPU), with a higher tier when you want more concurrency and fewer CPU-side stalls.
Below are the recommended configurations and the “why” behind them.
Training / full fine-tuning flavors (FP32, seq_len 1024, Adam, single-batch baseline)
These configurations assume full fine-tuning (not LoRA/QLoRA) and are designed to keep GPUs consistently fed while maintaining stability for distributed training.
| GPU Count | vCPU Count | RAM (GB) | Workload scenario | Supported model size (approx.) |
|---|---|---|---|---|
| 1 | 14 | 282 | Single-GPU model training (baseline) | Small (7–10B parameters) |
| 1 | 16 | 320 | Single-GPU training (max memory use) | Small (7–10B parameters) |
| 2 | 28 | 564 | Distributed 2-GPU training (baseline) | Medium (10–20B) |
| 2 | 32 | 640 | Distributed 2-GPU training (high RAM) | Medium (10–20B) |
| 4 | 56 | 1,128 | Distributed 4-GPU training (baseline) | Large (~25B) |
| 4 | 64 | 1,280 | Distributed 4-GPU training (high RAM) | Large (~30B) |
| 8 | 112 | 2,256 | Distributed 8-GPU training (baseline) | X-Large (~40B) |
| 8 | 128 | 2,560 | Distributed 8-GPU training (high RAM) | X-Large (40–50B) |
Why these ratios:
- Host RAM baseline is aligned to “at least 2× total GPU memory,” with extra headroom in the higher tier for training stability and buffering.
- CPU scaling is designed to keep dataloading, preprocessing, and distributed runtime overhead from throttling GPU utilization, using a conservative “cores-per-GPU” stance for training-heavy pipelines.
Notes for leadership teams:
- If you’re doing parameter-efficient fine-tuning (LoRA/QLoRA) or using lower precision (BF16/FP16/FP8), you can often go larger than the table suggests, because optimizer and activation memory footprints change materially.
- For multi-node scaling, network topology and RDMA matter as much as GPU count.
Inference flavors (FP16, single concurrent user baseline, seq_len 1024, reduced KV-cache usage)
Inference sizing is dominated by three variables you should align to business requirements: context length, concurrency, and tail latency. KV cache grows with context and turns into the real limiter long before “weights fit” becomes the issue.
| GPU Count | vCPU Count | RAM (GB) | Workload scenario | Supported model size (approx.) |
|---|---|---|---|---|
| 1 | 10 | 212 | Single-GPU inference (low throughput) | Small (~40–45B) |
| 1 | 12 | 256 | Single-GPU inference (max memory) | Small (~50–60B) |
| 2 | 20 | 424 | Multi-GPU inference (moderate load) | Medium (~70–120B) |
| 2 | 24 | 512 | Multi-GPU inference (high throughput) | Medium (~70–120B) |
| 4 | 40 | 848 | Multi-GPU inference (batch workloads) | Large (~130–180B) |
| 4 | 48 | 1,024 | Multi-GPU inference (max memory/batch) | Large (~130–180B) |
| 8 | 80 | 1,696 | Parallel inference for large models | X-Large (~300B) |
| 8 | 96 | 2,048 | High-throughput inference (max resources) | X-Large (~300B) |
Why this works in production:
- The vCPU tiers reflect the difference between “it runs” and “it serves reliably.” Tokenization, request routing, batching, streaming responses, and any retrieval pipeline can become CPU-bound at higher QPS.
- The model-size guidance assumes you need operational headroom for KV cache and runtime overhead (especially with longer prompts and multi-turn chat), not just enough space to load weights.
How to choose the right H200 NVL flavor quickly
Use these decision rules to pick the smallest configuration that meets your SLA.
- If you are training or doing full fine-tuning
- Start with the “high RAM” tier when you care about stability and fewer out-of-memory edge cases (especially with larger batches, longer sequences, or heavier dataloaders).
- Move from 1 → 2 → 4 → 8 GPUs based on time-to-train targets and your parallelism strategy, but plan networking early if you will scale across nodes.
- If you are doing inference for an internal tool (low concurrency)
- 1 GPU can be sufficient for mid-sized models if your context window is modest and concurrency is low.
- Choose the higher tier when you see CPU-side saturation or want more consistent latency.
- If you are doing inference for customer-facing products (SLA-driven)
- Prefer 2+ GPUs when you need long contexts, higher concurrency, or predictable tail latency. KV cache growth is the common failure mode here.
What this means for AceCloud customers
These flavors are designed to be practical starting points: balanced CPU-to-GPU, host RAM aligned to established GPU-ready guidance, and clear choices for both training/fine-tuning and inference.
If you share two pieces of information internally (model family/parameter size, and your target context + concurrency), a cloud team can typically right-size quickly and avoid the two most expensive outcomes: overprovisioning “just in case,” or underprovisioning and discovering bottlenecks after rollout.
If you want, I can also rewrite this into an “executive + technical appendix” format (same numbers, but with a one-page decision tree at the top and deeper notes on KV cache, quantization, and multi-node networking in the appendix).
