Enterprise AI infrastructure rarely fails because โthe GPU wasnโt fast enough.โ It fails because the system around the GPU (CPU, host RAM, networking, and storage) is under-provisioned, creating invisible bottlenecks that show up as low GPU utilization, unstable throughput, or unpredictable latency.
This guide explains the sizing logic behind AceCloudโs H200 NVL IaaS flavors and provides practical, workload-aligned configurations for 1, 2, 4, and 8ร H200 GPUs.
Why H200 NVL changes the sizing conversation
NVIDIAโs H200 platform is defined by two numbers that matter to every exec tracking cost-per-token and time-to-train: 141 GB of HBM3e memory per GPU and up to 4.8 TB/s of memory bandwidth.
More memory per GPU increases the โmodel-per-GPUโ envelope (especially for inference), while higher bandwidth improves the real-world efficiency of attention-heavy workloads where data movement is the bottleneck.
On the NVL platform side, NVIDIAโs enterprise guidance emphasizes validated server and networking configurations (for example, a PCIe-optimized 2-socket / 8-GPU / multi-NIC pattern) with NVLink + RDMA fabrics to reduce CPU overhead and improve multi-GPU scaling.
The sizing principles we apply (and why they matter)
1) Start with memory reality, not GPU count
For inference, GPU memory isnโt just model weights. Itโs also KV cache (which grows with context length and concurrency), plus framework/runtime overhead. Even when a model โfits,โ throughput can collapse as KV cache expands.
A helpful baseline is:
Model memory โ parameters ร bytes per parameter
Example: a 70B model in FP16/BF16 is ~70B ร 2 bytes โ 140 GB just for weights, leaving very little headroom for KV cache and overhead on a 141 GB GPU.
Thatโs why single-GPU inference sizing is often limited by operational headroom (context + concurrency), not just โweights fit.โ
2) Host RAM should scale with total GPU VRAM
NVIDIAโs GPU-ready guidance recommends system memory of at least 2ร total GPU memory, with 4ร being optimal for deep learning training.
This matters because the CPU side still stages data, pins memory for transfers, and supports preprocessing, dataloaders, and system services. Under-sizing host RAM is one of the fastest ways to create non-obvious stalls.
3) CPU cores should scale with GPUs (and with your workloadโs CPU share)
Two anchor points we use:
- NVIDIAโs reference guidance commonly pairs 2 high-end CPUs for 8 GPUs in balanced designs.
- A practical industry rule of thumb is at least 4 CPU cores per GPU accelerator, increasing when CPU work is significant.
In practice, training/fine-tuning generally needs more CPU per GPU than pure inference (dataloader + augmentation + distributed runtime overhead), so we offer separate CPU tiers per scenario.
4) Donโt ignore networking and storage when you scale beyond one server
For multi-node training, NVIDIA recommends at least one 100 Gb NIC (with RDMA) per two GPUs, and emphasizes topology/alignment to reduce bottlenecks.
Storage locality (NVMe/SSD close to GPU PCIe domains) matters for dataset-heavy pipelines.
AceCloud H200 NVL flavor philosophy
We provide two tiers per GPU count (1, 2, 4, 8 GPUs), tuned by workload type:
- Training / Full fine-tuning: host RAM targets around 2ร GPU VRAM as a baseline, with a higher tier for headroom.
- Inference: host RAM targets are lower (because the dominant working set is on-GPU), with a higher tier when you want more concurrency and fewer CPU-side stalls.
Below are the recommended configurations and the โwhyโ behind them.
Training / full fine-tuning flavors (FP32, seq_len 1024, Adam, single-batch baseline)
These configurations assume full fine-tuning (not LoRA/QLoRA) and are designed to keep GPUs consistently fed while maintaining stability for distributed training.
| GPU Count | vCPU Count | RAM (GB) | Workload scenario | Supported model size (approx.) |
|---|---|---|---|---|
| 1 | 14 | 282 | Single-GPU model training (baseline) | Small (7โ10B parameters) |
| 1 | 16 | 320 | Single-GPU training (max memory use) | Small (7โ10B parameters) |
| 2 | 28 | 564 | Distributed 2-GPU training (baseline) | Medium (10โ20B) |
| 2 | 32 | 640 | Distributed 2-GPU training (high RAM) | Medium (10โ20B) |
| 4 | 56 | 1,128 | Distributed 4-GPU training (baseline) | Large (~25B) |
| 4 | 64 | 1,280 | Distributed 4-GPU training (high RAM) | Large (~30B) |
| 8 | 112 | 2,256 | Distributed 8-GPU training (baseline) | X-Large (~40B) |
| 8 | 128 | 2,560 | Distributed 8-GPU training (high RAM) | X-Large (40โ50B) |
Why these ratios:
- Host RAM baseline is aligned to โat least 2ร total GPU memory,โ with extra headroom in the higher tier for training stability and buffering.
- CPU scaling is designed to keep dataloading, preprocessing, and distributed runtime overhead from throttling GPU utilization, using a conservative โcores-per-GPUโ stance for training-heavy pipelines.
Notes for leadership teams:
- If youโre doing parameter-efficient fine-tuning (LoRA/QLoRA) or using lower precision (BF16/FP16/FP8), you can often go larger than the table suggests, because optimizer and activation memory footprints change materially.
- For multi-node scaling, network topology and RDMA matter as much as GPU count.
Inference flavors (FP16, single concurrent user baseline, seq_len 1024, reduced KV-cache usage)
Inference sizing is dominated by three variables you should align to business requirements: context length, concurrency, and tail latency. KV cache grows with context and turns into the real limiter long before โweights fitโ becomes the issue.
| GPU Count | vCPU Count | RAM (GB) | Workload scenario | Supported model size (approx.) |
|---|---|---|---|---|
| 1 | 10 | 212 | Single-GPU inference (low throughput) | Small (~40โ45B) |
| 1 | 12 | 256 | Single-GPU inference (max memory) | Small (~50โ60B) |
| 2 | 20 | 424 | Multi-GPU inference (moderate load) | Medium (~70โ120B) |
| 2 | 24 | 512 | Multi-GPU inference (high throughput) | Medium (~70โ120B) |
| 4 | 40 | 848 | Multi-GPU inference (batch workloads) | Large (~130โ180B) |
| 4 | 48 | 1,024 | Multi-GPU inference (max memory/batch) | Large (~130โ180B) |
| 8 | 80 | 1,696 | Parallel inference for large models | X-Large (~300B) |
| 8 | 96 | 2,048 | High-throughput inference (max resources) | X-Large (~300B) |
Why this works in production:
- The vCPU tiers reflect the difference between โit runsโ and โit serves reliably.โ Tokenization, request routing, batching, streaming responses, and any retrieval pipeline can become CPU-bound at higher QPS.
- The model-size guidance assumes you need operational headroom for KV cache and runtime overhead (especially with longer prompts and multi-turn chat), not just enough space to load weights.
How to choose the right H200 NVL flavor quickly
Use these decision rules to pick the smallest configuration that meets your SLA.
- If you are training or doing full fine-tuning
- Start with the โhigh RAMโ tier when you care about stability and fewer out-of-memory edge cases (especially with larger batches, longer sequences, or heavier dataloaders).
- Move from 1 โ 2 โ 4 โ 8 GPUs based on time-to-train targets and your parallelism strategy, but plan networking early if you will scale across nodes.
- If you are doing inference for an internal tool (low concurrency)
- 1 GPU can be sufficient for mid-sized models if your context window is modest and concurrency is low.
- Choose the higher tier when you see CPU-side saturation or want more consistent latency.
- If you are doing inference for customer-facing products (SLA-driven)
- Prefer 2+ GPUs when you need long contexts, higher concurrency, or predictable tail latency. KV cache growth is the common failure mode here.
What this means for AceCloud customers
These flavors are designed to be practical starting points: balanced CPU-to-GPU, host RAM aligned to established GPU-ready guidance, and clear choices for both training/fine-tuning and inference.
If you share two pieces of information internally (model family/parameter size, and your target context + concurrency), a cloud team can typically right-size quickly and avoid the two most expensive outcomes: overprovisioning โjust in case,โ or underprovisioning and discovering bottlenecks after rollout.
If you want, I can also rewrite this into an โexecutive + technical appendixโ format (same numbers, but with a one-page decision tree at the top and deeper notes on KV cache, quantization, and multi-node networking in the appendix).
