Multi-GPU LLM Training on Kubernetes: Build a Production-Ready Pipeline

Multi-GPU LLM training has become essential as teams move from small model experiments to larger fine-tuning and pretraining workloads. A single GPU often becomes a bottleneck because of memory limits, long training cycles, and slow iteration speed.

However, production-grade multi-GPU training is not just about requesting more GPUs. A reliable pipeline needs GPU-aware scheduling, reproducible containers, distributed launch coordination, checkpoint recovery, NCCL-ready networking, monitoring, cost controls, and security guardrails.

Kubernetes provides the operational foundation for this. It gives platform teams a consistent way to schedule GPU workloads, isolate teams, enforce quotas, connect with CI/CD, and standardize training jobs across environments.

In this guide, we will build a practical production pipeline for multi-GPU LLM training on Kubernetes using NVIDIA GPU Operator, PyTorch torchrun, Kubeflow Trainer, checkpoint storage, DCGM-based GPU monitoring, and scheduling policies.

Reference Architecture: What You Are Building

Use this as the target architecture for a production-ready training platform.

At a high level, the pipeline contains:

The NVIDIA GPU Operator is useful because it can automate several GPU software components, including NVIDIA drivers, the Kubernetes device plugin, NVIDIA Container Toolkit, GPU Feature Discovery, and DCGM-based monitoring.

Prerequisites to Consider for Multi-GPU LLM Training

Before building the pipeline, align on these platform requirements.

Single-Node vs Multi-Node Multi-GPU Training

Before implementing anything, decide which training pattern you actually need.

Kubeflow Trainer supports PyTorch distributed training and can run DDP, FSDP, FSDP2, and other PyTorch-supported distributed algorithms.

A practical rollout path is:

1. Validate one GPU.
2. Validate one node with multiple GPUs.
3. Validate multi-node distributed training.
4. Add checkpoint restore testing.
5. Add queueing, quotas, monitoring, and cost controls.

Step 1: Provision GPU Nodes and Isolate Them

Create a dedicated GPU node pool instead of mixing GPU and CPU workloads on the same nodes. Then label and taint GPU nodes so only approved workloads can land there.

Example labels:

kubectl label node gpu-node-1 accelerator=nvidia
kubectl label node gpu-node-1 gpu.model=a100
kubectl label node gpu-node-1 workload-type=gpu-training
kubectl label node gpu-node-1 topology.kubernetes.io/zone=zone-a

Example taint:

kubectl taint nodes gpu-node-1 dedicated=gpu-training:NoSchedule

Training workloads must include a matching toleration:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu-training"
  effect: "NoSchedule"

Use node affinity to keep training pods on the intended GPU pool:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: workload-type
          operator: In
          values:
          - gpu-training
        - key: gpu.model
          operator: In
          values:
          - a100

This avoids accidental placement on the wrong nodes and makes scheduling behavior easier to debug.

Step 2: Enable GPUs for Kubernetes Scheduling

Kubernetes uses device plugins to expose hardware resources such as GPUs to the kubelet. After the GPU plugin is installed, nodes advertise resources such as nvidia.com/gpu, and pods can request them in their container limits.

For production, prefer NVIDIA GPU Operator because it manages the GPU software lifecycle more completely. For a minimal setup, you can use the NVIDIA device plugin.

Validate GPU capacity

kubectl describe node gpu-node-1 | grep -A5 "Capacity"
kubectl describe node gpu-node-1 | grep -A5 "Allocatable"

You should see something similar to:

nvidia.com/gpu: 8

Run a GPU smoke test

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu-training"
    effect: "NoSchedule"
  nodeSelector:
    workload-type: gpu-training
  containers:
  - name: cuda
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["bash", "-lc", "nvidia-smi && sleep 10"]
    resources:
      limits:
        nvidia.com/gpu: 1

Apply and inspect logs:

kubectl apply -f gpu-smoke-test.yaml
kubectl logs gpu-smoke-test

NOTE: Kubernetes GPU resources should be specified in limits. If requests are also specified, the request and limit values must match.

Step 3: Containerize Training for Reproducibility

Your training image should be treated as an immutable release artifact.

A basic Dockerfile can look like this:

FROM pytorch/pytorch:2.4.1-cuda12.4-cudnn9-runtime

WORKDIR /workspace

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY train.py .
COPY configs/ ./configs/

ENV PYTHONUNBUFFERED=1
ENV TOKENIZERS_PARALLELISM=false

CMD ["python", "train.py"]

Recommended image practices:

Example image tag:

docker build -t registry.example.com/llm-trainer:${GIT_SHA} .
docker push registry.example.com/llm-trainer:${GIT_SHA}

For every training run, log:

run_id
git_commit
image_digest
dataset_version
config_hash
world_size
checkpoint_path

Step 4: Run a Single-Node Multi-GPU Training Job

Start with single-node multi-GPU training before moving to multi-node. It is easier to debug and avoids distributed networking issues.

Example Kubernetes Job using PyTorch torchrun:

apiVersion: batch/v1
kind: Job
metadata:
  name: llm-train-single-node
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: llm-training
        run-id: run-2026-01-20-a
    spec:
      restartPolicy: Never
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "gpu-training"
        effect: "NoSchedule"
      nodeSelector:
        workload-type: gpu-training
        gpu.model: a100
      containers:
      - name: trainer
        image: registry.example.com/llm-trainer:REPLACE_WITH_GIT_SHA
        command:
        - bash
        - -lc
        - |
          torchrun \
            --standalone \
            --nproc_per_node=8 \
            train.py \
            --config configs/train.yaml \
            --output_dir /checkpoints/run-2026-01-20-a
        env:
        - name: NCCL_DEBUG
          value: "WARN"
        - name: CUDA_DEVICE_MAX_CONNECTIONS
          value: "1"
        resources:
          requests:
            cpu: "32"
            memory: "256Gi"
          limits:
            nvidia.com/gpu: 8
            memory: "256Gi"
        volumeMounts:
        - name: checkpoints
          mountPath: /checkpoints
      volumes:
      - name: checkpoints
        persistentVolumeClaim:
          claimName: llm-checkpoints-pvc

This pattern is useful when one node has enough GPU memory and compute for the workload. For example, one 8-GPU node is often easier to operate than an 8-GPU job spread across several nodes.

Step 5: Move to Multi-Node Training

Move to multi-node training only after single-node training is stable. Multi-node jobs add coordination, networking, rank assignment, failure handling, and NCCL tuning.

For production teams, use a controller such as Kubeflow Trainer instead of hand-rolling worker pods. Kubeflow Trainer automatically configures PyTorch distributed environment variables such as WORLD_SIZE, RANK, and LOCAL_RANK, and it uses torchrun for PyTorch distributed jobs.

Example Kubeflow Trainer-style Python workflow:

from kubeflow.trainer import TrainerClient, CustomTrainer

def train_func():
    import os
    import torch
    import torch.distributed as dist

    backend = "nccl" if torch.cuda.is_available() else "gloo"
    dist.init_process_group(backend=backend)

    local_rank = int(os.getenv("LOCAL_RANK", "0"))
    torch.cuda.set_device(local_rank)

    print({
        "world_size": dist.get_world_size(),
        "rank": dist.get_rank(),
        "local_rank": local_rank,
    })

    # Import and call your actual training loop here.
    # train_model(config_path="/workspace/configs/train.yaml")

client = TrainerClient()

client.train(
    trainer=CustomTrainer(
        func=train_func,
        resources_per_node={
            "gpu": 8,
            "cpu": 32,
            "memory": "256Gi",
        },
        num_nodes=2,
    )
)

Use multi-node training when:

Step 6: Add Gang Scheduling or Queueing for Large Jobs

Distributed training jobs often require all workers to start together. If only half the workers start, the job can hang while still holding expensive GPUs.

This is where gang scheduling or queue-based admission helps. Kubernetes v1.35 includes alpha gang scheduling support that schedules a group of pods on an all-or-nothing basis when the configured minimum pod count can be satisfied.

Use gang scheduling, Volcano, Kueue, or another batch-aware scheduler when:

For most teams, this rollout works well:

1. Start with native Kubernetes Jobs.
2. Move distributed jobs to Kubeflow Trainer.
3. Add Kueue, Volcano, or gang scheduling when large multi-pod jobs create contention.
4. Add priority classes and quotas for team-level fairness.

Step 7: Configure Checkpoint Storage and Restore

Checkpointing is not optional for production LLM training. Node failures, preemptions, networking issues, and job evictions are normal at scale.

A production checkpoint should include:

Example PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llm-checkpoints-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Ti
  storageClassName: fast-shared-storage

Recommended checkpoint policy:

A checkpoint is only useful if it can be restored. Add a staging restore drill to every major training pipeline release.

Step 8: Validate Networking and NCCL Readiness

Multi-node training often fails at the network layer before it fails at the model layer.

Before scaling, validate:

Useful NCCL environment variables:

env:
- name: NCCL_DEBUG
  value: "WARN"
- name: NCCL_IB_DISABLE
  value: "0"
- name: NCCL_SOCKET_IFNAME
  value: "eth0"
- name: TORCH_DISTRIBUTED_DEBUG
  value: "DETAIL"

Do not blindly copy these into every environment. Validate the correct network interface, transport, and driver behavior for your cluster.

Step 9: Build Observability into the Pipeline

Observability should be part of the training platform from day one.

Track three layers of metrics.

GPU metrics

Training metrics

Kubernetes metrics

Example Prometheus queries:

avg by (pod) (DCGM_FI_DEV_GPU_UTIL)
avg by (pod) (DCGM_FI_DEV_FB_USED)
increase(kube_pod_container_status_restarts_total[15m])
kube_pod_status_phase{phase="Pending"}

Recommended alerts:

Step 10: Optimize Dataset Throughput

Many teams add GPUs before fixing the input pipeline. That usually increases cost without improving throughput.

Common dataset bottlenecks:

Recommended practices:

• Use sharded datasets.
• Warm hot data to local NVMe when possible.
• Track data loader time separately from GPU compute time.
• Keep checkpoint writes from blocking training.
• Log samples/sec and tokens/sec for every run.

A simple rule: if GPU utilization is low, do not scale to more GPUs until the input pipeline is fixed.

Step 11: Add Cost Controls

Multi-GPU training can become expensive quickly. Cost controls should be enforced at the platform level, not left to individual teams.

Basic cost formula:

training_cost = GPU_hour_price × GPU_count × wall_clock_hours

Example:

$3.00 per GPU-hour × 8 GPUs × 20 hours = $480

Cost levers:

Example namespace quota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-training-quota
  namespace: team-llm
spec:
  hard:
    requests.cpu: "512"
    requests.memory: "4Ti"
    limits.nvidia.com/gpu: "32"

Example priority class:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: production-training
value: 100000
globalDefault: false
description: "Priority class for approved production LLM training jobs."

Use lower priority for experiments and higher priority for approved production runs.

Step 12: Use GPU Sharing Carefully

GPU sharing can improve utilization, but it is not always appropriate for LLM training.

NVIDIA GPU time-slicing allows workloads to share GPU time, but unlike MIG, time-slicing does not provide memory or fault isolation between replicas.

For serious multi-GPU LLM training, full GPU allocation is usually the safest default.

Step 13: Secure the Training Platform

Security controls matter because training jobs often access private datasets, model weights, credentials, and internal infrastructure.

Baseline controls:

Example service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-trainer
  namespace: team-llm

Example pod usage:

spec:
  serviceAccountName: llm-trainer

Do not run training jobs with broad cluster-admin permissions.

Production Readiness Checklist

Use this checklist before declaring the pipeline production-ready.

Troubleshooting Multi-GPU LLM Training on Kubernetes

Orchestration Options for Multi-GPU LLM Training

Recommended default path:

Native Job → torchrun → Kubeflow Trainer → Kueue/Volcano for queueing and gang scheduling

Launch Your Multi-GPU LLM Training Faster with AceCloud

A production-ready multi-GPU LLM training pipeline is not just about spinning up GPUs. It is about repeatability, scheduling discipline, checkpoint resilience, observability, cost control, and secure operations.

AceCloud helps teams operationalize GPU-first AI infrastructure for training, fine-tuning, and inference workloads. Its AI/ML infrastructure page highlights multi-GPU and multi-node distributed training, checkpointing for long-running jobs, preconfigured PyTorch and Hugging Face environments, Kubernetes autoscaling, and MLOps support.

Whether you are moving from experimentation to production or standardizing GPU infrastructure for multiple teams, AceCloud can help you deploy reliable multi-GPU training environments faster.

CTA: Ready to scale beyond one GPU? Book your free consultation with our cloud GPU expert to deploy on AceCloud cloud GPUs and operationalize multi-GPU training on Kubernetes.

Author Popular Post

Carolyn Weitz

author

Carolyn began her cloud career at a fast-growing SaaS company, where she led the migration from on-prem infrastructure to a fully containerized, cloud-native architecture using Kubernetes. Since then, she has worked with a range of companies from early-stage startups to global enterprises helping them implement best practices in cloud operations, infrastructure automation, and container orchestration. Her technical expertise spans across AWS, Azure, and GCP, with a focus on building scalable IaaS environments and streamlining CI/CD pipelines. Carolyn is also a frequent contributor to cloud-native open-source communities and enjoys mentoring aspiring engineers in the Kubernetes ecosystem.

• 7 Pillars Of Cost Optimization with AWS
• IaaS Architecture and Components – A 2025 Guide for Businesses
• 5 Key Benefits of Modernizing on AWS Cloud
• Cloud Computing Matrix: Public, Private, and Hybrid Cloud Architectures
• OpenStack: The Catalyst of the Public Cloud Market