Quick Answer
The most practical open-weight LLMs to self-host include gpt-oss-20b on an NVIDIA L4 24GB, Qwen3.8-27B on an L40S 48GB and gpt-oss-120b on an H100 80GB using compatible optimized checkpoints. Choose by workload, quantization, context length and concurrency because fitting model weights into VRAM does not guarantee production-ready latency or throughput.
Key Takeaways
- Single GPU: Consider gpt-oss-20b for compact reasoning, Qwen3.8-27B for coding and private RAG, and Gemma 4 for multilingual AI.
- High-memory GPU: OpenAI documents gpt-oss-120b running on one 80GB GPU using its optimized MXFP4 checkpoint.
- Multi-GPU: DeepSeek-V4.1-Flash, GLM-5.3, MiniMax M3.1-Flash and Nemotron 3 Ultra require substantial deployment planning.
- Frontier scale:Kimi K3.1 Pro targets advanced agentic workloads, but organizations should compare dedicated cluster costs against API access.
Best Open-Source LLMs to Self-Host in 2026: GPU Comparison
The following comparison uses official developer model cards, published GPU configurations, and estimated model-weight memory.
| Model | Total / Active Parameters | License | Best For | Weight Memory INT4 / FP8 / BF16 | Recommended AceCloud GPU |
|---|---|---|---|---|---|
| gpt-oss-20b (OpenAI) | 21B / 3.6B | Apache 2.0 | Local reasoning, agents | ~10.5 / 21 / 42 GB | L4 24GB, optimized MXFP4 |
| Qwen3.8-27B (Alibaba) | 27B / 27B | Apache 2.0 | Coding, private RAG | ~13.5 / 27 / 54 GB | L40S 48GB, optimized FP8/INT4 |
| Gemma 4 31B (Google) | 30.7B / 30.7B | Apache 2.0 | Multilingual AI, private assistants | ~15.4 / 30.7 / 61.4 GB | L40S 48GB low-bit or RTX PRO 6000 96GB BF16 |
| gpt-oss-120b (OpenAI) | 117B / 5.1B | Apache 2.0 | Reasoning, complex agents | ~58.5 / 117 / 234 GB | H100 80GB, optimized MXFP4 |
| Llama 4 Scout (Meta) | 109B / 17B | Llama 4 Community | Long-context, multimodal | ~54.5 / 109 / 218 GB | H200 141GB, validated low-bit |
| Mistral Small 4 (Mistral AI) | 119B / ~6.5B | Apache 2.0 | Multilingual enterprise AI | ~59.5 / 119 / 238 GB | 2× H200 141GB (FP8); single-GPU low-bit deployment requires validation |
| MiniMax M3.1-Flash (MiniMax) | ~428B / 23B | MiniMax Community | Multimodal coding, agents | ~214 / 428 / 856 GB | Multi-H200 configuration |
| DeepSeek-V4.1-Flash (DeepSeek) | 552B backbone / 8B prefill, 16B decode | MIT | Coding, agentic inference | ~276 / 552 / 1,104 GB | Multi-H200 configuration |
| GLM-5.3 (Z.ai) | ~753B / ~40B | GLM-5.3 custom | Complex reasoning, agents | ~376.5 / 753 / 1,506 GB | Multi-H200 configuration |
| Nemotron 3 Ultra (NVIDIA) | 550B / 55B | OpenMDW-1.1 | Enterprise agents, RAG | ~275 / 550 / 1,100 GB | Supported multi-GPU setup |
| Kimi K3 (Moonshot AI) | 2.8T / 104B | Kimi K3 License | Frontier reasoning, agents | ~1,400 / 2,800 / 5,600 GB | Distributed GPU cluster or API |
Important: These figures represent theoretical raw-weight memory in decimal GB, assuming uniform numerical precision. They are not measured peak VRAM or guaranteed deployment requirements. They exclude KV cache, quantization metadata, runtime buffers, and other allocations.
DeepSeek-V4.1-Flash requires separate checkpoint-level sizing. Its officially reported 552B backbone does not represent every tensor stored in its model repository. Applying the generic formula to 552B alone could underestimate actual memory requirements.
Similarly, gpt-oss and Nemotron use specialized mixed-precision formats. Follow their official deployment documentation rather than treating generic INT4 calculations as validated requirements.
Which LLM Should You Choose for Your GPU Tier?
The best model depends on available GPU infrastructure, not benchmark performance alone. Smaller models can serve private applications on one GPU, while frontier models often need distributed inference.
| Deployment Tier | Recommended Models | Practical Considerations |
|---|---|---|
| Single GPU: 24–48GB | gpt-oss-20b, Qwen3.8-27B, quantized Gemma 4 31B | Private assistants, coding and smaller RAG workloads |
| High-memory single GPU: 80–141GB | gpt-oss-120b, low-bit Llama 4 Scout, low-bit Mistral Small 4 | Validate checkpoint, context and runtime compatibility |
| Multi-GPU server or cluster | MiniMax M3.1-Flash, DeepSeek-V4.1-Flash, GLM-5.3, Nemotron 3 Ultra | Requires memory planning, sharding and suitable interconnects |
| Frontier-scale cluster | Kimi K3.1 Pro | Compare cluster operating costs against API access |
For example, eight H200 141GB GPUs provide 1,128GB of aggregate VRAM, but GLM-5.3’s theoretical BF16 weights exceed this capacity.
This is an illustrative memory calculation, not a claim that AceCloud offers an eight-H200 single-node configuration. Confirm available GPU counts, topology and parallelism support before selecting the infrastructure.
Which is the Best Open-Source LLM Overall?
Kimi K3.1 Pro: Best for Frontier-Scale Reasoning and Agents
Kimi K3.1 Pro is Moonshot AI’s open-weight multimodal model for long-horizon coding, advanced research, 3D creation and complex agentic workflows. It builds on Kimi K3 with improved reasoning, coding and long-context capabilities.
According to Moonshot AI’s official model card, Kimi K3.1 Pro has 2.8 trillion total parameters, 104 billion active parameters and a context window of 1,048,576 tokens. It supports text and image inputs, with MXFP4-quantized expert weights for more efficient inference.
However, Kimi K3.1 Pro is not practical for single-GPU deployment. Despite activating only a fraction of its parameters per token, its full model weights require substantial distributed GPU memory.
Published vLLM deployment configurations include eight NVIDIA B300 GPUs or two GB300 trays. Teams should also account for KV cache, concurrency and runtime overhead.
For enterprises without dedicated GPU clusters, API access is a practical starting point. Self-hosting becomes more attractive when sustained utilisation, data privacy or infrastructure control justifies the investment.
Best for: Frontier reasoning, long-horizon coding and complex multimodal agents.
GPU recommendation: 8× B300 or 8× GB300 distributed deployment, subject to hardware availability and validation; consider API access first.
Which Open-Source LLM is Best for Coding?
DeepSeek-V4.1-Flash: Best for Large-Scale Coding and Agents
DeepSeek-V4.1-Flash targets coding agents, repository-level development tasks and high-volume inference workloads.
DeepSeek describes a 552B-parameter backbone with different active-parameter paths for prefill and decoding. This architecture aims to improve computational efficiency without activating the entire model for each token.
Its reported coding and reasoning improvements are based on vendor evaluations, not AceCloud benchmarks. Despite its sparse computation, the model’s total parameter count makes it a multi-GPU serving candidate.
Best for: Coding automation, repository-level development and complex agent workflows.
GPU recommendation: Multi-GPU H200 configuration, subject to checkpoint and deployment validation.
Qwen3.8-27B: Best for Private Coding on a Single GPU
Qwen3.8-27B replaces the older Qwen3.6-27B recommendation. The Apache 2.0-licensed dense model supports coding, document reasoning, tool use, and image-based workflows.
Its 27B parameters require approximately 27GB of raw weight memory at idealized FP8 precision, making an NVIDIA L40S 48GB a reasonable candidate with a supported optimized checkpoint. However, higher concurrency, longer context windows, and multimodal workloads can increase memory requirements.
Qwen3.8-27B is worth evaluating for private development assistants, internal coding tools and enterprise knowledge applications where organizations want direct control over model deployment.
Best for: Private coding assistants, internal development tools and RAG.
GPU recommendation: NVIDIA L40S 48GB, with validated quantization.
For a detailed comparison, read Qwen3.8-27B vs Flash-Next.
Which Open-Source LLM is Best for Reasoning?
GLM-5.3: Best for Complex Reasoning and Long-Running Agents
GLM-5.3 replaces GLM-5.2 as the primary Z.ai model in this comparison. With approximately 753B total parameters, GLM-5.3 targets complex reasoning, multi-step coding, and long-running agentic workflows.
Its MoE architecture reduces the amount of computation required per token, but the complete model still demands substantial GPU memory.
It is therefore more suitable for multi-GPU deployment than conventional single-GPU inference. Teams with limited infrastructure can also evaluate GLM-5.3-Flash, a separate and smaller model variant.
For document-intensive applications, combine reasoning models with retrieval, summarization, and document routing rather than relying exclusively on large context windows.
Best for: Advanced reasoning, research agents, and complex coding.
GPU recommendation: Multi-GPU configuration with validated precision and parallelism.
gpt-oss-120b: Best for Optimized Single-GPU Reasoning
OpenAI’s gpt-oss-120b contains approximately 117B total parameters and 5.1B active parameters. Its optimized MXFP4 checkpoint can run on a single 80GB GPU with a compatible serving runtime.
This makes it particularly relevant to organizations seeking sophisticated reasoning capabilities without operating a large multi-GPU cluster. However, GPU memory compatibility does not guarantee the required latency, throughput, or concurrency.
Teams should benchmark production workloads before choosing between gpt-oss-120b and larger distributed models.
Best for: Reasoning, enterprise assistants and tool-based agents.
GPU recommendation: NVIDIA H100 80GB using optimized MXFP4.
Which Open-Source LLM is Best for Enterprise AI and RAG?
Nemotron 3 Ultra: Best for Enterprise Agents and High-Stakes RAG
NVIDIA Nemotron 3 Ultra features 550B total parameters and 55B active parameters. It targets enterprise agents, tool use, and complex private AI workloads where response quality, reliability and operational control matter.
Its scale makes multi-GPU infrastructure necessary for typical supported deployment configurations. For enterprise RAG, evaluate not only model reasoning but also retrieval accuracy, source grounding, auditability, and inference latency.
Organizations should additionally review the model’s OpenMDW-1.1 license before commercial deployment.
Best for: Enterprise AI agents, high-stakes RAG and advanced reasoning.
GPU recommendation: Vendor-supported multi-GPU configuration.
For a smaller NVIDIA model, see AceCloud’s Nemotron 3 Nano Multi-GPU Fine-Tuning Guide.
Qwen3.8-27B: Best for Smaller Private RAG Systems
A larger generator does not automatically produce better retrieval-augmented generation results.
A reliable RAG system also depends on embeddings, document chunking, retrieval quality, reranking and source attribution. Qwen3.8-27B is a practical candidate for internal knowledge assistants and private document retrieval systems.
Start with an appropriate context window, retrieve relevant passages, rerank results and evaluate answer faithfulness before increasing model size. For smaller enterprise RAG workloads, this approach may offer better resource efficiency than deploying an unnecessarily large model.
Which Open-Source LLM Is Best for Agents and Tool Calling?
The best model for agentic AI depends on tool-use reliability, reasoning complexity, and infrastructure constraints. Strong candidates include:
- DeepSeek-V4.1-Flash: Large-scale coding automation and complex agent workflows.
- GLM-5.3: Multi-step reasoning and research-oriented agents.
- Nemotron 3 Ultra: Enterprise agents requiring private deployment.
- Qwen3.8-27B: Smaller private coding and tool-based workflows.
- gpt-oss-20b: Compact reasoning and local agent applications.
- MiniMax M3.1-Flash: Multimodal agent workflows and coding tasks.
MiniMax M3.1-Flash has approximately 428B total and 23B active parameters, making it suitable for evaluation on appropriately configured multi-GPU infrastructure. However, agent quality cannot be judged using language-generation benchmarks alone.
Evaluate JSON schema validity, tool-selection accuracy, argument correctness, failed-call recovery and successful task completion. A model that generates fluent responses but repeatedly selects incorrect tools may be unsuitable for production agents.
Which is the Best Multilingual Open-Source LLM?
Mistral Small 4: Best for Multilingual Enterprise Deployment
Mistral Small 4 combines multilingual understanding, instruction-following, coding, and multimodal capabilities. Its MoE architecture contains 119B total parameters and approximately 6.5B active parameters, with Apache 2.0 licensing.
The model is relevant to multilingual document processing, structured enterprise workflows, and private AI assistants. However, its total weight size makes it more appropriate for high-memory GPU servers than lightweight local systems.
Best for: Multilingual enterprise applications and structured workflows.
GPU recommendation: H200 141GB with a validated low-bit checkpoint.
Gemma 4: Best for Local and Edge AI
Google’s Gemma 4 family includes smaller edge-oriented models and a dense 31B-parameter variant. The family supports more than 140 languages, with context length and multimodal capabilities varying by model version.
The 31B variant contains approximately 30.7B parameters, requiring about 61.4GB for raw BF16 language-model weights before additional memory allocations. An RTX PRO 6000 96GB is worth evaluating for BF16 inference, while an L40S 48GB may support a suitable quantized checkpoint.
For multilingual enterprise applications, test actual performance on the target languages, regional terminology, and document formats.
Best for: Local assistants, multilingual AI and private applications.
GPU recommendation: L40S 48GB for optimized low-bit deployment or RTX PRO 6000 96GB for BF16.
How Much GPU VRAM Do You Need to Self-Host an LLM?
GPU memory is a primary constraint when selecting an open-weight model. A model may perform well on benchmarks but fail to serve a production workload if its weights, KV cache and runtime exceed available memory.
How to Estimate LLM Weight Memory
A simplified calculation is:
Raw weight memory (GB) ≈ Total parameters in billions × Bytes per parameter
Common precision estimates are:
| Precision | Bytes Per Parameter | Approximate Memory for 27B Parameters |
|---|---|---|
| INT4 | 0.5 | 13.5GB |
| FP8 | 1 | 27GB |
| BF16 | 2 | 54GB |
These values represent theoretical weight storage, not total inference memory. For a 27B dense model, applying a preliminary 20% allowance gives approximately 16.2GB at INT4, 32.4GB at FP8 and 64.8GB at BF16.
This allowance is a planning assumption, not a universal runtime requirement. KV cache, quantization metadata, context length, and concurrent requests must be evaluated separately.
For comparison, Google’s official Gemma 4 memory guide estimates approximately 69.9GB to load Gemma 4 31B at BF16, including its documented loading allowance.
Why MoE Models Need Special Memory Planning
Mixture-of-Experts models activate only a subset of their parameters during each token’s computation. However, active parameters should not be confused with the complete model’s memory requirements.
For example, a model with 16B active parameters may contain several hundred billion total parameters. Active parameters primarily influence per-token computation, while total parameters determine the overall model-weight storage and memory budget.
Unless the serving implementation supports offloading or specialized expert management, the full model weights must be accommodated across the deployment infrastructure. For multi-GPU systems, also evaluate tensor or expert parallelism and GPU interconnect performance.
How to Choose the Right Open-Source LLM
Selecting a model requires balancing capability, infrastructure requirements, and production economics.
- Define Your WorkloadIdentify whether the model will support coding assistants, enterprise RAG, multilingual applications, structured extraction, or autonomous agents. Define measurable success criteria before comparing models.
- Assess Infrastructure RequirementsDocument GPU memory, context length, expected concurrency, P95 latency, data residency, privacy and licensing requirements. Ensure the target model has a compatible serving framework and checkpoint format.
- Evaluate Models on Real TasksCompare shortlisted models using representative internal prompts and datasets. Include difficult queries, tool failures, retrieval errors, and other scenarios likely to occur in production.
- Benchmark the Deployment ConfigurationTest the exact model checkpoint, precision, GPU configuration, and serving engine. Measure throughput, latency, peak memory usage, and completed-task accuracy under realistic traffic.
- Compare Total Deployment CostsConsider GPU rental, utilization, storage, networking, operational overhead and retries. The cheapest GPU or smallest model does not necessarily deliver the lowest cost per successful task.
What Does It Cost to Self-Host an LLM on Cloud GPUs?
Self-hosting costs depend on more than GPU hourly rates. Organizations should evaluate total infrastructure expenditure against successful workload completion.
Cost per successful task = Total serving cost ÷ Successfully completed tasks
The calculation should include GPU usage, storage, networking, engineering effort, retries and operational overhead.
For example, a dedicated GPU may be more economical under sustained and predictable utilization, while a managed API may be better suited to intermittent workloads. AceCloud offers GPU options ranging from NVIDIA L4 and L40S to RTX PRO 6000, H100 and H200 for different inference requirements.
Use the current AceCloud GPU pricing to estimate infrastructure expenses, then benchmark your own model throughput to determine the cost per request or successful task.
These costs should not be confused with the theoretical model-weight requirements or public benchmark scores.
What Are the Main Limitations of Open-Source LLM Rankings?
Public benchmarks provide a useful starting point, but their results depend on prompts, datasets, reasoning budgets, context lengths, and evaluation methods.
A model with a large context window may still struggle to retrieve relevant details, while a quantized checkpoint can behave differently from the version used in published evaluations.
Self-hosted models can also hallucinate, mishandle sensitive information or follow malicious instructions retrieved from external sources.
Production deployments need access controls, output validation, tool sandboxing, prompt-injection defenses, monitoring, and audit logs.
Use leaderboards to create a shortlist, then validate performance, security, and cost using your own workloads.
Deploy the Right Open-Weight LLM on AceCloud GPUs
The best open-weight LLM is not necessarily the largest or highest-ranked model. It is the one that meets your workload’s quality, performance, privacy, and cost requirements.
Start with gpt-oss-20b for compact reasoning, Qwen3.8-27B for private coding and RAG or optimized gpt-oss-120b for advanced single-GPU reasoning. Larger models require careful infrastructure planning.
AceCloud offers NVIDIA L4, L40S, RTX PRO 6000, H100 and H200 cloud GPUs for AI inference and enterprise deployment requirements.
Explore AceCloud Cloud GPUs or book a free consultation to identify a suitable GPU configuration based on your model, inference workload and budget.
Frequently Asked Questions
For single-GPU deployment, gpt-oss-20b and Qwen3.8-27B are practical starting points. Optimized gpt-oss-120b supports an 80GB GPU configuration. Larger models such as GLM-5.3 and Kimi K3.1 Pro require more substantial infrastructure.
gpt-oss-20b has an optimized checkpoint designed to operate within approximately 16GB of memory, making an L4 24GB a candidate. Actual capacity depends on runtime compatibility, context length, and concurrency.
It depends on the checkpoint and precision. OpenAI documents gpt-oss-120b running on one 80GB GPU using MXFP4. A generic 120B BF16 checkpoint would require substantially more memory.
Self-hosting can be cost-effective with predictable traffic and high utilization. APIs may be more economical for intermittent workloads. Compare total infrastructure costs against successful tasks at the required response quality and latency.
Qwen3.8-27B is a candidate for private coding and RAG on an optimized single GPU. DeepSeek-V4.1-Flash suits evaluation for larger coding agents with multi-GPU infrastructure. Retrieval quality and answer faithfulness remain important for RAG.
Many can, but terms vary. Always verify the exact checkpoint’s license, attribution obligations, redistribution conditions and hosted-service restrictions.
Use total parameters to estimate complete model-weight memory, not only active parameters. Then account for precision, KV cache, runtime buffers, concurrency, GPU interconnects, and offloading.