DeepInfer Explained: DeepSeek's Open-Source Inference Engine

Quick Answer

DeepSeek's in-house production inference framework, open-sourced Oct 20, 2026: runs V4.1-Flash, on NVIDIA SM100 and Huawei Ascend 950, Apache 2.0.

On October 20, 2026, DeepSeek open-sourced the inference engine behind its own production services: DeepInfer. It is not a model — it is the infrastructure software that turns trained weights into fast, cheap answers: splitting work across chips, batching requests, deciding how fast each token arrives. For the first time, DeepSeek has published its complete production inference stack: the first release runs DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950 under Apache 2.0. This guide explains what it is, where the speed comes from, who it is for, and how it differs from vLLM and SGLang.

What DeepInfer Is, in 150 Seconds

An inference framework is the software layer that turns trained weights into a working service: it splits the model across chips, batches requests, and decides how fast each token comes back. A large share of what an AI company pays per answer goes into this layer. DeepInfer is the software DeepSeek uses to run its own production services. On October 20, 2026, deepseek-ai/DeepInfer went open source on GitHub under Apache 2.0, alongside deepseek-recipe — DeepSeek's production tokenization component, which keeps token encoding consistent with the official API. The hard facts of the first release (all from the official README, verified Oct 21, 2026): one supported model, DeepSeek-V4.1-Flash; offline inference only; two supported hardware targets, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Four days after release, the repo had 9,221 GitHub stars (verified via the GitHub API; the repo was created Oct 16, 2026).

Where the Speed Comes From

First, the foundation: DeepInfer officially acknowledges it was "originally built primarily on vLLM", with ideas from SGLang and NVIDIA TensorRT-LLM. The speedup is a combination of several techniques: 1. Optimized kernels: it bundles DeepSeek's previously open-sourced FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom kernels written in TileLang. Kernels are the low-level routines that squeeze work out of specific chips — the same model can differ by an order of magnitude in throughput with a different kernel set. 2. Parallelism: attention data parallelism (Attention DP), balanced context parallelism, and MoE expert parallelism with expert load balancing (EPLB). MoE models activate only a subset of experts per token; expert parallelism spreads different experts' compute across cards. 3. Speculative decoding (DSpark): the model drafts up to 5 tokens at a time (controlled by DEEPINFER_USE_NEXTN_DECODE, default 5; master switch DEEPINFER_ENABLE_MTP_VARLEN=1), then a confidence-based scheduler decides which to keep. Correct guesses skip the wait of generating token by token. 4. Decoder SWA bounded replay: reduces decoder prefill work by replaying only a bounded suffix (DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, 140 retained tokens by default). 5. Structured outputs: generation can be constrained to a regex, a JSON schema or an EBNF grammar — handy when feeding agents and business systems without an extra parsing layer. 6. Batch-invariant inference and deterministic sampling (NVIDIA only): with DEEPINFER_BATCH_INVARIANT=1, a request returns identical results regardless of batch composition; with a fixed seed you get deterministic sampling. That matters for benchmarks and audits that must be reproducible. Asynchronous scheduling (CPU batch scheduling overlapping device execution) is on by default, pushing hardware utilization further.

Hardware and Environment: Not Built for Your Desktop

The README's minimum requirements: Linux, GCC >= 13.0, Python 3.12; CUDA 13 on NVIDIA, CANN 9.2 on Ascend. The only supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs — both data-center hardware. Consumer cards like the RTX 4090 are not on the list. Build: create an empty Python 3.12 environment, set DEEPINFER_ACCELERATOR=nvidia (or ascend), then run ./build.sh. The notable part is the domestic-chip support: a production-grade inference stack natively supporting both NVIDIA and Huawei Ascend is rare among public projects. For teams running Ascend 950 clusters, this is the first DeepSeek production implementation they can study directly.

The Official Performance Numbers (With a Caveat)

These figures were published by DeepSeek (via press coverage) and have not been independently verified — cite them with that in mind: - 155 trillion tokens processed per day per production cluster - 96% input cache hit rate — the vast majority of request context is served from cache instead of being recomputed - Around 400 ms average time-to-first-token - 230 tokens/second generation speed, up to 300 tokens/second for coding - Official benchmarks measured on 4 SM100 GPUs (prefill) and 64 SM100 GPUs (decode), each within one scale-up domain Don't read these as numbers you can reproduce — read them as the design target: high throughput, low latency, optimized for large-scale production. Reproducing them takes a cluster, not a workstation.

Getting Started: Build From Source to First Offline Run

You need supported hardware (SM100 or Ascend 950) and the environment above. Steps from the official README (verified Oct 21, 2026): Step 1: prepare the environment. Linux + GCC >= 13.0 + Python 3.12, create and activate an empty Python 3.12 virtual environment; CUDA 13 on NVIDIA, CANN 9.2 on Ascend. Step 2: build from source. git clone https://github.com/deepseek-ai/DeepInfer , set DEEPINFER_ACCELERATOR=nvidia (ascend on the Ascend platform), then run ./build.sh. Step 3: first offline run. The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on first run; to use a local copy, set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017. Then run: python examples/offline_inference.py. Two practical tips: enable speculative decoding with DEEPINFER_ENABLE_MTP_VARLEN=1; constrain output format by passing so_args (with schema_type and schema) to SamplingParams — see examples/_structured_output.py in the repo. More usage in the repo's examples/README.md.

Who It's For — and Who Should Skip It

Good fit: - Researchers and engineers studying DeepSeek's production stack: the first time anyone has open-sourced the complete inference implementation behind its "cheap and fast" reputation - Teams with Ascend 950 clusters: a directly usable reference implementation on domestic hardware - Kernel and inference-optimization learners: FlashMLA / DeepGEMM / DeepEP / TileLang kernels are all in the repo - Offline batch inference: bulk document processing, offline evaluation — no online service needed Not a fit: - Running on consumer GPUs: the bar is SM100 / Ascend 950, desktops are out of scope - Serving a public API out of the box: v1 is offline inference, and the README explicitly lists what's missing — a KV cache system, prefill/decode disaggregation, load balancing across replicas; you build the rest yourself - Running other models: only DeepSeek-V4.1-Flash for now (DeepSeek says inference code for future models will land in the same repo, but that's later) - Plug-and-play expectations: build from source with strict environment requirements, not a pip install If you are looking for deployment guides for open models on consumer GPUs, this site's ComfyUI VRAM and quantization guide (https://yuliuswiki.com/comfyui-diffusion-model-vs-checkpoint) is a better fit.

Sources & Disclaimer

Key facts verified Oct 21, 2026: DeepInfer official README (features, hardware, build steps, benchmark setup): https://github.com/deepseek-ai/DeepInfer GitHub API (9,221 stars, created 2026-10-16, Apache-2.0): https://api.github.com/repos/deepseek-ai/DeepInfer DeepSeek's published production figures (via press): https://cryptobriefing.com/deepseek-open-sources-deepinfer-inference-framework/ Background and the vLLM connection: https://madrobot.blog/2026/10/20/deepseek-deepinfer-open-source-inference-engine-huawei-ascend-nvidia/ deepseek-recipe and release details: https://ailetter.ai.kr/en/d/2026-10-20/9 Feature details (DSpark, structured outputs, offline limits): https://goldenera.si/news/deepseek-deepinfer-open-source-inference/ Disclaimer: this site is not affiliated with or endorsed by DeepSeek; this article is a technical introduction only. DeepInfer, DeepSeek and related trademarks and code copyrights belong to their respective owners. Performance figures are DeepSeek's own published numbers and have not been independently verified; hardware support and features follow the latest official README.

DeepInfer vs Popular Inference Frameworks

Dimension	DeepInfer	vLLM	SGLang	TensorRT-LLM
Positioning & base	DeepSeek's production stack, open-sourced; officially built primarily on vLLM	Community-standard general inference engine (PagedAttention)	Berkeley open source, RadixAttention, strong prefix caching	Official NVIDIA stack, deeply optimized for its own GPUs
Models in first release	DeepSeek-V4.1-Flash only	Broad Hugging Face ecosystem coverage	Mainstream open models	Mainstream open models (NVIDIA GPUs)
License	Apache 2.0	Apache 2.0	Apache 2.0	Apache 2.0
Hardware support	NVIDIA SM100 + Huawei Ascend 950 (per README)	NVIDIA CUDA first, community AMD / domestic-chip ports	Primarily NVIDIA	NVIDIA GPUs only
Structured output	Yes (regex / JSON schema / EBNF grammar)	Yes (JSON / regex / grammar-guided)	Yes	Yes
Online serving	Offline inference only in v1; no KV cache, no prefill/decode disaggregation, no load balancing	Full online serving	Full online serving	Full online serving
Best for	Studying DeepSeek's production stack, Ascend 950 deployments	General self-hosted inference	Multi-turn chat / prefix-reuse workloads	Deep tuning inside the NVIDIA ecosystem

Frequently Asked Questions

How is DeepInfer related to vLLM? Will it replace it?

The README states DeepInfer was "originally built primarily on vLLM", with ideas from SGLang and TensorRT-LLM. Think of it as DeepSeek's deeply customized fork for its own production workload — the first release supports only its own model, so it won't replace general-purpose vLLM. For general inference serving, vLLM remains the safer pick.

Can I run DeepInfer on an RTX 4090 or other consumer GPUs?

No. Officially supported hardware is limited to NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs — both data-center hardware. Consumer cards are not on the list, and the project is designed for production clusters anyway.

Does DeepInfer currently only support DeepSeek's own models?

Yes. The README's supported-models list contains only DeepSeek-V4.1-Flash. DeepSeek says inference code for future models will land in the same repository, but for now it is a V4.1-Flash engine.

Can I serve a public API with DeepInfer out of the box?

Not yet. The first release does offline inference only; the README explicitly lists what a complete online service still needs: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. It fits research and offline batch workloads.

What is deepseek-recipe, released alongside DeepInfer?

DeepInfer is the inference engine (it runs the model); deepseek-recipe is DeepSeek's production tokenization component, keeping a self-hosted service's tokenization behavior consistent with the official API. Together they reproduce DeepSeek's full production pipeline.

Is DeepInfer OK for commercial use? What about the model weights?

The framework code is Apache 2.0 — commercial use, modification and redistribution are allowed. Model weights are separate: the default deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint carries its own license on Hugging Face, so check the license field on the repo page before using it.