DeepInfer: The Engine DeepSeek Said It Wouldn't Release
It is a fork of vLLM rewritten past recognition, built around five in-house kernel libraries, tuned for one model family on two hardware targets. Here is what is actually inside.
- DeepInfer is not a general-purpose server that happens to support DeepSeek models; it is a model-shaped engine where the checkpoint is the source of truth for hardware-facing constants.
- The scheduler runs one step ahead of the GPU while resource release lags one step behind, so the host never waits on compute.
- The release deliberately omits a KV cache system, prefill/decode disaggregation, and data-parallel load balancing, and each omission is a scope decision about offline inference.
- DeepInfer competes on fit rather than breadth: one model family, two chip targets, five in-house kernel libraries.
DeepSeek Said No. Then It Said Yes.
In 2025, DeepSeek published an unusual document. It was not a launch note. It was an explanation of why the company would not open-source its inference engine. The internal engine, the team wrote, was a heavily modified fork of vLLM welded to internal infrastructure, and the maintenance cost of publishing it as a standalone artifact was too high. The plan instead was to upstream improvements and share reusable components.
The reasoning was practical and specific. The announcement named codebase divergence from upstream vLLM and internal infrastructure dependencies as the blockers, and pointed to limited maintenance capacity as the reason a full release was impractical. The stated alternative was to work with existing open-source projects rather than ship the whole engine.
Moving forward, we will work closely with existing open-source projects to:
Then the company shipped the whole engine anyway. DeepInfer arrives under Apache-2.0, and it is not a trimmed-down sample or a reference implementation. It is the production core with the parts DeepSeek chose to keep back removed and named in the README.
Not a Server. An Engine Shaped Like a Model.
The first thing to understand about DeepInfer is what it is not. It is not a general-purpose server that happens to run DeepSeek models. It is an engine built around one model family, on two specific hardware targets, with the model checkpoint treated as the source of truth for constants the kernels depend on.
The stack splits into four layers. A C++ hot path holds the scheduler, the block manager, the sequence and output pools, input preparation, and sampling prep, exposed to Python through pybind11. Python orchestration sits on top: the LLM entrypoint, LLMEngine, the model executor, and the TileLang kernel wrappers. Underneath, five in-house kernel libraries do the arithmetic. Hardware support is narrow and explicit.
- C++ hot path: scheduler, block manager, sequence and output pools, input preparation, sampling prep, behind pybind11.
- Python orchestration: the LLM entrypoint, LLMEngine, model executor, and TileLang kernel wrappers.
- Five in-house kernel libraries: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect.
- Two hardware targets: NVIDIA SM100 and Huawei Ascend 950 with CANN 9.2.
- License: Apache-2.0, with a default checkpoint of DeepSeek-V4.1-Flash.
The lineage is stated in every Python file: # Derived from vLLM; modified by DeepSeek. That comment is honest about the starting point and almost useless as a description of the result. This is a fork rewritten past recognition, to the point where the model config, the sampling policy, and the sequence representation all carry DeepSeek-specific assumptions.
tokens_per_block = -1
The thesis of the whole project hides in one default value. In the engine's configuration dataclass, tokens_per_block defaults to -1. That is a sentinel, not a size. When the engine builds its configs, it derives the KV block size from the model metadata and refuses any user value that disagrees.
tokens_per_block: int = -1 # sentinel: derive from the model
# in create_engine_configs:
resolved_tokens_per_block = model_metadata.resolved_tokens_per_block
if tokens_per_block > 0:
assert tokens_per_block == resolved_tokens_per_block
The assertion is the design principle made executable. The kernels were built for one block size, so a user override that disagrees silently misconfigures the hardware. The engine eliminates that class of bug by treating the model as authoritative and allowing user input only when it matches.
The same discipline shows up in the runtime length arithmetic. The user-facing max_model_len is not the position bound the runtime actually needs, because speculative decoding and zero-overhead scheduling each reserve extra positions. The engine computes the true bound once, in one function, and calls the result the exclusive upper bound of every runtime position.
def _get_runtime_max_model_len(max_model_len):
nextn = envs.DEEPINFER_USE_NEXTN_DECODE
scheduler_overhead = nextn + (
nextn + 1 if envs.DEEPINFER_ZERO_OVERHEAD_SCHEDULING else 0
)
draft_overhead = nextn
return max_model_len + scheduler_overhead + draft_overhead
Getting this arithmetic wrong causes out-of-bounds KV writes, the kind of bug that appears only under a specific combination of features. Encoding it in a single function is not cleverness. It is the minimum required to keep a verticalized engine correct.
The Scheduler Runs Ahead. The Release Runs Behind.
The most interesting mechanism in DeepInfer is a two-part decoupling of the host from the GPU. The scheduler runs one step ahead of the model under a mode called zero-overhead scheduling, preparing the work for step N+1 while the GPU is still executing step N. That way the GPU never finishes a step and finds the host unprepared.
The other half is deferred release. When a step produces resources, the engine does not free them immediately. It queues the sequence metadata, the request outputs, and the unpin operations in a pool, and it records a torch.cuda.Event at the moment the work that makes them safe to release is enqueued. The actual release happens at the start of the next step, and it synchronizes on that event first.
The result is that the host blocks only when it genuinely needs the memory back. Freeing a KV block the GPU is still reading would corrupt results silently, so the event guard is not optional. Delaying the release by one step hides the synchronization cost behind the next step's compute. The tradeoff is that memory is held one step longer, which matters under pressure. The class even asserts that no prior unpin is pending, enforcing single-outstanding-step semantics.
The Model Bets on Its Own Drafts
DeepInfer leans hard on multi-token prediction. The model proposes nextn draft tokens, a confidence head predicts each draft's acceptance probability, and the scheduler uses those estimates to decide how much speculation to attempt. That last part is the interesting one: the engine schedules verification based on how confident the model is in its own guesses.
The data structure that carries this is a single class called SampledInfo. It can represent either the token sampled in the current step or a token obtained by verification, and the number of verified tokens is not fixed. Rather than branching per request on whether a draft was accepted, the engine masks over the whole batch.
def _where_update(is_final, current, candidate):
if candidate is None:
return current
if current is None:
return candidate.clone()
return torch.where(is_final, candidate, current)
The torch.where call keeps the batch graph static, which matters for CUDA graphs. It avoids a host-device synchronization per request. The cost is that every candidate tensor is computed for the full batch even when most of it is discarded, a deliberate trade of redundant arithmetic for a graph that never branches.
The engine is equally disciplined about what crosses the bus. Only token_ids and confidence_rates travel to the CPU for the scheduler. The logprobs and their indices are written directly to a GPU-side pool and never return to Python. Only what the scheduler actually needs makes the trip.
No KV Cache System. No PD Disaggregation. No Load Balancer.
The README lists three things this release does not include, and the list is worth reading as a scope statement rather than a gap. There is no KV cache system, no prefill/decode disaggregation, and no data-parallel load balancing. LLMEngine's own docstring repeats the point: this release does not include an online-serving wrapper.
Read together, those omissions describe an offline inference core. A KV cache system and PD disaggregation are the machinery you build when you are trying to hold thousands of live sessions and split prefill from decode across machines. Load balancing is what you add when you are routing traffic across replicas. DeepInfer is the compute engine that sits underneath all of that, not the serving product that wraps it.
That framing also explains the defaults. disable_log_stats is true and stop_preempted_request is false out of the box, because an offline batch run does not want the preemption behavior an online server needs. The engine is tuned for throughput on a fixed workload, not for the latency games of interactive serving.
One Model, Two Chips, Five Kernels
Against the general-purpose engines, DeepInfer is playing a different game. It does not compete on how many models it can load or how many vendors' hardware it abstracts. It competes on how tightly one model family can be welded to two specific chip targets.
| Engine | Scope | Hardware targets | Notable |
|---|---|---|---|
| DeepInfer | One model family (DeepSeek V4.1-Flash) | NVIDIA SM100, Huawei Ascend 950 | Model-derived constants, five in-house kernel libraries, offline inference core |
| vLLM | Broad model ecosystem | NVIDIA, with growing accelerator support | PagedAttention, continuous batching; DeepInfer's documented precursor |
| SGLang | Broad model ecosystem | NVIDIA, with growing accelerator support | Radix and prefix caching, structured generation at scale |
| TensorRT-LLM | Broad model ecosystem | NVIDIA only | Deep NVIDIA kernel and quantization stack, multi-GPU deployment |
| LMDeploy | Broad model ecosystem | NVIDIA, with growing accelerator support | Deployment toolkit with quantization and serving options |
The reported reaction to the release was less about benchmarks than about the fact of it. The engine that DeepSeek spent 2025 explaining it would not publish is now a public repository under a permissive license.
🚨 DeepSeek open-sourced DeepInfer the inference engine powering its OWN production servers This is the infrastructure behind serving AI models at scale How many more companies will follow? https://t.co/uWfL9BNDc5 https://t.co/kuEWoXiRfc
What to Check Before You Adopt It
DeepInfer is a production system with a history, and the honest way to read it is as a snapshot of how one frontier lab builds inference for its own models. That makes it valuable to study and narrow to adopt. Before choosing it over vLLM, SGLang, TensorRT-LLM, or LMDeploy, verify the exact model revisions and hardware it supports, whether the reported performance figures reproduce under an equivalent configuration, and how actively the repository is maintained. The 2025 announcement's own argument about maintenance cost is a reasonable thing to hold the 2026 release to.





