DeepInfer: The Engine DeepSeek Said It Wouldn't Release

It is a fork of vLLM rewritten past recognition, built around five in-house kernel libraries, tuned for one model family on two hardware targets. Here is what is actually inside.

11 min read • View on GitHub • More from deepseek-ai

An editorial illustration of a ship's engine room dominated by a single enormous marine diesel engine. Two fuel lines enter from opposite sides, one heavy and flanged, one ribbed and differently jointed. A blueprint of one specific hull hangs on the bulkhead, and two tiny crew figures check dials at the base.
One production engine, built for one specific hull, fed by two different fuel sources. That is DeepInfer's design compressed into a single image.
Key Takeaways

DeepSeek Said No. Then It Said Yes.

In 2025, DeepSeek published an unusual document. It was not a launch note. It was an explanation of why the company would not open-source its inference engine. The internal engine, the team wrote, was a heavily modified fork of vLLM welded to internal infrastructure, and the maintenance cost of publishing it as a standalone artifact was too high. The plan instead was to upstream improvements and share reusable components.

The reasoning was practical and specific. The announcement named codebase divergence from upstream vLLM and internal infrastructure dependencies as the blockers, and pointed to limited maintenance capacity as the reason a full release was impractical. The stated alternative was to work with existing open-source projects rather than ship the whole engine.

Moving forward, we will work closely with existing open-source projects to:

DeepSeek team, Project authors, Open Infra announcement · The path to open-sourcing the DeepSeek inference engine

Then the company shipped the whole engine anyway. DeepInfer arrives under Apache-2.0, and it is not a trimmed-down sample or a reference implementation. It is the production core with the parts DeepSeek chose to keep back removed and named in the README.

Not a Server. An Engine Shaped Like a Model.

The first thing to understand about DeepInfer is what it is not. It is not a general-purpose server that happens to run DeepSeek models. It is an engine built around one model family, on two specific hardware targets, with the model checkpoint treated as the source of truth for constants the kernels depend on.

The stack splits into four layers. A C++ hot path holds the scheduler, the block manager, the sequence and output pools, input preparation, and sampling prep, exposed to Python through pybind11. Python orchestration sits on top: the LLM entrypoint, LLMEngine, the model executor, and the TileLang kernel wrappers. Underneath, five in-house kernel libraries do the arithmetic. Hardware support is narrow and explicit.

The lineage is stated in every Python file: # Derived from vLLM; modified by DeepSeek. That comment is honest about the starting point and almost useless as a description of the result. This is a fork rewritten past recognition, to the point where the model config, the sampling policy, and the sequence representation all carry DeepSeek-specific assumptions.

tokens_per_block = -1

The thesis of the whole project hides in one default value. In the engine's configuration dataclass, tokens_per_block defaults to -1. That is a sentinel, not a size. When the engine builds its configs, it derives the KV block size from the model metadata and refuses any user value that disagrees.

tokens_per_block: int = -1  # sentinel: derive from the model

# in create_engine_configs:
resolved_tokens_per_block = model_metadata.resolved_tokens_per_block
if tokens_per_block > 0:
    assert tokens_per_block == resolved_tokens_per_block

The assertion is the design principle made executable. The kernels were built for one block size, so a user override that disagrees silently misconfigures the hardware. The engine eliminates that class of bug by treating the model as authoritative and allowing user input only when it matches.

The same discipline shows up in the runtime length arithmetic. The user-facing max_model_len is not the position bound the runtime actually needs, because speculative decoding and zero-overhead scheduling each reserve extra positions. The engine computes the true bound once, in one function, and calls the result the exclusive upper bound of every runtime position.

def _get_runtime_max_model_len(max_model_len):
    nextn = envs.DEEPINFER_USE_NEXTN_DECODE
    scheduler_overhead = nextn + (
        nextn + 1 if envs.DEEPINFER_ZERO_OVERHEAD_SCHEDULING else 0
    )
    draft_overhead = nextn
    return max_model_len + scheduler_overhead + draft_overhead

Getting this arithmetic wrong causes out-of-bounds KV writes, the kind of bug that appears only under a specific combination of features. Encoding it in a single function is not cleverness. It is the minimum required to keep a verticalized engine correct.

The Scheduler Runs Ahead. The Release Runs Behind.

A close-up editorial illustration of a printing press in mid-operation. One freshly printed sheet is lifted away by a hand at the left edge while the inked roller continues across the next sheet in the center. The paper texture, the roller's crosshatched surface, and the mechanical linkage are detailed.
The deferred-release pattern as a physical act: the finished output is being taken away at the same moment the next output is being made.

The most interesting mechanism in DeepInfer is a two-part decoupling of the host from the GPU. The scheduler runs one step ahead of the model under a mode called zero-overhead scheduling, preparing the work for step N+1 while the GPU is still executing step N. That way the GPU never finishes a step and finds the host unprepared.

The other half is deferred release. When a step produces resources, the engine does not free them immediately. It queues the sequence metadata, the request outputs, and the unpin operations in a pool, and it records a torch.cuda.Event at the moment the work that makes them safe to release is enqueued. The actual release happens at the start of the next step, and it synchronizes on that event first.

The result is that the host blocks only when it genuinely needs the memory back. Freeing a KV block the GPU is still reading would corrupt results silently, so the event guard is not optional. Delaying the release by one step hides the synchronization cost behind the next step's compute. The tradeoff is that memory is held one step longer, which matters under pressure. The class even asserts that no prior unpin is pending, enforcing single-outstanding-step semantics.

The scheduler prepares step N+1 while the GPU still executes step N, and step N's resources are only freed at the start of step N+1. Both halves exist to keep the host off the critical path.

The Model Bets on Its Own Drafts

DeepInfer leans hard on multi-token prediction. The model proposes nextn draft tokens, a confidence head predicts each draft's acceptance probability, and the scheduler uses those estimates to decide how much speculation to attempt. That last part is the interesting one: the engine schedules verification based on how confident the model is in its own guesses.

The data structure that carries this is a single class called SampledInfo. It can represent either the token sampled in the current step or a token obtained by verification, and the number of verified tokens is not fixed. Rather than branching per request on whether a draft was accepted, the engine masks over the whole batch.

def _where_update(is_final, current, candidate):
    if candidate is None:
        return current
    if current is None:
        return candidate.clone()
    return torch.where(is_final, candidate, current)

The torch.where call keeps the batch graph static, which matters for CUDA graphs. It avoids a host-device synchronization per request. The cost is that every candidate tensor is computed for the full batch even when most of it is discarded, a deliberate trade of redundant arithmetic for a graph that never branches.

The engine is equally disciplined about what crosses the bus. Only token_ids and confidence_rates travel to the CPU for the scheduler. The logprobs and their indices are written directly to a GPU-side pool and never return to Python. Only what the scheduler actually needs makes the trip.

No KV Cache System. No PD Disaggregation. No Load Balancer.

The README lists three things this release does not include, and the list is worth reading as a scope statement rather than a gap. There is no KV cache system, no prefill/decode disaggregation, and no data-parallel load balancing. LLMEngine's own docstring repeats the point: this release does not include an online-serving wrapper.

Read together, those omissions describe an offline inference core. A KV cache system and PD disaggregation are the machinery you build when you are trying to hold thousands of live sessions and split prefill from decode across machines. Load balancing is what you add when you are routing traffic across replicas. DeepInfer is the compute engine that sits underneath all of that, not the serving product that wraps it.

That framing also explains the defaults. disable_log_stats is true and stop_preempted_request is false out of the box, because an offline batch run does not want the preemption behavior an online server needs. The engine is tuned for throughput on a fixed workload, not for the latency games of interactive serving.

One Model, Two Chips, Five Kernels

Against the general-purpose engines, DeepInfer is playing a different game. It does not compete on how many models it can load or how many vendors' hardware it abstracts. It competes on how tightly one model family can be welded to two specific chip targets.

EngineScopeHardware targetsNotable
DeepInferOne model family (DeepSeek V4.1-Flash)NVIDIA SM100, Huawei Ascend 950Model-derived constants, five in-house kernel libraries, offline inference core
vLLMBroad model ecosystemNVIDIA, with growing accelerator supportPagedAttention, continuous batching; DeepInfer's documented precursor
SGLangBroad model ecosystemNVIDIA, with growing accelerator supportRadix and prefix caching, structured generation at scale
TensorRT-LLMBroad model ecosystemNVIDIA onlyDeep NVIDIA kernel and quantization stack, multi-GPU deployment
LMDeployBroad model ecosystemNVIDIA, with growing accelerator supportDeployment toolkit with quantization and serving options

The reported reaction to the release was less about benchmarks than about the fact of it. The engine that DeepSeek spent 2025 explaining it would not publish is now a public repository under a permissive license.

🚨 DeepSeek open-sourced DeepInfer the inference engine powering its OWN production servers This is the infrastructure behind serving AI models at scale How many more companies will follow? https://t.co/uWfL9BNDc5 https://t.co/kuEWoXiRfc

Salio, @Mr_Salio on X · @Mr_Salio on X

What to Check Before You Adopt It

DeepInfer is a production system with a history, and the honest way to read it is as a snapshot of how one frontier lab builds inference for its own models. That makes it valuable to study and narrow to adopt. Before choosing it over vLLM, SGLang, TensorRT-LLM, or LMDeploy, verify the exact model revisions and hardware it supports, whether the reported performance figures reproduce under an equivalent configuration, and how actively the repository is maintained. The 2025 announcement's own argument about maintenance cost is a reasonable thing to hold the 2026 release to.