# Before renting GPUs for DeepInfer, read the missing-pieces list **Summary:** DeepInfer releases DeepSeek's inference engine, but leaves key service layers to the operator. I would price that integration and measure accepted output under a latency target before renting a production cluster. - Canonical: https://markhuang.ai/news/deepinfer-missing-serving-system - Language: en - Author: [Mark Huang](https://markhuang.ai/about) - Published: 2026-10-20 - Section: News - Tags: DeepInfer, DeepSeek, LLM Inference, AI Infrastructure, Inference Benchmarks, Self-Hosting - Source: [DeepSeek on GitHub](https://github.com/deepseek-ai/DeepInfer) - License: https://creativecommons.org/licenses/by-nc/4.0/ --- ![An illuminated compute core sits among disconnected service modules and exposed fiber couplers](https://cdn.markhuang.ai/news/deepinfer-missing-serving-system/hero.webp) *A conceptual view of DeepInfer's release boundary: the compute engine is available, while a complete serving system still needs integration.* DeepSeek has released [DeepInfer](https://github.com/deepseek-ai/DeepInfer), the inference framework it says powers its large-scale production services. The Apache 2.0 release provides offline inference for DeepSeek-V4.1-Flash. Its README reports benchmarks on 4 NVIDIA SM100 GPUs for prefill, which processes the input, and 64 for decode, which generates the output. Each setup stays within one scale-up domain. The README also lists the components still required for a complete service. That list is where I would start before renting hardware. I welcome being able to inspect the engine behind DeepSeek's service, but the missing infrastructure belongs in the deployment estimate. An engine throughput result cannot tell me what a complete service will cost under my traffic and latency requirements. For a team considering replacing an API bill with its own cluster, this is a release worth investigating. It also leaves enough service integration to the operator that I would want a working pilot before committing to a hardware budget. ## The engine deserves a serious look The README lists optimized kernels, draft-token decoding with confidence-scheduled verification, and asynchronous scheduling that overlaps CPU batch scheduling with device execution. It also supports structured outputs constrained by a regex, JSON schema, or EBNF grammar. A team investigating inference performance has code to examine alongside the published results. Batch-invariant inference catches my attention: DeepSeek says it keeps model results identical across different batch compositions. The documented option is NVIDIA only. When comparing outputs, I want to know whether a changed result came from the model or from the other requests running beside it. This is a worthwhile property to test on the intended deployment. The supported hardware list is narrow: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. I would check compatibility before planning around another accelerator. Equally, the 64-GPU decode measurement is a benchmark configuration, not a stated minimum hardware requirement. ## The omitted pieces affect what users feel DeepSeek names a KV cache system, vision-encoder/prefill/decode disaggregation, and load balancing across data-parallel replicas as requirements for a complete inference service that this release does not include. It adds that most engine functionality needed for disaggregation is already implemented. The project has done substantial underlying work, while the operator still has a system to finish. I read this as an integration boundary, rather than a claim that the engine has no internal cache handling. Before deployment, I would ask who owns the service-level cache and how requests reach the appropriate prefill and decode replicas. Those questions belong in the same estimate as the accelerators. The [vLLM documentation on disaggregated prefilling](https://docs.vllm.ai/en/latest/features/disagg_prefill/) explains why this work matters to a user: separating prefill and decode lets an operator tune time to first token and the delay between output tokens independently. It also describes controlling tail latency. That gives me context for the missing integration, though it does not measure DeepInfer. Both delays need measuring across the assembled service before I could promise a faster experience. ## Read the throughput accounting before pricing the output The [benchmark documentation](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md) contains a detail I would keep beside any performance chart. Its raw decode-throughput metric uses full-acceptance accounting: it counts the main token plus the configured draft tokens for each request and step, regardless of the actual acceptance rate. The project's [plotting code](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/plot_pareto.py) separately takes a draft-acceptance rate and uses it to adjust the decode curve for the fixed-draft configuration it supports. Before interpreting a plotted result, I would check that input, then measure acceptance on representative requests. Reading only the raw metric or only the chart would leave part of the experiment unexplained. My cost calculation would use accepted output that reaches the caller within the application's latency target, including queueing and the cache behavior of the complete system. DeepSeek's benchmark setup gives an operator a starting point for reproduction. Pricing my own workload still requires a separate experiment. ## I would ask the service to earn its hardware The early [Hacker News discussion](https://news.ycombinator.com/item?id=50163277) focuses on the release and inference economics. It had only three comments when I inspected it, too few to establish a public verdict on deployability. A commenter links to [SemiAnalysis's InferenceX profit estimator](https://inferencex.semianalysis.com/profit-estimator), whose controls expose cache-hit rates, a target p90 per-user token rate, token pricing, and a total-cost basis. Those inputs help explain what an economic claim depends on. The estimator does not independently reproduce DeepInfer's results. The strongest objection to my caution is fair: an open inference engine does not owe everyone a finished service. A team that already operates suitable infrastructure may find the release useful immediately. I would judge it on that scope and give DeepSeek credit for stating the omissions plainly. For a new deployment, my pilot would replay representative requests through the assembled system, record accepted output and tail latency, and include the cost of operating the missing layers. The comparison is with the API service I am considering replacing. A lower operating cost or a concrete need for deployment control could justify the work. Until that case holds, I would keep the cluster rental small enough to learn from.