Ultra-Low Latency LLM Inference: Benchmarks and Emerging Enterprise Pricing Tier

Kog AI claims approximately 3,000 tokens per second per request on AMD MI300X using a monokernel architecture [4], validated by AMD [6] but not independently replicated. SemiAnalysis now publishes longitudinal inference benchmarks via InferenceX for DeepSeek V4, showing AMD ROCm improved 100x by Day 26 while NVIDIA's TensorRT-LLM had launch issues requiring community fixes [9]. The inference optimization landscape continues to expand: Makora AI's sequential Monte Carlo speculative decoding eliminates rewind costs in draft token verification [13], selective KV cache retention reduces memory overhead in long-context generation [14], and Qualcomm's CEO projects 40x global token demand growth by 2030 driven by agentic AI [12]. The central market question — whether ultra-low per-request latency inference commands a 20–50x per-token price premium [11] — remains unresolved.

If agentic AI drives the next wave of token demand at machine pace, as Qualcomm projects [12], the dominant infrastructure need may be aggregate throughput at lower cost rather than per-request speed — which would partially undercut the enterprise pricing case SemiAnalysis has framed [11]. SemiAnalysis's push for longitudinal over snapshot benchmarking [9] adds a methodological lens under which one-time claims like Kog AI's face additional scrutiny.

SemiAnalysis established the structural baseline: roughly 48% of end-to-end LLM inference latency comes from prefill and 52% from decode [1]. The DistServe paper at OSDI 2024 proposed disaggregating these two phases onto separate resources to optimize both simultaneously [2], and subsequent research on prefill-decode multiplexing continues this direction [3]. Into this landscape, Kog AI claims a monokernel architecture — a single kernel purpose-built for LLM inference — delivering approximately 3,000 tokens per second per request on an 8-GPU AMD MI300X node and 2,100 tokens/s on comparable NVIDIA H200, using a 2B parameter model at FP16 precision without speculative decoding [4][5]. AMD published a blog validating the result as a 3.5x improvement over its inference baseline [6], and discussion spread from initial amplifiers to Hacker News [7]. All figures originate from Kog AI itself; no neutral third-party replication has been documented. A metric distinction matters practically: Kog AI's 3,000 tokens/s is per-request throughput — the speed for a single user's request — while Moreh separately reports 21,000 tokens/s on the same MI300X hardware measuring aggregate output across concurrent requests using expert parallelism on a DeepSeek model [8]. These optimize for different objectives and are not directly comparable.

SemiAnalysis, via its InferenceX initiative, is now tracking DeepSeek V4 inference performance longitudinally across hardware platforms — Huawei Ascend, NVIDIA GB300 NVL72 and B200, AMD MI355X — framing this as a transparency effort against one-off snapshot benchmarks. Their data shows AMD ROCm achieved a 100x performance improvement for DeepSeek V4 by Day 26 under focused engineering effort, while NVIDIA's own TensorRT-LLM did not work well at launch and required community-contributed fixes to its open-source kernel launch code [9]. DeepSeek V4 was co-designed in part for Huawei Ascend, which demonstrated Day 0 inference support [9]. Intel announced plans to debut a new AI data center chip before end of 2026 featuring lower-cost memory and cooling technology compared to current AMD and NVIDIA offerings [10], expanding hardware competition beyond a two-player race.

SemiAnalysis frames ultra-low latency inference as an emerging enterprise pricing experiment: roughly 10x speed for a 20–50x per-token cost premium over standard inference [11], with enterprise willingness-to-pay explicitly unresolved. The demand-side context comes from Qualcomm's CEO projecting global AI token demand will reach 1.27 trillion tokens every 10 seconds by 2030, up from 31.7 billion today, driven primarily by agentic AI operating at machine pace rather than conversational human pace [12] — a projection that favors aggregate throughput infrastructure over per-request speed optimization. Algorithmic efficiency alternatives continue to emerge alongside the hardware competition: Makora AI's sequential Monte Carlo speculative decoding keeps multiple draft tokens alive in parallel rather than rewinding on mismatch, eliminating that rewind cost [13]; selective KV cache token retention reduces memory overhead in long-context generation by predicting which past tokens will remain relevant [14]; and context compression cutting token use by 25–50% plus multi-model routing both offer orthogonal cost reduction levers [15][16].