Kimi K3, and what we can still learn from the pelican benchmark
Simon Willison · Simon Willison · 2026-07-16
Moonshot AI releases Kimi K3, a 2.8 trillion parameter model priced at $3/$15 per million tokens, while Simon Willison uses the release to honestly assess why his 21-month-old pelican SVG benchmark no longer reliably ranks frontier models.
Extraction
Topics: llm-releasellm-benchmarksai-in-chinallm-pricingmodel-evaluation
Claims
- Kimi K3 has 2.8 trillion parameters, surpassing DeepSeek v4 Pro as the largest publicly announced model by a Chinese lab.
- K3 is priced at $3/million input and $15/million output tokens, the most expensive model released by a Chinese AI lab.
- K3 currently offers only one reasoning effort level ('max'), making it expensive even for simple prompts—a pelican SVG cost 25 cents.
- The pelican SVG benchmark no longer reliably correlates with frontier model quality, with smaller models now outperforming leading ones on the task.
- K3 appears to include a hidden system prompt of roughly 85 tokens based on token count anomalies.
- The benchmark's remaining value lies in confirming API access, estimating per-task cost, and verifying basic spatial reasoning in smaller models.
Key quotes
The biggest limitation of the pelican is that it doesn't touch at all on the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
That connection has been mostly severed now. The GPT-5.6 and Claude Fable 5 pelicans are outclassed by GLM-5.2, and much as I love GLM I don't think that's a Fable-class model.
Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers.