Rapid AI Benchmark Improvement: Small Models and New Entrants Closing Capability Gaps · history
Version 2
2026-06-26 08:28 UTC · 123 items
What
Three open-weights models released in the week of June 18–26, 2026 are advancing the argument that open-source AI is closing in on closed frontier systems. GLM-5.2 (Zhipu AI) leads open-weights models on SWE-Bench and traditional LLM benchmarks[1], scored 22.8% on ARC-AGI-2[3], and on June 26 topped PostTrainBench—an evaluation of autonomous model-training ability—at 34.29%, against 51.14% for human-built instruct pipelines[4]. VibeThinker-3B (Weibo AI) reports 94.3 on AIME26 and 80.2 on LiveCodeBench v6 from only 3 billion parameters[9]. Ornith-1.0-397B (DeepReinforce), a 397B Mixture-of-Experts model with 17B active parameters, scored 82.4 on SWE-Bench Verified under an MIT license[15]. The central debate is whether these benchmark results reflect genuine generalization or are artifacts of distillation and narrowly targeted post-training.
Why it matters
If open-weights models continue advancing on both agentic task benchmarks and real-world developer workflows, they alter the cost economics of capable AI deployment. The distillation question is the key caveat: models that learn from closed-source outputs may underperform on tasks outside typical training distributions, which is precisely where benchmark numbers are the weakest predictors of practical value.
Open questions
GLM-5.2's PostTrainBench score of 34.29% still trails human-built instruct models at 51.14%[4]—will agents close this gap, and what training or architectural changes would be required?
Is GLM-5.2's benchmark performance primarily a product of distillation from Claude Opus, and if so, how does it perform on tasks that fall outside typical benchmark distributions?[1]
VibeThinker-3B's AIME26 and LiveCodeBench numbers approach Claude Opus 4.5 at 3B parameters[9]—do they hold on genuinely novel reasoning problems not covered by its training data?
ARC-AGI-2 went from 3.0% best verified (May 2025) to GPT-5.5 at 85% (June 2026)[3]—does the benchmark still distinguish meaningfully between frontier and sub-frontier models, or is it now solved at the top?
Narrative
GLM-5.2, from Zhipu AI, released in mid-June 2026, circulated quickly as the strongest available open-weights language model, scoring near Claude Opus 4.7 on traditional benchmarks[1] and appearing on the DeepSWE software engineering leaderboard[2]. On ARC-AGI-2—a benchmark of abstract visual reasoning—it reached 22.8% at $0.25 per task[3], well below GPT-5.5's leading 85% but the highest among open-weights models. On June 26, GLM-5.2 also topped PostTrainBench at 34.29%, a benchmark that tests whether an AI agent can select training data, write training code, run fine-tuning experiments, and submit improved models within a 10-hour, single-GPU constraint—not raw question-answering, but end-to-end model improvement[4]. Human-built instruct models score 51.14% on the same benchmark, indicating a persistent capability gap above current agent-controlled pipelines[4]. Multiple users have confirmed that the DeepSWE results match their real-world experience[5][6][7], and developers report real cost savings when substituting GLM-5.2 for Claude and GPT in Cursor[8].
VibeThinker-3B, released by Weibo AI (Sina Weibo's AI subsidiary), is a dense 3-billion-parameter model built on Qwen2.5-Coder-3B using a post-training pipeline the authors call Spectrum-to-Signal, combining supervised fine-tuning and group relative policy optimization (GRPO)[9][10]. Its reported benchmarks—94.3 on AIME26, 80.2 Pass@1 on LiveCodeBench v6, 96.1% acceptance on recent LeetCode problems—approach Claude Opus 4.5, a model estimated to be orders of magnitude larger[9][11]. The model is MIT-licensed[12] and its paper appeared on arxiv alongside third-party coverage[13][14]. The central claim is that targeted post-training on verifiable reasoning tasks can embed performance normally associated with much larger models into a 3B footprint.
On June 25, DeepReinforce released Ornith-1.0, a family of MIT-licensed agentic coding language models[15]. The flagship Ornith-1.0-397B uses a Mixture-of-Experts architecture with 17 billion active parameters and scores 82.4 on SWE-Bench Verified, adding a third significant open-weights entrant to a space that moved substantially within a single week[15].
The primary analytical debate runs through Zvi Mowshowitz's evolving treatment of GLM-5.2. His initial analysis credited it as the best open-weights model while arguing it is heavily distilled from Claude Opus—noting the model's tendency to self-identify as Claude and its use of a Claude harness—and concluded it occupied a commercially awkward niche: not cheap enough for bulk tasks and not strong enough for the hardest tasks relative to closed alternatives[1]. His subsequent newsletter extended the framing, characterizing GLM-5.2 as a potential 'DeepSeek moment' for open-source AI agents, suggesting frontier agentic capabilities may now be available in open weights for the first time[16]. That stronger framing sits in tension with his own distillation argument and with the PostTrainBench data showing a 17-point gap to human-built pipelines. Community observers—including a normally benchmark-skeptical user who tested GLM-5.2 directly[17] and a developer who claims it outperformed Opus 4.8 in a live coding build-off[18]—generally found the real-world performance consistent with the benchmark rankings, adding weight to the more optimistic framing.
Timeline
- 2025-05: Best verified models on ARC-AGI-2 scored only 3.0%. [3]
- 2026-06-18: Weibo AI releases VibeThinker-3B, a 3B dense reasoning model using the Spectrum-to-Signal SFT+GRPO post-training pipeline. [23][24]
- 2026-06-18: Zhipu AI publishes GLM-5.2 full benchmark results, showing strong coding and reasoning performance. [25][26]
- 2026-06-19: VibeThinker-3B paper posted to arxiv; Marktechpost covers benchmark claims. [13][14]
- 2026-06-21: DeepSWE benchmark updated to include GLM-5.2 alongside updated scores for other models. [2]
- 2026-06-22: Zvi Mowshowitz publishes analysis calling GLM-5.2 the best open model but arguing it is heavily distilled from Claude Opus and commercially awkward. [1]
- 2026-06-22: Julian Goldie SEO claims GLM-5.2 outperformed Opus 4.8 in a live coding build-off. [18]
- 2026-06-23: A normally benchmark-skeptical observer reports finding GLM-5.2 genuinely impressive in hands-on use. [17][27]
- 2026-06-24: GLM-5.2 measured at 22.8% on ARC-AGI-2 at $0.25 per task, versus GPT-5.5's leading 85%. [3]
- 2026-06-24: Rohan Paul reports VibeThinker-3B scores 94.3 on AIME26 and 80.2 on LiveCodeBench v6, approaching Opus 4.5 on reasoning. [9]
- 2026-06-25: DeepReinforce releases Ornith-1.0-397B, a 397B MoE agentic coding model (17B active) scoring 82.4 on SWE-Bench Verified under an MIT license. [15]
- 2026-06-25: Zvi Mowshowitz frames GLM-5.2 as a potential 'DeepSeek moment' for open-source AI agents. [16]
- 2026-06-25: Developers report real cost savings using GLM-5.2 in Cursor compared to Claude and GPT. [8]
- 2026-06-26: GLM-5.2 tops PostTrainBench at 34.29%; human-built instruct models score 51.14% on the same benchmark. [4]
Perspectives
Zvi Mowshowitz
GLM-5.2 is the best available open-weights model and may represent a 'DeepSeek moment' for open-source agents, but is very likely heavily distilled from Claude Opus and occupies a commercially awkward niche below closed frontier models.
Evolution: Stance has strengthened: initial analysis was skeptical of commercial value; subsequent framing elevated the 'DeepSeek moment' characterization, which sits in tension with the distillation argument.
Rohan Paul
Reports benchmark data for GLM-5.2, VibeThinker-3B, and Ornith-1.0 without strong editorial endorsement or dismissal; consistently neutral-analytical.
Evolution: Consistent across all releases covered.
Zixuan Li (Zhipu AI / GLM team)
GLM-5.2 delivers a meaningful improvement in app development and long-horizon task capabilities.
Evolution: Promotional stance consistent with lab affiliation.
bendee983
Normally skeptical of AI benchmarks but finds GLM-5.2 genuinely impressive in direct use, not just on paper.
Evolution: A skeptic who updated positively after hands-on testing.
Julian Goldie SEO
GLM-5.2 outperformed Opus 4.8 in a live build-off, though he separately argues top-benchmark models often come last in real-world tests.
Evolution: Enthusiastic on GLM-5.2 in head-to-head comparison while skeptical of benchmark rankings as a general guide.
Thyago Liberalli
Evaluating models on a single benchmark is a significant error; performance must be assessed across multiple dimensions.
Evolution: Consistent skepticism of single-metric evaluation.
Weibo AI / WeiboLLM
VibeThinker-3B demonstrates that a 3B model can achieve frontier-adjacent verifiable reasoning through the Spectrum-to-Signal post-training pipeline.
Evolution: Initial release; no prior stance to compare.
DeepReinforce
Ornith-1.0-397B, an MIT-licensed MoE agentic coding model, achieves 82.4 on SWE-Bench Verified using only 17B active parameters.
Evolution: New entrant; no prior stance to compare.
Tensions
- Zvi Mowshowitz argues GLM-5.2 is heavily distilled from Claude Opus and likely generalizes poorly outside benchmark-like tasks; bendee983 and Julian Goldie find its performance genuinely strong in direct use. [1][17][18]
- Mowshowitz called GLM-5.2 commercially awkward—not cheap enough for bulk tasks and not strong enough for hard tasks—but developers report real cost savings using it in Cursor over Claude and GPT. [1][8]
- Mowshowitz's own framing is internally split: he characterizes GLM-5.2 as a potential 'DeepSeek moment' for open-source agents while simultaneously arguing its benchmarks overstate generalization due to distillation. [1][16]
- VibeThinker-3B's benchmark numbers approach Claude Opus 4.5 on math and coding at 3B parameters, but whether a model of that size can match a vastly larger model on tasks outside its training distribution remains untested. [9][11]
- ARC-AGI-2's rapid progression from 3.0% (May 2025) to GPT-5.5 at 85% (June 2026) invites the question of whether it still discriminates meaningfully at the top, while GLM-5.2's 22.8% shows a persistent large gap to the current leader. [3]
Sources
- [1] GLM-5.2 Is The New Best Open Model — Zvi's AI Roundups (2026-06-22)
- [2] DeepSWE Benchmark updated with GLM 5.2 and updated results for other models — reactive:ai-benchmark-race (2026-06-21)
- [3] GLM-5.2 got 22.8% on ARC-AGI-2:, $0.25/task — Rohan Paul Twitter (2026-06-24)
- [4] GLM 5.2 just took the top spot on PostTrainBench by scoring 34.29%. — Rohan Paul Twitter (2026-06-26)
- [5] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-25)
- [6] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-25)
- [7] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-25)
- [8] Developers are reporting real cost savings using GLM-5.2 in Cursor compared to Claude and GPT. — reactive:ai-benchmark-race (2026-06-25)
- [9] VibeThinker is a 3B param model, with almost head to head benchmark result with Opus 4.5 on reasoning with novel SFT+GRP… — Rohan Paul Twitter (2026-06-24)
- [10] VibeThinker-3B: A 3B Dense Reasoning Model Built on Qwen2.5-Coder-3B With the Spectrum-to-Signal Post-Training Pipeline ... — reactive:ai-benchmark-race (2026-06-21)
- [11] 1/ A 3 billion parameter model just beat Opus 4.5 (1T+ params) on math reasoning. — reactive:ai-benchmark-race (2026-06-23)
- [12] RT @Marktechpost: 🔥 VibeThinker-3B is a 3B open-source (MIT) reasoning model that reaches the band of systems hundreds o... — reactive:ai-benchmark-race (2026-06-19)
- [13] [2606.16140] VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models — reactive:ai-benchmark-race
- [14] VibeThinker-3B: A 3B Dense Reasoning Model Built on ... — reactive:ai-benchmark-race
- [15] Another fantastic open source release. — Rohan Paul Twitter (2026-06-25)
- [16] AI #174: You're It — Zvi's AI Roundups (2026-06-25)
- [17] RT @bendee983: I'm usually very skeptical of AI benchmarks, but GLM-5.2 is really impressive. Aside from the benchmark r... — reactive:ai-benchmark-race (2026-06-23)
- [18] GLM 5.2 JUST HUMILIATED OPUS 4.8 IN A LIVE BUILD-OFF — reactive:ai-benchmark-race (2026-06-19)
- [19] GLM-5.2 delivers a substantial leap in app development capabilities, which also represent demanding long-horizon tasks. — reactive:ai-benchmark-race (2026-06-19)
- [20] RT @bendee983: I'm usually very skeptical of AI benchmarks, but GLM-5.2 is really impressive. Aside from the benchmark r... — reactive:ai-benchmark-race (2026-06-23)
- [21] THE “BEST” AI MODEL CAME LAST IN MY REAL-WORLD TEST. — reactive:ai-benchmark-race (2026-06-24)
- [22] One of the biggest mistakes people make when evaluating LLMs is looking at a single benchmark and assuming it tells the ... — reactive:ai-benchmark-race (2026-06-23)
- [23] RT @WeiboLLM: ⭐ VibeThinker-3B is released — a dense 3B model for frontier-level verifiable reasoning. — reactive:ai-benchmark-race (2026-06-18)
- [24] RT @ModelScope2022: Meet VibeThinker-3B, a 3B reasoning model from Weibo AI focused on math, coding, and STEM reasoning.... — reactive:ai-benchmark-race (2026-06-18)
- [25] RT @ValsAI: Full results for GLM 5.2 are here! — reactive:ai-benchmark-race (2026-06-18)
- [26] RT @ValsAI: Full results for GLM 5.2 are here! — reactive:ai-benchmark-race (2026-06-18)
- [27] @AndrewCurran_ What GLM 5.2 did to the best of the best is as big of a breakthrough as the R1 improvement purely from th... — reactive:ai-benchmark-race (2026-06-23)