The Information Machine

Rapid AI Benchmark Improvement: Small Models and New Entrants Closing Capability Gaps · history

Version 2

2026-06-26 08:28 UTC · 123 items

What

Three open-weights models released in the week of June 18–26, 2026 are advancing the argument that open-source AI is closing in on closed frontier systems. GLM-5.2 (Zhipu AI) leads open-weights models on SWE-Bench and traditional LLM benchmarks[1], scored 22.8% on ARC-AGI-2[3], and on June 26 topped PostTrainBench—an evaluation of autonomous model-training ability—at 34.29%, against 51.14% for human-built instruct pipelines[4]. VibeThinker-3B (Weibo AI) reports 94.3 on AIME26 and 80.2 on LiveCodeBench v6 from only 3 billion parameters[9]. Ornith-1.0-397B (DeepReinforce), a 397B Mixture-of-Experts model with 17B active parameters, scored 82.4 on SWE-Bench Verified under an MIT license[15]. The central debate is whether these benchmark results reflect genuine generalization or are artifacts of distillation and narrowly targeted post-training.

Why it matters

If open-weights models continue advancing on both agentic task benchmarks and real-world developer workflows, they alter the cost economics of capable AI deployment. The distillation question is the key caveat: models that learn from closed-source outputs may underperform on tasks outside typical training distributions, which is precisely where benchmark numbers are the weakest predictors of practical value.

Open questions

  • GLM-5.2's PostTrainBench score of 34.29% still trails human-built instruct models at 51.14%[4]—will agents close this gap, and what training or architectural changes would be required?

  • Is GLM-5.2's benchmark performance primarily a product of distillation from Claude Opus, and if so, how does it perform on tasks that fall outside typical benchmark distributions?[1]

  • VibeThinker-3B's AIME26 and LiveCodeBench numbers approach Claude Opus 4.5 at 3B parameters[9]—do they hold on genuinely novel reasoning problems not covered by its training data?

  • ARC-AGI-2 went from 3.0% best verified (May 2025) to GPT-5.5 at 85% (June 2026)[3]—does the benchmark still distinguish meaningfully between frontier and sub-frontier models, or is it now solved at the top?

Narrative

GLM-5.2, from Zhipu AI, released in mid-June 2026, circulated quickly as the strongest available open-weights language model, scoring near Claude Opus 4.7 on traditional benchmarks[1] and appearing on the DeepSWE software engineering leaderboard[2]. On ARC-AGI-2—a benchmark of abstract visual reasoning—it reached 22.8% at $0.25 per task[3], well below GPT-5.5's leading 85% but the highest among open-weights models. On June 26, GLM-5.2 also topped PostTrainBench at 34.29%, a benchmark that tests whether an AI agent can select training data, write training code, run fine-tuning experiments, and submit improved models within a 10-hour, single-GPU constraint—not raw question-answering, but end-to-end model improvement[4]. Human-built instruct models score 51.14% on the same benchmark, indicating a persistent capability gap above current agent-controlled pipelines[4]. Multiple users have confirmed that the DeepSWE results match their real-world experience[5][6][7], and developers report real cost savings when substituting GLM-5.2 for Claude and GPT in Cursor[8].

VibeThinker-3B, released by Weibo AI (Sina Weibo's AI subsidiary), is a dense 3-billion-parameter model built on Qwen2.5-Coder-3B using a post-training pipeline the authors call Spectrum-to-Signal, combining supervised fine-tuning and group relative policy optimization (GRPO)[9][10]. Its reported benchmarks—94.3 on AIME26, 80.2 Pass@1 on LiveCodeBench v6, 96.1% acceptance on recent LeetCode problems—approach Claude Opus 4.5, a model estimated to be orders of magnitude larger[9][11]. The model is MIT-licensed[12] and its paper appeared on arxiv alongside third-party coverage[13][14]. The central claim is that targeted post-training on verifiable reasoning tasks can embed performance normally associated with much larger models into a 3B footprint.

On June 25, DeepReinforce released Ornith-1.0, a family of MIT-licensed agentic coding language models[15]. The flagship Ornith-1.0-397B uses a Mixture-of-Experts architecture with 17 billion active parameters and scores 82.4 on SWE-Bench Verified, adding a third significant open-weights entrant to a space that moved substantially within a single week[15].

The primary analytical debate runs through Zvi Mowshowitz's evolving treatment of GLM-5.2. His initial analysis credited it as the best open-weights model while arguing it is heavily distilled from Claude Opus—noting the model's tendency to self-identify as Claude and its use of a Claude harness—and concluded it occupied a commercially awkward niche: not cheap enough for bulk tasks and not strong enough for the hardest tasks relative to closed alternatives[1]. His subsequent newsletter extended the framing, characterizing GLM-5.2 as a potential 'DeepSeek moment' for open-source AI agents, suggesting frontier agentic capabilities may now be available in open weights for the first time[16]. That stronger framing sits in tension with his own distillation argument and with the PostTrainBench data showing a 17-point gap to human-built pipelines. Community observers—including a normally benchmark-skeptical user who tested GLM-5.2 directly[17] and a developer who claims it outperformed Opus 4.8 in a live coding build-off[18]—generally found the real-world performance consistent with the benchmark rankings, adding weight to the more optimistic framing.

Timeline

  • 2025-05: Best verified models on ARC-AGI-2 scored only 3.0%. [3]
  • 2026-06-18: Weibo AI releases VibeThinker-3B, a 3B dense reasoning model using the Spectrum-to-Signal SFT+GRPO post-training pipeline. [23][24]
  • 2026-06-18: Zhipu AI publishes GLM-5.2 full benchmark results, showing strong coding and reasoning performance. [25][26]
  • 2026-06-19: VibeThinker-3B paper posted to arxiv; Marktechpost covers benchmark claims. [13][14]
  • 2026-06-21: DeepSWE benchmark updated to include GLM-5.2 alongside updated scores for other models. [2]
  • 2026-06-22: Zvi Mowshowitz publishes analysis calling GLM-5.2 the best open model but arguing it is heavily distilled from Claude Opus and commercially awkward. [1]
  • 2026-06-22: Julian Goldie SEO claims GLM-5.2 outperformed Opus 4.8 in a live coding build-off. [18]
  • 2026-06-23: A normally benchmark-skeptical observer reports finding GLM-5.2 genuinely impressive in hands-on use. [17][27]
  • 2026-06-24: GLM-5.2 measured at 22.8% on ARC-AGI-2 at $0.25 per task, versus GPT-5.5's leading 85%. [3]
  • 2026-06-24: Rohan Paul reports VibeThinker-3B scores 94.3 on AIME26 and 80.2 on LiveCodeBench v6, approaching Opus 4.5 on reasoning. [9]
  • 2026-06-25: DeepReinforce releases Ornith-1.0-397B, a 397B MoE agentic coding model (17B active) scoring 82.4 on SWE-Bench Verified under an MIT license. [15]
  • 2026-06-25: Zvi Mowshowitz frames GLM-5.2 as a potential 'DeepSeek moment' for open-source AI agents. [16]
  • 2026-06-25: Developers report real cost savings using GLM-5.2 in Cursor compared to Claude and GPT. [8]
  • 2026-06-26: GLM-5.2 tops PostTrainBench at 34.29%; human-built instruct models score 51.14% on the same benchmark. [4]

Perspectives

Zvi Mowshowitz

GLM-5.2 is the best available open-weights model and may represent a 'DeepSeek moment' for open-source agents, but is very likely heavily distilled from Claude Opus and occupies a commercially awkward niche below closed frontier models.

Evolution: Stance has strengthened: initial analysis was skeptical of commercial value; subsequent framing elevated the 'DeepSeek moment' characterization, which sits in tension with the distillation argument.

Rohan Paul

Reports benchmark data for GLM-5.2, VibeThinker-3B, and Ornith-1.0 without strong editorial endorsement or dismissal; consistently neutral-analytical.

Evolution: Consistent across all releases covered.

Zixuan Li (Zhipu AI / GLM team)

GLM-5.2 delivers a meaningful improvement in app development and long-horizon task capabilities.

Evolution: Promotional stance consistent with lab affiliation.

bendee983

Normally skeptical of AI benchmarks but finds GLM-5.2 genuinely impressive in direct use, not just on paper.

Evolution: A skeptic who updated positively after hands-on testing.

Julian Goldie SEO

GLM-5.2 outperformed Opus 4.8 in a live build-off, though he separately argues top-benchmark models often come last in real-world tests.

Evolution: Enthusiastic on GLM-5.2 in head-to-head comparison while skeptical of benchmark rankings as a general guide.

Thyago Liberalli

Evaluating models on a single benchmark is a significant error; performance must be assessed across multiple dimensions.

Evolution: Consistent skepticism of single-metric evaluation.

Weibo AI / WeiboLLM

VibeThinker-3B demonstrates that a 3B model can achieve frontier-adjacent verifiable reasoning through the Spectrum-to-Signal post-training pipeline.

Evolution: Initial release; no prior stance to compare.

DeepReinforce

Ornith-1.0-397B, an MIT-licensed MoE agentic coding model, achieves 82.4 on SWE-Bench Verified using only 17B active parameters.

Evolution: New entrant; no prior stance to compare.

Tensions

  • Zvi Mowshowitz argues GLM-5.2 is heavily distilled from Claude Opus and likely generalizes poorly outside benchmark-like tasks; bendee983 and Julian Goldie find its performance genuinely strong in direct use. [1][17][18]
  • Mowshowitz called GLM-5.2 commercially awkward—not cheap enough for bulk tasks and not strong enough for hard tasks—but developers report real cost savings using it in Cursor over Claude and GPT. [1][8]
  • Mowshowitz's own framing is internally split: he characterizes GLM-5.2 as a potential 'DeepSeek moment' for open-source agents while simultaneously arguing its benchmarks overstate generalization due to distillation. [1][16]
  • VibeThinker-3B's benchmark numbers approach Claude Opus 4.5 on math and coding at 3B parameters, but whether a model of that size can match a vastly larger model on tasks outside its training distribution remains untested. [9][11]
  • ARC-AGI-2's rapid progression from 3.0% (May 2025) to GPT-5.5 at 85% (June 2026) invites the question of whether it still discriminates meaningfully at the top, while GLM-5.2's 22.8% shows a persistent large gap to the current leader. [3]

Sources

  1. [1] GLM-5.2 Is The New Best Open Model — Zvi's AI Roundups (2026-06-22)
  2. [2] DeepSWE Benchmark updated with GLM 5.2 and updated results for other models — reactive:ai-benchmark-race (2026-06-21)
  3. [3] GLM-5.2 got 22.8% on ARC-AGI-2:, $0.25/task — Rohan Paul Twitter (2026-06-24)
  4. [4] GLM 5.2 just took the top spot on PostTrainBench by scoring 34.29%. — Rohan Paul Twitter (2026-06-26)
  5. [5] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-25)
  6. [6] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-25)
  7. [7] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-25)
  8. [8] Developers are reporting real cost savings using GLM-5.2 in Cursor compared to Claude and GPT. — reactive:ai-benchmark-race (2026-06-25)
  9. [9] VibeThinker is a 3B param model, with almost head to head benchmark result with Opus 4.5 on reasoning with novel SFT+GRP… — Rohan Paul Twitter (2026-06-24)
  10. [10] VibeThinker-3B: A 3B Dense Reasoning Model Built on Qwen2.5-Coder-3B With the Spectrum-to-Signal Post-Training Pipeline ... — reactive:ai-benchmark-race (2026-06-21)
  11. [11] 1/ A 3 billion parameter model just beat Opus 4.5 (1T+ params) on math reasoning. — reactive:ai-benchmark-race (2026-06-23)
  12. [12] RT @Marktechpost: 🔥 VibeThinker-3B is a 3B open-source (MIT) reasoning model that reaches the band of systems hundreds o... — reactive:ai-benchmark-race (2026-06-19)
  13. [13] [2606.16140] VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models — reactive:ai-benchmark-race
  14. [14] VibeThinker-3B: A 3B Dense Reasoning Model Built on ... — reactive:ai-benchmark-race
  15. [15] Another fantastic open source release. — Rohan Paul Twitter (2026-06-25)
  16. [16] AI #174: You're It — Zvi's AI Roundups (2026-06-25)
  17. [17] RT @bendee983: I'm usually very skeptical of AI benchmarks, but GLM-5.2 is really impressive. Aside from the benchmark r... — reactive:ai-benchmark-race (2026-06-23)
  18. [18] GLM 5.2 JUST HUMILIATED OPUS 4.8 IN A LIVE BUILD-OFF — reactive:ai-benchmark-race (2026-06-19)
  19. [19] GLM-5.2 delivers a substantial leap in app development capabilities, which also represent demanding long-horizon tasks. — reactive:ai-benchmark-race (2026-06-19)
  20. [20] RT @bendee983: I'm usually very skeptical of AI benchmarks, but GLM-5.2 is really impressive. Aside from the benchmark r... — reactive:ai-benchmark-race (2026-06-23)
  21. [21] THE “BEST” AI MODEL CAME LAST IN MY REAL-WORLD TEST. — reactive:ai-benchmark-race (2026-06-24)
  22. [22] One of the biggest mistakes people make when evaluating LLMs is looking at a single benchmark and assuming it tells the ... — reactive:ai-benchmark-race (2026-06-23)
  23. [23] RT @WeiboLLM: ⭐ VibeThinker-3B is released — a dense 3B model for frontier-level verifiable reasoning. — reactive:ai-benchmark-race (2026-06-18)
  24. [24] RT @ModelScope2022: Meet VibeThinker-3B, a 3B reasoning model from Weibo AI focused on math, coding, and STEM reasoning.... — reactive:ai-benchmark-race (2026-06-18)
  25. [25] RT @ValsAI: Full results for GLM 5.2 are here! — reactive:ai-benchmark-race (2026-06-18)
  26. [26] RT @ValsAI: Full results for GLM 5.2 are here! — reactive:ai-benchmark-race (2026-06-18)
  27. [27] @AndrewCurran_ What GLM 5.2 did to the best of the best is as big of a breakthrough as the R1 improvement purely from th... — reactive:ai-benchmark-race (2026-06-23)