The Information Machine

Rapid AI Benchmark Improvement: Small Models and New Entrants Closing Capability Gaps · history

Version 4

2026-06-29 02:32 UTC · 142 items

What

Three open-weights models released in the week of June 18–26, 2026 have narrowed capability gaps with closed frontier systems on key benchmarks. GLM-5.2 (Zhipu AI) leads open-weights models on SWE-Bench, scored 22.8% on ARC-AGI-2 — officially verified by the ARC-AGI evaluation team[10] — and topped PostTrainBench at 34.29%[16]. VibeThinker-3B (Weibo AI) reports math and coding scores approaching Claude Opus 4.5 from only 3 billion parameters[3]. Ornith-1.0-397B (DeepReinforce) scores 82.4 on SWE-Bench Verified under MIT license[4]. A widely-shared analysis uses GLM-5.2's verified ARC-AGI score as a calibration anchor to estimate closed model capabilities from 8 private benchmark datapoints[12][13].

Why it matters

Open-weights models reliable enough to serve as calibration references for closed-model capability estimation offer independent capability auditing, not just cheaper API alternatives. The distillation question remains the key caveat: models trained on closed-source outputs may produce misleading calibration signals precisely on tasks where benchmark numbers overstate generalization.

Open questions

  • Can GLM-5.2's ARC-AGI score reliably anchor estimates of private model capabilities when combined with WeirdML results, or does distillation introduce systematic biases that make it a poor calibration reference?[12][13]

  • GLM-5.2 tops PostTrainBench at 34.29%, trailing human-built instruct models at 51.14%[16] — what training or architectural changes would be required to close that gap?

  • VibeThinker-3B's AIME26 and LiveCodeBench numbers approach Claude Opus 4.5 at 3B parameters[3] — do they hold on tasks outside the model's training distribution?

  • ARC-AGI-2 went from 3.0% best (May 2025) to 85% for GPT-5.5 (June 2026)[11] — does GLM-5.2's verified 22.8% show the benchmark still discriminates meaningfully between open and closed systems, or is the gap an artifact of scale?

Narrative

Three open-weights models released in the week of June 18–26, 2026 have advanced the argument that freely available AI can approach closed frontier performance on key benchmarks. GLM-5.2 from Zhipu AI scores near Claude Opus 4.7 on traditional LLM benchmarks[1] and leads open-weights models on the DeepSWE software engineering leaderboard[2]. VibeThinker-3B, released by Weibo AI (Sina Weibo's AI subsidiary), is a 3-billion-parameter dense model built on Qwen2.5-Coder-3B using a post-training pipeline called Spectrum-to-Signal combining supervised fine-tuning and GRPO; its reported scores — 94.3 on AIME26 and 80.2 Pass@1 on LiveCodeBench v6 — approach Claude Opus 4.5, a model estimated to be orders of magnitude larger, and the model is MIT-licensed[3]. Ornith-1.0-397B, released June 25 by DeepReinforce, is a Mixture-of-Experts model with 397 billion total parameters but only 17 billion active, scoring 82.4 on SWE-Bench Verified under MIT license[4].

The primary analytical debate surrounding GLM-5.2 comes from Zvi Mowshowitz, who has offered two framings in tension. His initial analysis called GLM-5.2 the best available open-weights model but argued it was heavily distilled from Claude Opus — noting its tendency to self-identify as Claude and its reliance on a Claude-based harness — and concluded it occupied a commercially awkward position: not cheap enough for bulk tasks, not capable enough for the hardest ones[1]. His subsequent newsletter elevated this to a potential 'DeepSeek moment' for open-source agents, suggesting frontier agentic capabilities may now be accessible in open weights for the first time[5]. Community testing has generally supported the optimistic view: a normally benchmark-skeptical observer found it genuinely impressive in direct use[6], a developer claimed it outperformed Opus 4.8 in a live coding build-off[7], multiple developers report real cost savings using GLM-5.2 in Cursor[8], and at least one developer confirms that DeepSWE benchmark results match their own hands-on experience[9].

On abstract visual reasoning, GLM-5.2 scored 22.8% on ARC-AGI-2 at $0.25 per task, a result that received official verification from the ARC-AGI evaluation team[10][11]. This places it well below GPT-5.5's leading 85% but as the highest-scoring open-weights model publicly reported on ARC-AGI-2. The benchmark's rapid progression — from a 3.0% best score in May 2025 to 85% by June 2026 — has prompted analysts to ask whether it still discriminates at the top, though the 62-point gap between GLM-5.2 and GPT-5.5 suggests meaningful separation persists between open and closed systems on this test. A widely-shared analysis combining GLM-5.2's officially verified ARC-AGI score with WeirdML benchmark results assembled 8 private benchmark datapoints, using GLM-5.2 as a calibration reference to triangulate estimates of closed model capabilities[12][13][14][15].

On June 26, GLM-5.2 also topped PostTrainBench — which tests whether an AI agent can autonomously select training data, write training code, run fine-tuning experiments, and submit an improved model within a 10-hour single-GPU constraint — at 34.29%, against a 51.14% human-built instruct model baseline[16]. Zhipu AI's official Hugging Face blog frames GLM-5.2 as built specifically for long-horizon tasks[17]. The cumulative picture is that open-weights models have made measurable advances on agentic coding, mathematical reasoning, and abstract visual tasks, but gaps to leading closed models remain large on the most demanding benchmarks, and the distillation question means rankings may overstate out-of-distribution generalization for all three models.

Timeline

  • 2025-05: Best verified models on ARC-AGI-2 scored only 3.0%. [11]
  • 2026-06-18: Weibo AI releases VibeThinker-3B using the Spectrum-to-Signal SFT+GRPO post-training pipeline. [20][25]
  • 2026-06-18: Zhipu AI publishes GLM-5.2 full benchmark results showing strong coding and reasoning performance. [26][27]
  • 2026-06-21: DeepSWE benchmark updated to include GLM-5.2. [2]
  • 2026-06-22: Zvi Mowshowitz calls GLM-5.2 the best open model but argues it is heavily distilled from Claude Opus and commercially awkward. [1]
  • 2026-06-22: Julian Goldie SEO claims GLM-5.2 outperformed Opus 4.8 in a live coding build-off. [7]
  • 2026-06-23: A normally benchmark-skeptical observer reports finding GLM-5.2 genuinely impressive in hands-on use. [6][28]
  • 2026-06-24: GLM-5.2 measured at 22.8% on ARC-AGI-2 at $0.25 per task; GPT-5.5 leads at 85%. [11]
  • 2026-06-24: Rohan Paul reports VibeThinker-3B scores 94.3 on AIME26 and 80.2 on LiveCodeBench v6, approaching Opus 4.5. [3]
  • 2026-06-25: DeepReinforce releases Ornith-1.0-397B, a 397B MoE agentic coding model (17B active) scoring 82.4 on SWE-Bench Verified under MIT license. [4]
  • 2026-06-25: Zvi Mowshowitz frames GLM-5.2 as a potential 'DeepSeek moment' for open-source AI agents. [5]
  • 2026-06-25: Developers report real cost savings using GLM-5.2 in Cursor compared to Claude and GPT. [8]
  • 2026-06-26: GLM-5.2 tops PostTrainBench at 34.29%; human-built instruct models score 51.14% on the same benchmark. [16]
  • 2026-06-26: ARC-AGI evaluation team officially verifies GLM-5.2's 22.8% score on ARC-AGI-2. [10]
  • 2026-06-26: Analysis combining GLM-5.2 ARC-AGI results with WeirdML data assembles 8 private benchmark datapoints as a calibration framework for estimating closed model capabilities. [12][13][14][15]
  • 2026-06-26: A developer confirms DeepSWE benchmark results for GLM-5.2 match their own hands-on coding experience. [9]

Perspectives

Zvi Mowshowitz

GLM-5.2 is the best available open-weights model and may represent a 'DeepSeek moment' for open-source agents, but is very likely heavily distilled from Claude Opus and occupies a commercially awkward niche.

Evolution: Stance has strengthened toward the 'DeepSeek moment' framing while retaining the distillation skepticism, leaving the two positions in tension.

Rohan Paul

Reports benchmark data for GLM-5.2, VibeThinker-3B, Ornith-1.0, and PostTrainBench without strong editorial endorsement or dismissal.

Evolution: Consistent neutral-analytical stance across all covered releases.

bendee983

Normally skeptical of AI benchmarks but finds GLM-5.2 genuinely impressive in direct use, not just on paper.

Evolution: A skeptic who updated positively after hands-on testing.

Julian Goldie SEO

GLM-5.2 outperformed Opus 4.8 in a live build-off, though top-benchmark models often come last in real-world tests.

Evolution: Enthusiastic on GLM-5.2 in head-to-head comparison while skeptical of benchmark rankings as a general guide.

Weibo AI / WeiboLLM

VibeThinker-3B demonstrates that a 3B model can achieve frontier-adjacent verifiable reasoning through the Spectrum-to-Signal post-training pipeline.

Evolution: Initial release; no prior stance to compare.

DeepReinforce

Ornith-1.0-397B achieves 82.4 on SWE-Bench Verified using only 17B active parameters under MIT license.

Evolution: New entrant; no prior stance to compare.

Zhipu AI / GLM team

GLM-5.2 delivers meaningful improvements in long-horizon task and app development capabilities.

Evolution: Consistent promotional stance; official Hugging Face blog reinforces 'long-horizon tasks' positioning.

Thyago Liberalli

Evaluating models on a single benchmark is a significant error; performance must be assessed across multiple dimensions.

Evolution: Consistent skepticism of single-metric evaluation.

Tensions

  • Mowshowitz argues GLM-5.2 is heavily distilled from Claude Opus and likely generalizes poorly outside benchmark-like tasks; bendee983, Julian Goldie, and a developer reporting DeepSWE experience all find its performance genuinely strong in direct use. [1][6][7][9]
  • Mowshowitz called GLM-5.2 commercially awkward — not cheap enough for bulk tasks and not strong enough for hard tasks — but developers report real cost savings using it in Cursor over Claude and GPT. [1][8]
  • Mowshowitz's own framing is internally split: he characterizes GLM-5.2 as a potential 'DeepSeek moment' for open-source agents while simultaneously arguing its benchmarks overstate generalization due to distillation. [1][5]
  • VibeThinker-3B's benchmark numbers approach Claude Opus 4.5 on math and coding at 3B parameters, but whether a model of that size can match a vastly larger model on tasks outside its training distribution remains untested. [3][24]
  • GLM-5.2 is being used as a calibration anchor to estimate closed model capabilities from 8 private benchmark datapoints, but if its scores are distillation artifacts, those calibration estimates may be systematically off. [12][13][1]

Sources

  1. [1] GLM-5.2 Is The New Best Open Model — Zvi's AI Roundups (2026-06-22)
  2. [2] DeepSWE Benchmark updated with GLM 5.2 and updated results for other models — reactive:ai-benchmark-race (2026-06-21)
  3. [3] VibeThinker is a 3B param model, with almost head to head benchmark result with Opus 4.5 on reasoning with novel SFT+GRP… — Rohan Paul Twitter (2026-06-24)
  4. [4] Another fantastic open source release. — Rohan Paul Twitter (2026-06-25)
  5. [5] AI #174: You're It — Zvi's AI Roundups (2026-06-25)
  6. [6] RT @bendee983: I'm usually very skeptical of AI benchmarks, but GLM-5.2 is really impressive. Aside from the benchmark r... — reactive:ai-benchmark-race (2026-06-23)
  7. [7] GLM 5.2 JUST HUMILIATED OPUS 4.8 IN A LIVE BUILD-OFF — reactive:ai-benchmark-race (2026-06-19)
  8. [8] Developers are reporting real cost savings using GLM-5.2 in Cursor compared to Claude and GPT. — reactive:ai-benchmark-race (2026-06-25)
  9. [9] Deepswe's benchmark results are my own experience. — reactive:ai-benchmark-race (2026-06-26)
  10. [10] GLM-5.2 model has received official verification from the ARC-AGI evaluation. According to the results, GLM-5.2 achieved... — reactive:ai-benchmark-race (2026-06-26)
  11. [11] GLM-5.2 got 22.8% on ARC-AGI-2:, $0.25/task — Rohan Paul Twitter (2026-06-24)
  12. [12] With the new ARC-AGI results from GLM 5.2, together with the WeirdML results, we now have 8 private benchmark datapoints... — reactive:ai-benchmark-race (2026-06-27)
  13. [13] With the new ARC-AGI results from GLM 5.2, together with the WeirdML results, we now have 8 private benchmark datapoints... — reactive:ai-benchmark-race (2026-06-26)
  14. [14] With the new ARC-AGI results from GLM 5.2, together with the WeirdML results, we now have 8 private benchmark datapoints... — reactive:ai-benchmark-race (2026-06-26)
  15. [15] With the new ARC-AGI results from GLM 5.2, together with the WeirdML results, we now have 8 private benchmark datapoints... — reactive:ai-benchmark-race (2026-06-26)
  16. [16] GLM 5.2 just took the top spot on PostTrainBench by scoring 34.29%. — Rohan Paul Twitter (2026-06-26)
  17. [17] GLM-5.2: Built for Long-Horizon Tasks - Hugging Face — reactive:ai-benchmark-race
  18. [18] RT @bendee983: I'm usually very skeptical of AI benchmarks, but GLM-5.2 is really impressive. Aside from the benchmark r... — reactive:ai-benchmark-race (2026-06-23)
  19. [19] THE “BEST” AI MODEL CAME LAST IN MY REAL-WORLD TEST. — reactive:ai-benchmark-race (2026-06-24)
  20. [20] RT @WeiboLLM: ⭐ VibeThinker-3B is released — a dense 3B model for frontier-level verifiable reasoning. — reactive:ai-benchmark-race (2026-06-18)
  21. [21] VibeThinker-3B: A 3B Dense Reasoning Model Built on Qwen2.5-Coder-3B With the Spectrum-to-Signal Post-Training Pipeline ... — reactive:ai-benchmark-race (2026-06-21)
  22. [22] GLM-5.2 delivers a substantial leap in app development capabilities, which also represent demanding long-horizon tasks. — reactive:ai-benchmark-race (2026-06-19)
  23. [23] One of the biggest mistakes people make when evaluating LLMs is looking at a single benchmark and assuming it tells the ... — reactive:ai-benchmark-race (2026-06-23)
  24. [24] 1/ A 3 billion parameter model just beat Opus 4.5 (1T+ params) on math reasoning. — reactive:ai-benchmark-race (2026-06-23)
  25. [25] RT @ModelScope2022: Meet VibeThinker-3B, a 3B reasoning model from Weibo AI focused on math, coding, and STEM reasoning.... — reactive:ai-benchmark-race (2026-06-18)
  26. [26] RT @ValsAI: Full results for GLM 5.2 are here! — reactive:ai-benchmark-race (2026-06-18)
  27. [27] RT @ValsAI: Full results for GLM 5.2 are here! — reactive:ai-benchmark-race (2026-06-18)
  28. [28] @AndrewCurran_ What GLM 5.2 did to the best of the best is as big of a breakthrough as the R1 improvement purely from th... — reactive:ai-benchmark-race (2026-06-23)