Introducing GeneBench-Pro

Claims

• GeneBench-Pro contains 129 synthetically generated computational biology problems requiring higher-order judgment on data quality, analysis path selection, and iterative assumption revision.
• GPT-5.6 Sol achieves a 31.5% pass rate with Pro mode enabled, compared to below 5% for GPT-5, indicating rapid frontier capability progress on scientific reasoning.
• Scaling test-time compute yields large returns: GPT-5.6 Sol at its highest reasoning level solves nearly six times as many problems as GPT-5.2 while using approximately two-thirds the tokens.
• Open-source models underperform GPT models significantly more on GeneBench-Pro than on coding benchmarks, suggesting GPT models have broader general scientific reasoning beyond coding specialization.
• Human experts require approximately 20-40 hours per problem at ~$200/hour while AI inference costs only several dollars per problem, creating a large economic opportunity even at partial reliability.
• Synthetic data generation with fully known causal structures enables deterministic grading and prevents analytic shortcuts or arbitrary answer-path ambiguities that plague other long-horizon biology benchmarks.