Using DSPy to evaluate and improve Datasette Agent's SQL system prompts

Simon Willison · Simon Willison · 2026-07-02

Simon Willison uses Claude Fable 5 to run an async DSPy research task on Datasette Agent's SQL system prompts, uncovering that the schema listing omits column names and triggers costly column-guessing and error-retry loops in the agent.

Open original ↗

Appears in

Claude Fable 5: Model Update, Safety Profile, Benchmarks, and Subscriber Trial Rollout

Extraction

Topics: dspyprompt-optimizationai-agentssql-agentsevals

Claims

DSPy can be applied to evaluate and optimize system prompts for Datasette Agent's SQL query execution feature.
Datasette Agent's current schema listing shows only table names, which causes the model to guess column names and enter error-retry loops.
Claude Fable 5 was used asynchronously via Claude Code for web to conduct the DSPy research and prompt evaluation task.
Fable 5 autonomously selected GPT-4.1 mini and nano as the evaluation models for the DSPy experiments.

Key quotes

The schema listing gives only table names; the 'don't call describe_table if you already have the information' advice caused column-name guessing (page_count, o.order_id, first_name) and error-retry loops in baseline traces. Either include column names in the prompt's schema listing or soften that advice.