Search and thinking settings produced modest results

Answer-key agreement on 500 SNAP/CalFresh questions. Sol improved with search at medium effort; Astra stayed near 94%.

Dot plot of five model and search conditions, with low and medium thinking scores. Astra: no search 94.0% low and 94.1% medium; search 93.6% low and 94.0% medium. Sol: no search 92.4% low and 91.8% medium; search 92.6% low and 93.2% medium. Opus 5.5: no search 92.4% low and 92.1% medium. Opus search was incomplete and is not plotted.

(Please use a modern browser to see the interactive version of this visualization)

Same questions and Sept. 24 policy date; no specialized guidance. Astra and Opus medium/no-search points average three runs; others show one run. Opus search runs did not complete after provider errors and are omitted. Small gaps may reflect run variation. Axis begins at 91%.

Source: Local Inspect evaluation logs, September 2026Get the data