Think Harder, Sample Once

Comparing single-shot LLM paper selection against multi-run consensus voting.
nlp
Published

January 3, 2025

I want to find the most interesting papers from ~200 recent AI publications. “Interesting” sounds subjective, but I gave the model specific criteria: prioritize counter-intuitive findings and emergent behaviors, downweight incremental benchmarks, and so on. The question: can I trust a single response, or do I need to ask 30 times and vote?


Below, I compare single-shot (one call, take what you get) against consensus (30 runs, keep papers that appear in >50%). The answer depends on how hard the model thinks.

Results for :

Single-shot

Papers from one sampled run

Top 5 by frequency

Muted = below 50% threshold

▼ All Selections % of 30 runs

* in top X · in single-shot

Why does reasoning effort matter so much? Recent work suggests that extended “thinking” performs internal sequential refinement—each reasoning step correcting the previous. Parallel voting approximates this externally, but at 30× the cost.

Takeaway: I’ll go with high reasoning + single-shot. More reliable and cheaper than running lower effort 30 times.