Research at AI Speed by John Langford #78

Activity

1. lintool

   added

   advice-by-others

   on Oct 13, 2026
2. lintool commented on Oct 25, 2026

   

   lintool

   on Oct 25, 2026

   OwnerAuthor

   More actions

   My main takeaway: as AI makes implementation cheaper, good tooling and experimental judgment become more valuable.

   Three pieces of advice stand out:

   • Invest in good tooling. Reliable experimental pipelines, clean code, tuned baselines, systematic tests, and accessible logs make rapid experimentation useful. Building and maintaining that infrastructure is valuable research work; use and support strong existing pipelines when possible.
   • Keep humans responsible for experimental design and review. Running more experiments helps when they answer meaningful questions and produce evidence you can trust.
   • Allocate compute in stages. Screen ideas cheaply, confirm promising results against shared controls, and scale up the survivors.

   For students, a useful question is: If you could implement ten times as many ideas, how would you decide which ones to test—and what evidence would make you abandon one?

   See especially slides 12–19 on agents and pipelines, and 46–51 on evaluating changes.

   Drafted by OpenAI Codex with direction from @lintool.