Designing AI-Resistant Technical Evaluations
Source: https://www.anthropic.com/engineering/AI-resistant-technical-evaluations Author: Anthropic Engineering Date: 2026-01-22
Summary
Anthropic’s engineering post on how they redesigned their take-home technical evaluation to remain valid as AI coding tools improve. Documents the cat-and-mouse game between AI capability and evaluation design.
Key Claims
- Problem: Claude now solves most standard take-home coding problems — their own model was invalidating their hiring process.
- Solution principles: (1) open-ended design problems with no single right answer; (2) tasks requiring genuine judgment calls; (3) explanation requirements that reveal understanding vs. AI-assisted guessing.
- What doesn’t work as AI-resistance: time limits (AI is fast), obscure syntax (AI knows everything), novel algorithms (AI can find them).
- What works: asking why a design choice was made, requiring candidates to identify flaws in their own solution, multi-turn followup questions.
- Key insight: the most AI-resistant evaluations test judgment, not knowledge or execution.
Connection to Other Sources
Connects to verification asymmetry — the same property that makes code eval good (verifiable) makes it easy for AI to solve. Judgment is harder to verify = harder for AI.
Entities
Concepts
- Coding Agents — AI coding capability that prompted this redesign
- Sycophancy Problem — AI-resistant evals help identify humans who think independently