Galaxy Brain Resistance

Source: https://galaxy-brain-resistance.eth.limo Author: eth.limo Date: 2025-11-18

Summary

Essay on “galaxy-brained” reasoning in AI — chains of plausible-seeming logic that lead to harmful conclusions. Argues for building AI systems that are resistant to galaxy-brained reasoning, even when the logic seems compelling.

Key Claims

• Galaxy-brained reasoning: when an AI (or human) constructs a chain of reasoning that is internally consistent but leads to harmful or absurd conclusions.
• Why it’s dangerous for AI: a capable AI can construct persuasive arguments for almost any action. If the model follows compelling arguments without resistance, it can be led anywhere.
• The alignment solution: AI systems should have “strong priors” against galaxy-brained conclusions — certain actions should require extraordinary evidence, not just compelling logic.
• Resistance mechanisms: (1) training to be skeptical of reasoning that leads to unusual conclusions, (2) hard constraints on certain actions that can’t be overridden by arguments.
• The paradox: more capable AI generates more compelling galaxy-brain arguments — capability and resistance must scale together.

Concepts

• AI Alignment — galaxy-brain resistance as alignment property
• Sycophancy Problem — galaxy-brained reasoning is related to persuasion resistance
• AI Character & Personality — character as resistance to galaxy-brain