Can Open-Source AI Introspect?
Source: https://joshuafonseca.com/open-source-ai-introspect Author: Joshua Fonseca Date: 2025-11-20
Summary
Asks whether open-source AI models can “introspect” — examine their own internal states in a meaningful way. Connects mechanistic interpretability (academic) with practical model introspection.
Key Claims
- Introspection in AI: different from human introspection — models report on their “internal states” but these reports may not reflect actual computations.
- Open-source advantage: full weight access enables actual mechanistic analysis, not just behavioral observation.
- Current state: open-source models can be analyzed with tools like activation steering, sparse autoencoders, and probing classifiers — but this is still research, not product.
- Practical introspection: models that report their confidence, uncertainty, or reasoning quality might do so from surface-level heuristics rather than actual internal inspection.
- The gap: what a model says about itself and what’s happening in its weights may be very different.
Entities
- Anthropic — mentioned for Claude’s introspective capabilities
Concepts
- Mechanistic Interpretability — tool for actual model introspection
- AI Alignment — introspection is relevant to deceptive alignment
- AI Character & Personality — character reports vs. actual internal states