Doc-to-LoRA: Documents as LoRA Adapters
Source: https://pub.sakana.ai/doc-to-lora/ Author: Sakana AI Date: 2025
Summary
Converts documents into compact LoRA adapters through a hypernetwork, enabling models to “internalize” new knowledge without retraining. Generates adapters in <1 second (vs 40-100s for context distillation). Maintains near-perfect retrieval accuracy on documents up to 40K tokens despite 8K context limit.
Key Claims
- Mechanism: frozen LLM encodes document activations → Perceiver-based hypernetwork → rank-8 LoRA matrices
- Speed: <1s adapter generation vs 40-100s for context distillation
- Memory: 50 MB storage vs 12+ GB for in-context retrieval
- Performance: 83.5% of full-context performance on SQuAD reading comprehension
- Long documents: near-perfect retrieval on 40K token docs despite 8K context via chunking
- Vision generalization: text-only model achieves 75% image classification accuracy after “reading” VLM-described images
Concepts
- Agent Memory — LoRA as externalized knowledge storage
- Synthetic Data — training hypernetwork with distillation signal