Brainstore Architecture: AI Observability
Braintrust’s custom database “Brainstore” for AI observability at scale. Built because traditional databases (Postgres, warehouses, DuckDB) couldn’t handle AI trace workloads.
Key Claims
- AI observability is a database problem — the novel workloads AI products generate require custom storage
- AI traces are 2-3 orders of magnitude larger than traditional APM traces: typical span ~50KB, typical trace ~10MB, p90 traces can reach tens of GB
- Agent traces are particularly challenging: multi-turn, complicated tool calls, run for days
- Three-part stack (warehouse + Postgres + browser DuckDB) broke down under AI workloads
- Brainstore architecture: Tantivy (fast search/indexing) + object storage, with WAL for writes
Architecture Details
- All data on object storage (unbounded scale, no persistent disks)
- Per-customer partitioning (not “one massive table”)
- Semi-structured data as first-class citizen
- Write path: WAL → processing (segment assignment) → compaction (inverted index, row store, column store)
- Read path: query pipeline with AST parsing, filter pushdown, Tantivy queries
- Real-time: merges WAL + processed-not-compacted + fully indexed data
Takeaways
- 100K spans/second ingest requirement
- Agent-era observability needs: traces that run for days, out-of-order feedback, PII in prompts
- Custom BTQL query language unified across stack
- The “build vs buy” decision for databases: justified when existing tools regularly fail at your workload
- Operational simplicity matters for self-hosted deployments
Connections
- Welcome to the Machine — infrastructure must adapt for agent workloads
- Context Engineering — observability of what goes into agent context
- Anthropic Infrastructure Postmortem — production infrastructure challenges