Personal knowledge system built on Nate Jones’ Open Brain (OB1) skeleton, run fully local: no cloud embeddings, no remote MCP. Full narrative: Setting up my Digital Brain….
Invariant: the database is source of truth; the wiki is a regenerable view. Hand-edits to wiki pages are discarded on next compile.
Architecture
graph TB
VAULT["Obsidian vault
plain markdown, current-year journal"] PG["Postgres + pgvector
atomic thoughts rows"] OLLAMA["Local Ollama
mxbai-embed-large · qwen3"] GRAPH["Entities + typed
reasoning edges"] WIKI["Compiled wiki
entity + topic pages"] MCP["Claude Code ↔ stdio MCP
search / capture"] BRIEF["Daily brief
calendar · mail · Scholar"] VAULT -->|incremental import| PG PG -->|embed + extract| OLLAMA OLLAMA --> GRAPH GRAPH --> WIKI PG -.-> MCP WIKI -.-> MCP BRIEF -->|atomic notes| VAULT
plain markdown, current-year journal"] PG["Postgres + pgvector
atomic thoughts rows"] OLLAMA["Local Ollama
mxbai-embed-large · qwen3"] GRAPH["Entities + typed
reasoning edges"] WIKI["Compiled wiki
entity + topic pages"] MCP["Claude Code ↔ stdio MCP
search / capture"] BRIEF["Daily brief
calendar · mail · Scholar"] VAULT -->|incremental import| PG PG -->|embed + extract| OLLAMA OLLAMA --> GRAPH GRAPH --> WIKI PG -.-> MCP WIKI -.-> MCP BRIEF -->|atomic notes| VAULT
| Layer | Implementation choice |
|---|---|
| Notes | Obsidian vault (markdown), CarbonInterface journal lineage since 2015 |
| Store | Postgres + pgvector, one row per atomic thought |
| Embeddings | Ollama mxbai-embed-large → 1024-dim (not OpenAI 1536; models are not interchangeable mid-corpus) |
| Synthesis LLM | Local qwen3 (entity extraction, edge classification, wiki compile) |
| Agent interface | Local stdio MCP (edge functions cannot reach on-machine Ollama) |
| Wiki | Karpathy-style LLM-wiki: entity + topic pages, never hand-authored as SoT |
| Separations | Learning Biography (technical corpus only) ≠ personal autobiography |
Nightly job: import vault delta → drain entity queue → classify edges → regenerate wiki. Persistent across sleep; starts Ollama if down.
What works well
- Ask-past-me retrieval: semantic search over years of stuck-points beats manual
rgacross annual repos. - Local-only data path: embeddings, extraction, and MCP stay on-machine — appropriate for a decade of learning/struggle notes.
- Daily brief as a real habit change: calendar collisions, RSVP gaps, email triage, Scholar-alert skim, yesterday’s unfinished tasks — before opening either app.
- Scholar alerts as an ingestion channel: same pipeline as the curriculum sketch; keepers become atomic thoughts instead of dying in Gmail.
- Learning Biography as a compiled artifact: technical synthesis without absorbing personal journal content (and vice versa).
- Upstream fixes over silent forks: OB1 PRs #443 (kanban DnD), #444 (reading-list schema incl. abandoned), #445 (template-based daily-digest edge path, zero LLM cost).
In progress / still developing
| Surface | Status |
|---|---|
| Vault → Postgres incremental import | Working for current-year vault; historical backfill of 2015–2025 journals incomplete |
| Entity extraction + typed edges | Running via local scripts; quality/coverage uneven on sparse notes |
| Wiki compile (entity + topic pages) | Nightly path works; page quality and cross-link density still iterating |
| Daily brief (calendar / mail / Scholar) | In daily use; triage rules and false-positive rate under tuning |
| Learning Biography regenerator | Produces output; selection criteria and section schema still evolving |
| Claude Code MCP search/capture | Usable day-to-day; retrieval ranking and capture schemas need evaluation |
| OB1 upstream PRs (#443–#445) | Open / awaiting merge |
| Metrics dashboard for “learning growth” | Conceptual — refine in coming weeks |
| Embedding model migration story | Blocked by design: switching models ⇒ full re-embed; no partial migrate |
Evaluate closely / be wary of
- Embedding lock-in.
mxbai-embed-large@ 1024-d is a corpus-wide commitment. Re-embedding is a batch job, not a config flip. Benchmark recall before any model change. - Wiki overwrite hazard. Any human edit to compiled pages is ephemeral. Edit atomic thoughts (or extraction prompts), never the rendered wiki.
- Entity extraction hallucination / over-linking. Local LLMs will invent tidy graphs from thin notes. Spot-check high-degree entities; prefer typed edges with evidence pointers.
- Retrieval false confidence. Top-k semantic hits are not “I already solved this.” Always open the source thought; treat RAG as a pointer, not an answer.
- Daily brief scope creep. Calendar/mail automation that auto-archives or unsubscribes needs a narrow allowlist and an undo path. Prefer propose → confirm for destructive actions.
- Ollama availability. Nightly compile fails silently if the daemon isn’t up; the watchdog helps, but failed nights mean stale wiki — monitor logs.
- Personal / technical corpus bleed. Keep Learning Biography filters strict; contamination is hard to reverse once synthesized pages exist.
- MCP trust boundary. Stdio MCP talking to Claude Code still exposes vault content to the agent session. Treat prompts and tool results as sensitive; don’t point this at shared/work secrets without a policy.
- Evaluation debt. No frozen eval set yet for retrieval (precision@k on “have I seen this bug?” queries) or for wiki faithfulness. Add before claiming reliability.
Related
- Build note: /notepad/digital_brain_karpathy_wiki_rag/
- Downstream curriculum use of the same sources: /garden/multi-agent-edu-stack/ (MultiAgentEDUstack, in development)
- Literature-review context for Scholar ingestion: /notepad/week1_literature_review/