Most “chat with your notes” systems start from the same assumption: if you chunk documents, embed them, and retrieve the nearest pieces at query time, you have a workable knowledge layer for an LLM. In practice, that approach works often enough to be popular — but not consistently enough to be satisfying.

This article explores a different path. Instead of treating retrieval as a similarity-search problem over raw text, it treats knowledge building as a distillation problem: let the LLM read the source first, turn it into structured wiki pages, and use that wiki as the thing your assistant actually reads later.

This is Part 1 of a 4-part series on building a daily-use second brain without a vector DB. The goal here is to set the foundation: why “chunk + cosine” RAG often falls short, and how Andrej Karpathy’s LLM Wiki idea reframes the problem.

personal-llm-wiki-distill-instead-of-vector-rag

Why “chunk + vector” RAG isn’t always the answer

Most “chat with your notes” tutorials follow the same recipe:

  1. Split documents into 500–1000-token chunks.
  2. Embed each chunk into a vector DB.
  3. At query time, embed the question, retrieve cosine top-k chunks, stuff them into the prompt, and let the LLM answer.

It works, but two cracks show up quickly.

Similarity is not the same thing as relevance. A chunk whose embedding is close to the question does not necessarily contain the answer. This becomes especially obvious when the question requires reasoning, such as “compare A and B” or “summarize the steps.” In those cases, top-k often returns chunks that sound related but do not actually answer the question.

Chunk boundaries are arbitrary. You do not control where the cut lands, and chunks routinely split in the middle of a sentence, an argument, or an important transition. The LLM sees fragments, glues them together with outside knowledge, and produces an answer that sounds fluent but may still be hallucinated. The more documents you add, the worse this tends to get.

Karpathy proposes a different framing

In late 2024, Andrej Karpathy floated a simple but powerful idea: instead of storing raw text and retrieving by similarity, let the LLM distill the source into a structured wiki page first. Then later, when you ask a question, the LLM reads the wiki directly. The knowledge is already condensed. No fancy retrieval required.

That shifts the problem in two important ways:

  • The storage unit becomes a concept page, not a chunk. Each page wraps one idea, such as “What is RAG” or “Bi-temporal schema,” and rewrites it with headings, bullets, and cross-links.
  • “Retrieval” becomes dumping the whole wiki into the prompt. When the wiki is small enough — a few hundred pages or less — modern 80k-token contexts can swallow it whole, with no embeddings and no vector DB.

The payoff is substantial: you end up with a knowledge base that humans can read, audit, and edit by hand, instead of a binary blob only the machine understands.

The three-folder layout

wiki-root/
raw/     # original sources — immutable, content-addressed by sha256
wiki/    # distilled pages — 1 page = 1 .json file
log.md   # append-only journal of every operation

Each folder has a clear role:

  • raw/ is the audit root. It is never edited. Filenames follow the pattern <sha256-short>_<slug>.<ext>, so the same content is not stored twice, and you can detect drift when an external source changes underneath you.
  • wiki/ is what the LLM actually reads at query time. It is editable, supersedable, and archivable.
  • log.md records entries like {ts, action, title, ...}, one JSON line at a time, append-only. If something goes wrong, you can replay what happened.

What a wiki page looks like

{
  "title": "Retrieval-Augmented Generation",
  "summary": "Combine document retrieval with generation so the LLM answers with sources.",
  "content": "## What is RAG\nRAG fuses retrieval and generation...\n\n## Vector RAG vs Reasoning RAG\n...",
  "tags": ["rag", "llm", "retrieval"],
  "links": ["Vector DB", "Embedding"],
  "domain": "Learning",
  "source": "Lewis et al. 2020",
  "source_authority": 0.95,
  "confidence": 0.9,
  "ingested_at": "2026-05-10T...",
  "updated_at": "2026-05-20T..."
}

Three things matter here.

  • content is markdown the LLM rewrote from the source, not a raw copy. It is cleaned up, structured with headings, and can include [[wikilinks]] to other pages.
  • source_authority captures how trustworthy the source is: official docs = 1.0, a paper = 0.9, a blog = 0.5, “some tweet” = 0.3. confidence scores the page content itself, as judged by the LLM.
  • tags and domain are cheap but useful filters. domain is single-valued, such as Work, Learning, or Personal; tags are free-form and multi-valued.

The ingest pipeline — LLM distills, never chunks

The ingest flow has four steps:

raw text / PDF / URL
↓ stored as raw/<sha>_<slug>.txt (content-addressed)
↓
LLM compile prompt:
"Read this source → produce 1–3 wiki pages, one concept each,
use [[wikilinks]] to reference existing pages."
↓
JSON: { pages: [{title, content, tags, links}, ...] }
↓
save_page (upsert by title) → wiki/<slug>.json + log.md

A minimal backend sketch:

async def ingest_text(content: str, source_label: str, authority: float):
    sha = sha256(content)
    raw_ref = save_raw(content, source_label, sha)

    pages = await llm_compile_to_pages(content, source_label, authority)
    # LLM returns: [{"title": "...", "content": "## ...",
    #               "tags": [...], "links": [...]}, ...]

    saved = []
    for p in pages:
        await save_page({**p,
                         "source_sha256": sha,
                         "source_authority": authority})
        saved.append(p["title"])
    return saved

The compile prompt, trimmed:

You are a wiki compiler. Read the source below and produce 1–3 pages.

Rules:
- Each page = one core concept. No catch-all pages.
- Use ## / ### headings, bullets, short examples.
- Cross-reference other pages with [[Concept Name]]. PREFER linking to
  existing pages (see "Existing pages" list below).
- DO NOT fabricate — write only what the source supports.
- confidence = 0.9 if the source is clear; 0.6 if it's vague.

Existing pages: {{titles}}
Source (authority = {{authority}}):
{{raw_text}}

Return JSON ONLY:
{"pages":[{"title":"...","content":"## ...","tags":[...],
           "links":[...],"confidence":0.85}]}

Two practical notes matter here:

  • LLM JSON output is fragile. Use something like tolerant_json_loads with a handful of repair-and-retry passes instead of plain json.loads. Open-weight models like Gemma and Llama often emit stray \$ and \( escapes, so strip those before parsing.
  • LLM context limits are real. Dumping a 50-page PDF into one compile call degrades quality. The better fix is to chunk by natural structure — table of contents, headings, sections — rather than arbitrary token windows.

[[Wikilinks]] make knowledge weave itself

The key wiki insight is simple: a graph beats a list. Pages can embed [[Page Name]] to point at other pages. At render time:

const wikiRe = /\[\[([^\]]+)\]\]/g;
const html = text.replace(wikiRe, (_, t) =>
  `<span class="wikilink" data-page="${t}">${t}</span>`);

Click the link and open that page. During compile, the prompt tells the LLM to prefer linking to existing pages by using the existing_titles list passed in. The more you use the system, the denser the graph becomes.

That gives you a form of free retrieval. While reading the RAG page, you see [[Vector DB]] inline, click once, and you are there. No search. No embedding.

A small but useful touch: if [[Name]] does not yet match an existing page, render it as a broken link in a different color. That becomes a writing prompt for you — the KB starts generating its own TODO list.

save_page with update-in-place

Same title, merge it — do not spawn a “v2”:

async def save_page(page: dict):
    title = page["title"]
    existing = get_page(title)
    now_ts = now()

    merged = {
        "title": title,
        "summary": page.get("summary", ""),
        "content": page["content"],  # overwrite with the new LLM compile
        "tags": page.get("tags", []),
        "links": page.get("links", []),
        "source_authority": page["source_authority"],
        # Keep the trail
        "ingested_at": (existing or {}).get("ingested_at", now_ts),
        "updated_at": now_ts,
        # ... (bi-temporal fields — Part 2)
    }
    page_path(title).write_text(json.dumps(merged, ensure_ascii=False))
    append_log("update" if existing else "create", title=title)

Update-in-place is not just an implementation detail — it is a principle. It prevents the slow drift into “RAG,” “RAG v2,” “RAG final,” and “RAG real (final).” One concept should map to exactly one page.

Versioning lives in log.md plus the bi-temporal mechanism introduced in Part 2, not in the page title itself.

the bi-temporal mechanism

When this pattern fits and when it does not

This approach is a good fit in the following cases:

  • Personal or small-team knowledge bases with fewer than about 1000 pages. An 80k-token whole-wiki dump can still work.
  • Domains that need concept connection — a graph — rather than exact lookup.
  • Cases where you want a KB that is human-readable, hand-editable, and auditable.
  • Narrative-shaped sources such as papers, blog posts, and transcripts, where distillation pays off.

This approach is less suitable in the following cases:

  • Huge KBs with millions of documents, where LLM compile cost becomes prohibitive.
  • Exact lookup workloads such as price tables or product codes, where SQL is the better tool.
  • Highly structured sources such as CSVs or API logs, where no distillation is needed.

Once your wiki grows past roughly 80k tokens, there are still vector-free escape hatches. Part 4 covers two of them: TOC-driven ingest, where each section becomes a page, and domain scoping, where the LLM only reads the domain the user asked about.

Try it before moving on

The minimum needed to spin up a mockup in a few hours is surprisingly small:

  • One OpenAI-compatible LLM endpoint, whether OpenAI, Anthropic, Gemma, or Ollama.
  • A filesystem with raw/, wiki/, and log.md. Your laptop is enough.
  • Roughly 200 lines of Python for ingest_text, save_page, and a chat loop that dumps the wiki into the prompt.

Start small. Take 3–5 blog posts you read this week, ingest them, and ask a few questions. You may notice something surprising: the LLM often answers more cleanly from the distilled wiki than from the raw source text, because an LLM has already done the first pass of reading and compression for you.

What the next three parts cover

  • Part 2. Forget ≠ Delete: bi-temporal schema, supersession, and decay_scan. Why “don’t delete, flag it stale.” Is the page you wrote in March last year still true?
  • Part 3. Cite-or-Refuse: a 2-pass enforcement mechanism that forces the LLM to answer from your wiki instead of sneaking in outside knowledge.
  • Part 4. Scaling out: multi-user sharing plus TOC-driven ingest. What to do when the wiki outgrows the context window, or when two people share the same KB.

This series is based on a real system I built with GreenNode AgentBase, Notion, and HuggingFace Space. Still, the point is not to copy the stack. The real value is in the design thinking underneath it. Once you grasp that, you can rebuild the same pattern with simpler tools like FastAPI, SQLite, and local files and make it your own.

TagAI AgentTutorial