RAG: chat with your own documents, offline

Cloud AI cannot read your private documents without you shipping them to someone else’s server. Local RAG fixes that: your files, your embeddings, your model, zero leaks.

The 3-minute mental model

RAG = Retrieve + Augment + Generate.

1. Your documents are split into passages and turned into embeddings (vectors of meaning).
2. Your question is embedded the same way.
3. The most similar passages are retrieved and handed to the LLM as context.
4. The model answers grounded in those passages — usually with citations.

No fine-tuning required. Your model’s knowledge stays generic; your data stays yours.

The 15-minute hands-on (no code)

AnythingLLM is the fastest path:

1. Install the desktop app.
2. Point it at your local Ollama (it auto-detects localhost:11434).
3. Create a workspace → upload PDFs, Markdown, DOCX.
4. Ask questions. Toggle “show citations” and check the sources panel.

Everything — embeddings included — is stored locally by default.

The concepts you just used

• Embedding model: BGE-M3-class, multilingual, runs in under 2GB.
• Vector store: AnythingLLM embeds one internally; bigger setups use LanceDB, Qdrant or Postgres+pgvector.
• Chunking: documents are split into ~500–1000 token passages. Too small loses meaning; too big drowns the context.

When to graduate from no-code

Build your own pipeline when you need: custom metadata filters, hybrid search (BM25 + vectors), or production APIs. The stack is boring by design: llama-index or langchain, a local embedding server, SQLite/pgvector, your Ollama endpoint.

Quality checklist

• Answers cite passages you can open
• Wrong-answer test: ask something NOT in the docs — it should say so
• Multilingual docs? Use a multilingual embedder (BGE-M3)
• Long PDFs? Increase chunk overlap slightly

Where to next

• Glossary: RAG, embeddings, vector database
• Models: BGE-M3 and Qwen3 8B make the default local RAG pair.