Glossary

Speak fluent local AI

Every term that matters, in plain language. Bookmark this — model cards make much more sense after it.

Foundations

LLM: Large Language Model — a neural network trained on massive text corpora to predict the next token, which turns out to produce general-purpose language ability.
Token: The chunk of text (≈4 characters or ¾ of a word) models read and write. Everything is priced and measured in tokens.
Context window: How many tokens the model can consider at once — its working memory. Bigger windows cost more VRAM.
Inference: Running a trained model to get outputs. Local inference means this happens on your hardware, no server involved.
Tokens per second (tok/s): Generation speed. Reading pace is ~7 tok/s; comfortable chat is 15+; 40+ feels instant.
Multimodal: Models that handle more than text: images (vision), audio, or both in one conversation.

Open weights: Model parameters you can download and run yourself. Differs from open-source: training code and data may not be included.
Permissive license: Licenses (MIT, Apache-2.0) allowing commercial use with minimal strings — the safe choice for building products on local models.
Self-hosting: Running services on hardware you control. The philosophy behind every recommendation on this site.

Quantization: Shrinking weights from 16-bit floats to 4–8 bit integers so models fit in less VRAM, trading a little quality for a lot of memory.
GGUF: The file format used by llama.cpp and friends: one file containing weights + metadata + tokenizer, ready to quantize.
Q4_K_M: The community's default 4-bit quantization — the quality/size sweet spot most VRAM estimates assume.
KV cache: Memory that stores attention states for the context so generation stays fast. Grows with context length — long chats eat VRAM.

MoE (Mixture of Experts): Architecture where only a few 'expert' sub-networks activate per token — big-model knowledge at small-model speed, if you have the memory.

Distillation: Training a smaller model to mimic a larger one's outputs — how 32B models inherited reasoning from frontier giants.
Fine-tuning: Further training a model on your data to change its behavior or knowledge. LoRA makes this feasible on a single GPU.
LoRA / QLoRA: Low-Rank Adaptation: training tiny adapter matrices instead of all weights. QLoRA adds 4-bit base-model compression — fine-tunes that fit in 12–24GB.

Embedding: A vector of numbers representing meaning. Similar texts get similar vectors — the foundation of search and RAG.
RAG: Retrieval-Augmented Generation: fetch relevant passages from your documents, then let the model answer using them. Kills hallucinations about your private data.
Vector database: Storage optimized for searching embeddings by similarity — the filing cabinet of every RAG pipeline.

VRAM: Memory on your GPU. The single number that decides which models you can run. Buy VRAM first, marketing later.
Unified memory: Apple-silicon design where CPU and GPU share one memory pool — a 64GB Mac can load a 70B model that no consumer PC could.
GPU offload: Splitting model layers between GPU (fast) and CPU/RAM (slow). More VRAM = more layers offloaded = more speed.

Prompt: The instructions and context you give a model. The highest-leverage skill in AI — and completely free to improve.
System prompt: Standing instructions that shape every response — the model's 'job description' for your session.
Temperature: Randomness dial: 0 = deterministic and literal, 1+ = creative and loose. Lower for code and facts, higher for brainstorming.
Hallucination: Confident nonsense. Reduced by grounding (RAG), smaller scopes and asking for sources — never fully eliminated.

Diffusion: Image generation by iterative denoising — how SDXL and FLUX turn noise into pictures, guided by your prompt.
Sampler & steps: The denoising recipe for image models. FLUX schnell needs 4 steps; SDXL typically 25–35. More steps ≠ better, just slower.

Agent: A model in a loop with tools: it plans, calls functions, observes results and continues — local agents keep your data in-house.
Function calling: Structured outputs that trigger real code — the bridge between chat and automation. Many local models support it natively now.

Speculative decoding: A small model drafts tokens, the big one verifies in parallel — same quality, 2–3× speed when paired well.
Prompt caching: Reusing computed attention for repeated context prefixes — instant follow-ups on long documents, locally and in the cloud.