Retrieval-Augmented Generation

Source Excerpt

Category: AI Inference / Knowledge Retrieval

Maturity: Established

Executive Summary

Retrieval-Augmented Generation (RAG) is a pattern that enhances Large Language Model responses by retrieving relevant external documents before generating answers. Instead of relying solely on training data, the model accesses a knowledge base at inference time, producing more accurate and up-to-date responses. retired internal project's wiki functions as a RAG-like system — agents query the wiki for context before making decisions.

Definition

RAG combines two stages:

1. Retrieval — Given a user query, search a document corpus and return the most relevant passages
2. Generation — Feed the retrieved passages as context to an Large Language Model, which generates a response grounded in those passages

This contrasts with fine-tuning (which bakes knowledge into model weights) by keeping knowledge external and updateable.

Key Components

1. Document Indexing

• Documents are chunked into manageable sizes (typically 500-2000 tokens)
• Each chunk is embedded using an embedding model (e.g., text-embedding-3-small)
• Embeddings are stored in a vector database (e.g., pgvector, Chroma, FAISS)

2. Query Processing

• User query is embedded using the same model
• Vector similarity search finds top-K most relevant chunks
• Optional re-ranking step improves result quality

3. Context Assembly

• Retrieved chunks are formatted into a context window
• System prompt instructs the model to ground responses in retrieved content
• Model generates response with citations to source documents

RAG vs Wiki (retired internal project Pattern)

retired internal project's wiki is essentially a human-readable RAG system. Agents use search_files, read_file, and wiki_search.py as their retrieval mechanism, then synthesize answers from the retrieved content.

Common Patterns

Naive RAG

Simple embedding + similarity search. Works for small corpora but degrades with scale.

Advanced RAG

• Hybrid search — Combine vector + keyword (BM25) search
• Re-ranking — Cross-encoder model re-scores results
• Query transformation — Rewrite queries for better retrieval
• Metadata filtering — Filter by date, source, category before retrieval

Modular RAG

Separate retrieval and generation into independent modules that can be improved independently.

Implementation at retired internal project

Our wiki serves as the primary knowledge retrieval layer:

• Indexing: Pages are organized with YAML frontmatter tags and categories
• Retrieval: wiki_search.py (content search), search_files (grep), index.md navigation
• Generation: Agents synthesize retrieved content into responses

The Karpathy Large Language Model Wiki pattern is essentially RAG designed for Large Language Model agents as the primary users, with humans as secondary consumers.

Related Concepts

• concepts/large-language-models — Foundation models that power generation
• Knowledge Management — Organizing knowledge for retrieval
• concepts/vector-databases — Storage systems for embedding-based retrieval