Building RAG from Scratch (No LangChain, No LlamaIndex)

Retrieval-Augmented Generation frameworks abstract the pipeline so aggressively that debugging a bad answer means reading three layers of abstraction before you reach the actual retrieval logic. Building it yourself takes a few hundred lines of Python and gives you complete control over every decision.

What RAG Actually Is

RAG is three steps chained together:

Retrieve — find the chunks of text most relevant to the query
Augment — prepend those chunks to the prompt as context
Generate — ask the LLM to answer using only that context

The intelligence is mostly in step 1. A better retrieval system gives the LLM better context and produces better answers.

Step 1: Chunking

Text needs to be split into chunks small enough for a model's context window but large enough to be coherent. A sliding window with overlap prevents cutting a thought in half:

python

def chunk_text(text: str, max_words: int = 450, overlap: int = 70) -> list[str]:
    words = text.split()
    step = max_words - overlap
    chunks = []
    for i in range(0, len(words), step):
        chunk = " ".join(words[i : i + max_words])
        if chunk.strip():
            chunks.append(chunk)
    return chunks

At 450 words with 70-word overlap, adjacent chunks share context so a sentence split across a boundary still appears in full in at least one chunk.

For PDFs and HTML, strip formatting first:

python

from pypdf import PdfReader
import re
 
def extract_pdf(path: str) -> str:
    reader = PdfReader(path)
    text = "\n".join(page.extract_text() or "" for page in reader.pages)
    return re.sub(r'\s+', ' ', text).strip()

Step 2: Embedding and Indexing

Embed each chunk and store both the text and vector:

python

import sqlite3
import json
import numpy as np
from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("all-MiniLM-L6-v2")
 
conn = sqlite3.connect("rag.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS chunks (
        id INTEGER PRIMARY KEY,
        source TEXT,
        text TEXT,
        vector TEXT
    )
""")
 
def index_document(source: str, text: str):
    chunks = chunk_text(text)
    for chunk in chunks:
        vec = model.encode(chunk, normalize_embeddings=True)
        conn.execute(
            "INSERT INTO chunks (source, text, vector) VALUES (?, ?, ?)",
            (source, chunk, json.dumps(vec.tolist()))
        )
    conn.commit()

Step 3: Retrieval

For each query, embed it and compute cosine similarity against every stored chunk:

python

def retrieve(query: str, top_k: int = 5) -> list[dict]:
    q_vec = model.encode(query, normalize_embeddings=True)
 
    rows = conn.execute("SELECT id, source, text, vector FROM chunks").fetchall()
    scored = []
    for row_id, source, text, vector_json in rows:
        vec = np.array(json.loads(vector_json))
        score = float(np.dot(q_vec, vec))  # cosine sim (both normalized)
        scored.append({"id": row_id, "source": source, "text": text, "score": score})
 
    return sorted(scored, key=lambda x: x["score"], reverse=True)[:top_k]

For large corpora, replace the brute-force scan with ChromaDB or FAISS. For hybrid retrieval, add an FTS5 keyword layer and merge with RRF before returning.

Step 4: Synthesis with Citation

Build the prompt with explicit source labels:

python

def build_prompt(query: str, chunks: list[dict]) -> str:
    context_parts = []
    for i, chunk in enumerate(chunks, 1):
        context_parts.append(f"[{i}] {chunk['source']}\n{chunk['text']}")
 
    context = "\n\n".join(context_parts)
    return f"""Answer the question using only the provided sources. 
Cite sources as [1], [2], etc.
 
Sources:
{context}
 
Question: {query}
Answer:"""

Call your LLM of choice with this prompt. The explicit numbering makes the model cite correctly.

Step 5: Parsing Citations Back

After synthesis, map citation numbers back to source paths:

python

import re
 
def parse_citations(answer: str, chunks: list[dict]) -> dict:
    cited_indices = set(int(m) for m in re.findall(r'\[(\d+)\]', answer))
    return {
        idx: chunks[idx - 1]["source"]
        for idx in cited_indices
        if 1 <= idx <= len(chunks)
    }

What You Control That Frameworks Hide

Chunk size and overlap: frameworks default to 512 tokens. For technical docs, 450 words with 70-word overlap often retrieves better.
Retrieval method: most frameworks default to vector-only. Adding keyword search with RRF is a significant quality improvement.
Synthesis prompt: you control the citation format, the instruction phrasing, and what happens when no relevant context is found.
Fallback behavior: if the LLM call fails, you can fall back to extractive retrieval — return the top chunk's text directly.

The whole pipeline without ChromaDB or an LLM dependency is about 150 lines. With those, still under 300. Readable, debuggable, and entirely yours.