RAG Implementation Plan — Converting the me/ Folder into Proper RAG

A step-by-step guide for converting the documents in the me/ folder from the current "context stuffing" approach into a real Retrieval-Augmented Generation (RAG) pipeline.

Honest framing: The total content is only ~26k characters across all files. At that size, stuffing everything into the prompt (the current approach) genuinely works fine — RAG won't make the answers better here. Treat this purely as a learning exercise in the mechanics. That's the right reason to do it.

---

The Conceptual Shift

Right now system_prompt() jams the entire summary + CV + LinkedIn into every request. RAG replaces "include everything" with "include only the few chunks relevant to this question."

Four moving parts:

1. Chunk the documents into smallish passages.
2. Embed each chunk into a vector (once, ahead of time) and store it.
3. At query time, embed the question and retrieve the nearest chunks.
4. Inject only those chunks into the prompt.

---

Step 1 — Chunking

There's a lucky break here: me/sources_Andras_Nemes.txt already has natural section boundaries (-------). That's a ready-made chunking scheme — each section (summary, elevator pitch, leadership style, key achievements, ama, Flexera intro, contact) becomes one chunk.

with open("me/sources_Andras_Nemes.txt", encoding="utf-8") as f:
    raw = f.read()
chunks = [c.strip() for c in raw.split("-------") if c.strip()]

For the PDFs (CV, LinkedIn), there's no delimiter, so use a generic splitter — fixed-size windows with overlap so a sentence isn't cut mid-thought:

from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
cv_chunks = splitter.split_text(self.read_pdf("me/Andras_Nemes_CV_2026.pdf"))

The two parameters that matter:

• chunk_size — too big → you retrieve irrelevant filler; too small → you lose context.
• chunk_overlap — so facts straddling a boundary survive.

500–1000 chars with ~15% overlap is a sane starting point.

---

Step 2 — Embed + Store

Turn each chunk into a vector and put it in a vector store. Two common choices:

Option A — Chroma (a real vector DB, persists to disk)

import chromadb
from openai import OpenAI

client = OpenAI()
def embed(texts):
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in resp.data]

db = chromadb.PersistentClient(path="me/vectorstore")
col = db.get_or_create_collection("andras")
col.add(ids=[f"c{i}" for i in range(len(chunks))],
        documents=chunks,
        embeddings=embed(chunks))

Option B — No database at all (instructive)

This demystifies what a vector DB does: store the vectors in a NumPy array and do cosine similarity by hand. For ~30 chunks this is genuinely all a vector DB is doing under the hood:

import numpy as np
vectors = np.array(embed(chunks))          # shape: (n_chunks, 1536)
# retrieval = dot product after normalizing

Recommendation: Do Option B first as the exercise — it makes RAG click — then swap in Chroma to see what the library buys you (persistence, scaling, metadata filtering).

This whole step runs once (or whenever the me/ files change), not on every chat.

---

Step 3 — Retrieve at Query Time

def retrieve(self, question, k=3):
    qvec = embed([question])[0]
    # Chroma:
    res = self.col.query(query_embeddings=[qvec], n_results=k)
    return res["documents"][0]
    # or NumPy: normalize, dot-product against `vectors`, argsort top-k

k (how many chunks to pull) is the third knob. Start with 3–5.

---

Step 4 — Wire It Into the Me Class

This is the satisfying part — it's a small change to the existing code:

• In __init__: build (or load) the vector store instead of reading whole files into self.summary / self.cv / self.linkedin.
• Change system_prompt() into system_prompt(self, message) so it can take the user's question, retrieve, and inject only the matched chunks:

def system_prompt(self, message):
    context = "\n\n---\n\n".join(self.retrieve(message))
    prompt = f"You are acting as {self.name}... "
    prompt += f"\n\n## Relevant context:\n{context}\n\n"
    return prompt

• In chat(): self.run_with_tools(self.system_prompt(message), message, history).

Note: This also means the evaluator should retrieve the same way, or just receive the same retrieved context, so the judge sees what the agent saw.

---

Gotchas Worth Anticipating

• Build vs query separation — don't re-embed the whole corpus on every message (slow + costs money). Embed once, persist, load. The classic beginner bug is putting col.add(...) inside chat().
• Retrieval can miss — if someone asks "What car do you drive?", retrieval returns the closest chunks even though none are relevant. So keep the "if you don't know, call record_unknown_question" instruction — RAG doesn't replace it.
• Small-corpus caveat — with only ~30 chunks, top-k retrieval might pull most of the content anyway, so you may not see a quality difference. That's expected; the value shows up at hundreds/thousands of chunks.
• Embedding model — text-embedding-3-small is cheap and plenty for this. The OpenAI key is already configured.

---

Suggested Exercise Path

1. Start in a notebook, not app.py — chunk sources_Andras_Nemes.txt by -------, embed, and do top-k retrieval by hand with NumPy. Print the retrieved chunks for a few questions to build intuition.
2. Swap NumPy for Chroma with persistence.
3. Add the PDFs with a real text splitter.
4. Finally, fold it into a copy of app.py (e.g. app_rag.py) so the working app stays intact — same pattern as keeping app_original.py.

---

Quick Reference — The Three Tuning Knobs

Knob	Where	Starting value	Effect
chunk_size	text splitter	500–1000 chars	Granularity of retrieved passages
chunk_overlap	text splitter	~15% of chunk_size	Prevents facts being cut at boundaries
k	retrieval	3–5	How many chunks injected into the prompt