1. Prompting: The Foundation

Every production AI system starts with one thing: a well-written prompt. If you’ve used ChatGPT or Claude, you already know what a prompt is. You type something, the model responds. Simple enough.

But there’s a massive gap between “hey, summarise this for me” and a prompt that works reliably thousands of times without a human babysitting it. Production prompts are engineered, not typed.

The Prompt Structure Framework

A well-structured prompt has five components, and the more precisely you define each one, the more predictable your output becomes.

Types of Prompting

Now, even with a perfectly structured prompt, how you use it matters. There are three main prompting strategies, and each trades off between simplicity and accuracy.

Zero-shot is the simplest. You give the AI a task with no examples and expect it to figure it out. “Translate this sentence to French: The meeting is at 3pm.” Works well when the task is well-defined and the model already knows the pattern.

Few-shot is the next step up. You provide a few examples of input-output pairs so the model understands the exact pattern or format you want. Instead of describing your requirements in words, you show it: “Here are three examples of how I want emails summarised. Now summarise this one.” This is surprisingly effective for getting consistent formatting and tone.

Chain-of-thought is the heavy hitter. Instead of asking for a direct answer, you ask the AI to reason through the problem step by step before concluding. This is what powers “reasoning models” like OpenAI’s o1 or Claude’s extended thinking mode. It trades speed for accuracy, and it’s the go-to for complex analytical tasks where a snap answer would miss nuance.

2. From Casual Use to Production Systems

If you’ve spent any time with ChatGPT or Claude, you’ve probably developed a rhythm: send a message, get an answer you don’t love, tweak your ask, try again, repeat three or four times until the output clicks. That works when you’re the human in the loop, manually steering the conversation.

But in a production system, software needs to do this reliably, automatically, thousands of times a day. There’s no human sitting there hitting “regenerate” or “sending new instructions in a new message”. You can’t afford retries.

The solution is deceptively simple: stop asking one model to do everything in one shot. Instead, figure out the manual workflow first , what steps would a human take to solve this task? Then break those steps into smaller, discrete pieces, and assign each piece to a focused AI agent.

This is the core idea behind agentic AI systems. Instead of one model doing everything and hoping for the best, you decompose the problem into focused steps, each handled by a smaller, cheaper, more reliable agent. The compound effect is a system that’s faster, more predictable, and significantly easier to debug when something goes wrong.

3. Sub-Agents and Model Parameters

Now that you know the “why” behind breaking things down, let’s look at the building blocks.

A sub-agent is an AI model assigned to one specific, narrow task within a larger workflow. One agent extracts data from a PDF invoice. Another validates that data against a database. A third formats and sends a confirmation email. Because each sub-agent does a focused job, you can use smaller, faster, and cheaper models instead of one heavy model doing everything.

But assigning the right model isn’t enough. You also need to tune how the model behaves. The most important dial here is temperature.

Temperature controls how “creative” or “random” a model’s output is. Think of it as a spectrum. At the low end (close to 0), the model plays it safe, it picks the most predictable response every time. Ask the same question twice and you get the same answer. This is what you want for deterministic tasks like extracting data from a document or classifying a support ticket.

At the high end (close to 1), the model takes more risks and explores a wider range of possibilities. The output will vary each time you run it. This is useful for creative tasks like brainstorming, writing, or generating ideas.

The rule of thumb is simple: if the task needs consistency, go low. If the task needs creativity, go higher.

4. Agentic Workflow Paradigms

You have sub-agents. Now you need a way to connect them. There are two main paradigms, and understanding the difference between them is one of the most important architectural decisions you’ll make.

The first is chain-based workflows. This is the simplest pattern: the output of Agent 1 feeds into Agent 2, which feeds into Agent 3, and so on. Linear, predictable, easy to debug. LangChain is the most popular framework for building these. Its key benefit is abstraction → it doesn’t care whether you’re using Claude, GPT-4, or any other model under the hood. Switching providers requires minimal code changes. It also ships with ready-made components for common tasks like connecting to databases, handling memory, and formatting outputs, so you write far less boilerplate.

The second is orchestration-based workflows. This is where things get powerful. Instead of a fixed linear chain, a single Orchestrator Agent sits at the top of the system. You tell it what sub-agents are available and what each one does. When a task comes in, the orchestrator reads it, figures out a plan, and decides which sub-agents to call, in what order, and what to do with their outputs.

The critical difference: orchestration can be cyclic. The orchestrator can call Agent A, send its output to Agent B, get a result back, decide it needs to call Agent A again with new information, and loop until a condition is met. LangGraph is the framework for this. It extends LangChain, and the distinction is exactly this: LangChain is for linear chains, LangGraph is for graph-based workflows that can branch, loop, and route dynamically.

5. Agentic Patterns

Beyond how agents are wired together, there are established patterns for how an individual agent reasons and acts when given a task. Two of the most important ones are ReAct and Plan and Execute.

ReAct (Reasoning and Acting) is a loop. When given a task, the agent doesn’t immediately produce an answer. Instead, it cycles through three steps: Reason (what do I know, what do I still need?), Act (call a tool, fetch data), and Observe (is this enough to answer?). If the answer is no, it loops back to Reason and tries again.

This pattern is powerful because the agent is adaptive. It doesn’t commit to a fixed plan upfront. It responds to what it actually finds at each step, which makes it well suited for tasks where the path to the answer isn’t known in advance.

Plan and Execute takes the opposite approach. Instead of reasoning one step at a time, the agent first builds a complete plan before doing anything. A Planner Agent generates the full step-by-step breakdown, and then an Executor works through that plan sequentially. The advantage is predictability and efficiency, you know the full plan upfront, which makes it easier to parallelise steps, estimate cost, and debug failures. The trade-off is rigidity: if something unexpected comes up mid-execution, the plan may need to be revised.

The decision between the two comes down to the nature of the task. Use ReAct when the task is exploratory or unpredictable and the agent needs to adapt based on what it finds. Use Plan and Execute when the task is well-defined and you want efficiency, parallelism, and a clear audit trail of what was supposed to happen.

6. Context Engineering

Your AI agent can only make good decisions if it has the right information. Context engineering is the discipline of figuring out what information to inject into each prompt, and doing it efficiently.

The naive approach is to dump all user data into every prompt. The problem: prompts get huge, slow, and expensive. AI models charge by the token (roughly by word count), so sending a 50-page document when you only need two paragraphs from it is burning money for no reason.

The smart approach is to dynamically fetch only what’s relevant, right before sending the prompt. There are two main techniques depending on where your data lives.

If the relevant data lives in a structured database (rows and columns), you use tool calling to run a SQL query and pull only the relevant rows. A user asks “what’s my order status?” → the system queries the orders database for that specific user’s recent orders, injects just those rows into the prompt, and the agent answers accurately.

If the relevant data lives in unstructured form (documents, PDFs, notes, emails), you can’t just run a SQL query. This is where RAG (Retrieval-Augmented Generation) comes in. You build a pipeline that breaks all your documents into small chunks, converts those chunks into numerical vectors (a way of representing meaning mathematically), and when a query comes in, finds the chunks that are closest in meaning to the query. The AI sees only the most relevant pieces of your knowledge base, not everything. Check my others post for a detailed guide into RAG.

The key insight here is that context engineering is about precision, not volume. The less irrelevant noise you put in the prompt, the better the agent performs.

7. Harness Engineering

If context engineering is about what information the agent receives, harness engineering is about what capabilities and behaviours the agent is equipped with. Think of it like fishing, the right bait gets you the right output.

The most common tool in harness engineering is Skills. A skill is a markdown file (a simple text file) that describes how the agent should behave in a specific situation. It’s not a prompt for a user task, it’s a behaviour guide embedded in the agent’s system.

For example, an email reply agent might have a file called email-reply-skill.md that specifies: always start with the customer’s name, never promise refunds without checking the policy tool, keep replies under 150 words, and match the tone of the incoming email.

The agent reads this skill file as part of its setup and follows these rules every time it writes an email. Skills make agents more predictable and significantly easier to update → you change a behavior by editing a markdown file, not by rewriting the entire prompt.

Together, these two layers cover the full picture. Context engineering ensures your agent has the right information. Harness engineering ensures it does the right thing with that information.

8. RAG vs Fine-Tuning

These two come up in almost every conversation about making an AI model better at a specific use case, and they’re often confused. They solve fundamentally different problems.

RAG changes the information the model sees. You’re giving a smart person the right reference book before an exam. The model itself doesn’t change, you’re just making sure it has access to the right data at the right time. It’s relatively cheap, you can update the data anytime, and it’s the first thing you should try.

Fine-tuning changes the model’s internal weights, how it thinks. You’re sending that person to school for a year. The model learns to behave differently: a specific tone, a specific format, a specific reasoning style. It’s expensive, requires training compute and labeled data, and the changes are baked into the model.

The rule of thumb: try RAG first. It’s faster, cheaper, and easier to update. Only fine-tune when you’ve confirmed that the model’s behavior or thinking pattern is the bottleneck, not the information it has. If your agent keeps getting the facts wrong, that’s a context problem → use RAG. If your agent gets the facts right but writes in the wrong tone or follows the wrong reasoning pattern, that’s a behavior problem → consider fine-tuning.

9. Tool Calling and MCP

Your AI agent knows a lot from its training data, and you can give it relevant context via the techniques we just covered. But sometimes it needs real-time, external data that neither its training nor your database has.

Say you want an agent to check a creator’s Instagram follower count before making a recommendation. That number changes daily and lives outside your system. This is where tools come in.

You define a tool by giving the agent a simple spec: a name, a description of what it does, what inputs it expects, and what it returns. The agent doesn’t need to know the implementation details, it just needs to know the tool exists and what it’s for. When the agent decides it needs follower data mid-task, it generates the right API call, gets the data back, and uses it in its reasoning.

Tools are essentially a catalog of external capabilities you hand to the agent. The agent decides when and which ones to use.

Now, defining tools manually for every external service gets tedious fast. MCP (Model Context Protocol) is the solution. MCP is an open standard that lets external services publish their tools in a standardised format. Instead of you writing every tool definition, the service hosts an MCP server that your agent connects to. Once connected, the agent automatically discovers all available tools from that server and knows how to use them.

Think of it as a plugin store for AI agents. Companies like Apify, Slack, and Google have published MCP servers. Your agent connects to one endpoint and instantly gains access to dozens of tools without manual integration work.

10. Reliability in Production: The Judge Agent

Getting an agentic system to work once in a demo is easy. Getting it to work correctly thousands of times in production is the hard part. There are two sides to reliability: making sure the output is good, and making sure the system stays up.

The Judge Agent

A judge agent is a separate AI model whose only job is to review the output of your system and decide: does this actually answer the user’s question correctly? It reads both the original input and the final output, then produces a verdict.

There are two ways to deploy a judge. In a sequential setup, every output goes through the judge before reaching the user. If the judge rejects it, a Refiner Agent tries to correct the output, and the loop repeats until it passes. This adds latency but guarantees quality, use it when accuracy is critical.

In a parallel setup, the user gets their answer immediately. The judge reviews it in the background, and if something looks wrong, it fires an alert (typically a Slack message or a review queue entry). Zero added latency for users, while still catching errors for the team. This is the more common production pattern.

Two critical details: first, use a different model for your judge than the one powering your system. A model tends to rate its own outputs favourably. If your system runs on GPT-4, use Claude as the judge, and vice versa. Second, enable extended thinking mode on your judge, instead of a simple pass/fail, the judge reasons through why the output is good or bad, which makes verdicts more accurate and gives your team much more useful debugging information.

The LLM Gateway

Evaluating output quality is one side of reliability. The other side is simpler: is the system even running?

Every AI provider has rate limits. Hit the ceiling and your requests start failing. Providers also go down sometimes. If your entire system is hardwired to one model from one provider, their bad day becomes your bad day.

An LLM Gateway sits in front of all your model calls and acts as a traffic controller. It handles two things: rate limit management (if you’re hitting OpenAI’s limit, it routes overflow to a second API key or a different provider entirely) and failover (if one provider starts failing consistently, it reroutes to another). Your system keeps running. Your users see nothing.

Think of it as a load balancer, but specifically for AI model APIs. Popular open-source options like LiteLLM also give you a unified API, so your code doesn’t need to know which model it’s talking to at any given moment.

11. Memory in Agentic Systems

When a conversation is short and simple, memory isn’t a problem. But what happens when a conversation runs fifty messages deep? Or when an agentic workflow crashes halfway through and needs to resume? Memory management becomes a critical layer in production systems.

The brute force approach is what you’ve already seen in ChatGPT. Every time you send message five, the system quietly packages up your previous four messages and four AI responses and sends all of it to the model again. That’s how the AI “remembers” what was said earlier. It works, but it doesn’t scale. As conversations get longer, you’re sending more tokens on every request. Costs go up, speed goes down, and eventually you hit the model’s context limit.

The smarter approach is to add a Summariser Agent to the system. After every batch of messages (say every ten), the summariser reads the conversation so far and produces a compact memory object: the key facts, decisions, and context, stored as a structured summary. Going forward, instead of sending all fifty past messages, you send the compact summary plus the last few messages in full. The prompt stays lean while preserving the important context.

But memory isn’t just for conversations. In a multi-step agentic workflow, if your system has five sub-agents and crashes at step three, you don’t want to start over from step one. Checkpointing solves this. After each sub-agent completes its task, its output is saved to a persistent store. If the workflow fails mid-run, the retry picks up from the last successful checkpoint. This is both a reliability pattern and a cost-saving one, you don’t waste compute redoing work that already succeeded.

12. Observability: Knowing What Actually Happened

Building an agentic system is one thing. Understanding what it did on any given run is another. Observability is the practice of logging and inspecting every step of an agentic execution, and it’s the difference between “something went wrong” and “I know exactly what went wrong and why.”

The tool most commonly used for this is Langfuse. Every time your agentic system runs, Langfuse records a trace: a detailed log of every step, showing which agent was called, what prompt it received, what the model returned, how long each step took, and how many tokens were used.

This matters for two reasons. First, debugging. When your system produces a wrong output, you don’t have to guess. You open the trace for that run and walk through it step by step. Was the wrong context injected? Did a sub-agent get malformed input? Did the judge agent fire correctly? The trace tells you.

Second, evaluation. When you’re trying to understand whether your system is improving or regressing over time, traces give you the raw evidence. You can compare runs, measure latency across sub-agents, and identify which step is the bottleneck.

Think of observability as your system’s black box recorder. You hope you don’t need it, but when something goes wrong, it’s the first place you look.

13. Human in the Loop (HITL)

Not every decision should be made entirely by an AI agent. HITL is the design pattern of intentionally building moments into your workflow where a human reviews, approves, or redirects before the system proceeds.

The simplest form is a chat interface. After every AI response, the human can react: ask for a revision, correct something, or confirm before the next step begins. Simple, effective, but not always the best UX.

The smarter approach is purpose-built interfaces. Consider an AI system that generates a webpage. A naive HITL design would be: show the page, let the user describe what they don’t like in a chat box, regenerate the whole thing. That’s slow and frustrating. A better approach is a direct editing interface where the user can click on any element, change an image, tweak the copy, adjust the layout → without going back to the AI for minor fixes. The AI handles the heavy lifting, the human handles the fine-tuning through purpose-built controls.

HITL also serves as a safety valve. Remember the judge agent from earlier? If the judge is uncertain about an output, rather than sending it to the user or retrying automatically, it can route the output to a human reviewer with a note: “I’m not confident this email should be sent. Can you review before it goes out?” This is especially important for irreversible actions, sending emails, making payments, deleting records, posting publicly. Any action you can’t undo is a good candidate for a human checkpoint.

Good HITL design is really a UX problem: what’s the minimum friction way to let a human correct or guide the system at the right moments?

14. Structured Outputs

LLMs produce free-form text by default. That’s fine when the output is meant for a human to read. But in a production agentic system, the output of one agent is usually the input of the next. If Agent 2 expects a specific JSON format and Agent 1 returns something slightly different, the whole pipeline breaks.

Structured outputs solve this. Instead of letting the agent return whatever text it wants, you define an exact schema: a data structure that specifies which fields must be present and what type each field should be. For example, an invoice extraction agent might be required to return a JSON object with vendor_name (string), invoice_date (date), total_amount (number), and line_items (array). Every time the agent runs, its output is validated against this schema the moment it arrives, before it touches anything downstream.

This matters because without schema validation, a missing field might not cause an error until three steps later in the pipeline, at which point tracing it back to the original agent is painful. Catching it early makes errors obvious, fast to fix, and easy to log. Most LLM providers now support structured output or JSON mode natively, and frameworks like LangChain have built-in support for defining and enforcing output schemas.

15. Putting It All Together

Here’s how a production agentic system actually looks when all of these pieces combine. Every layer solves a specific problem: prompting ensures each agent gets clear instructions, sub-agents break the work into focused pieces, context engineering ensures agents have the right information, harness engineering ensures they behave consistently, tool calling and MCP connect them to the outside world, structured outputs keep the pipeline stable, judge agents guard quality, memory keeps context alive, observability lets you debug, and HITL keeps humans in control where it matters.

That’s the full picture. None of these layers exist in isolation, they’re designed to work together. Context engineering feeds the orchestrator the right information. The orchestrator, equipped with skills, routes work to focused sub-agents. Those agents use tools and MCP to reach the outside world. Structured outputs keep the data flowing cleanly between them. Judge agents catch errors before they reach the user. Memory keeps context alive across long conversations and multi-step workflows. Observability logs everything so you can debug and improve. And HITL keeps humans in the loop where the stakes are highest.

A new AI tool or framework launches every other day, but the fundamentals don’t change that fast. If you understand these concepts, you have the mental model to evaluate anything new that comes along.

If you found this useful, I write about AI engineering and system design on Substack. Subscribe!

Experiments with AI

Curious learner about this ever changing space, I like to document and publish my learnings.

By Utkarsh Umang

1. What is RAG?

RAG stands for Retrieval Augmented Generation. Let’s unpack what that actually means in plain terms.

Think of a regular LLM as a very well-read person who studied everything on the internet up until a certain date, then went into a room with no phone, no books, and no internet. Smart, but limited to what they already know.
RAG gives that person a library they can look things up in before answering. Here’s how the three parts map to that:

Retrieval is the act of going to the library and fetching the relevant pages. Technically, this means querying a knowledge base, which is where your documents live.

Augmented means the user’s question doesn’t go to the LLM alone. It goes with the retrieved pages attached. The LLM now sees: “Here’s what the user asked, and here’s the relevant context from the knowledge base. Now answer.”

Generation is the final step: the LLM reads everything and produces an answer grounded in your actual data.

Why do we need RAG?

LLMs have two core limitations when it comes to working with your data:

First, they have a knowledge cutoff. Anything that happened after they were trained, they simply don’t know about.
Second, they have a context window limit. You can’t just paste your entire company wiki into a prompt and expect it to work. Even models with large context windows get slower, more expensive, and less accurate as the context grows.

RAG solves both. Instead of stuffing everything into the prompt, you only retrieve what’s relevant to the current question and pass that in. It’s precise, it’s efficient, and it works with documents the model has never seen, your internal policies, product docs, support tickets, anything.

RAG vs fine-tuning

This is the most common question when people first encounter RAG: “Why not just fine-tune the model on my data?”

The distinction comes down to what you’re actually trying to change.

Fine-tuning changes how the model behaves. It’s the right tool when you want the model to adopt a specific tone, follow a certain format, or develop a skill it didn’t have before. For example, training a model to always respond like a customer support agent for your brand.

RAG changes what the model knows. It’s the right tool when the model’s behavior is fine, but it needs access to data it wasn’t trained on. For example, answering questions about a document uploaded by a user five minutes ago.

A useful mental model: fine-tuning is retraining the chef, RAG is handing them a recipe card before they cook.

2. The two pipelines of a RAG system

Every RAG system, no matter how simple or complex, is built on two pipelines. Understanding this split is the key to understanding how the whole thing works.

The first is the ingestion pipeline: this is where you prepare your knowledge base and store it in a way that can be searched efficiently. You run this once upfront, and again whenever your data changes.

The second is the retrieval pipeline: this runs every single time a user asks a question. It fetches the relevant context and hands it to the LLM to generate an answer.

A good analogy is a library. Ingestion is the process of acquiring books, cataloguing them, and putting them on shelves in an organized way. Retrieval is what happens when someone walks in and asks a librarian a question. The librarian goes to the right shelf, pulls the relevant pages, and uses them to answer.

3. The ingestion pipeline

The ingestion pipeline is where your raw data gets transformed into something a RAG system can actually search. If this step is done poorly, no amount of clever retrieval logic will save you. Garbage in, garbage out.

It has four steps: source, chunking, embedding, and storing. Let’s go through each.

Step 1: Source (knowledge base)

This is your raw data. It could be a folder of PDFs, a Notion workspace, a database, customer support tickets, product documentation, anything. The knowledge base is everything you want your RAG system to be able to answer questions about.

The important thing to recognize here is that this data hasn’t been processed yet. It’s just raw text in various formats, potentially hundreds of thousands of tokens worth of it. You can’t pass it to an LLM as-is, which is why the next step exists.

Step 2: Chunking

Chunking is the process of splitting your raw text into smaller, manageable pieces. You’re essentially deciding: “How big should each unit of searchable information be?”

A common default is 1,000 tokens per chunk, but you can tune this up or down depending on your use case.

There’s also a concept called chunk overlap: the last N characters of one chunk are repeated at the start of the next. This exists to prevent a sentence from being cut mid-thought at a chunk boundary, losing its meaning in the process. Think of it like a sliding window moving across your document.

Chunking sounds simple but it’s one of the most consequential decisions in your entire pipeline. Here’s why it can go wrong:

• Chunks too small: You preserve precision but lose context. A chunk that says “it increased by 23%” is meaningless without knowing what “it” refers to.

• Chunks too large: You retrieve too much noise along with the relevant information, which dilutes the LLM’s answer.

• Poor boundaries: Splitting mid-paragraph or mid-table breaks the logical flow of information.

• No structural awareness: A naive splitter doesn’t know the difference between a heading, a code block, and body text. It just cuts at character counts, which often produces nonsensical chunks.

This is why there are multiple chunking strategies, and choosing the right one for your document type matters more than most people initially realise.

Now let’s look at the different chunking strategies available and when to use each.

Chunking strategies:

Not all text is structured the same way, so there’s no single chunking strategy that works everywhere. Here are the five main approaches, roughly ordered from simplest to most sophisticated:

Character text splitter is the most basic approach. It splits text at a separator character (a double newline by default), then combines the resulting pieces until they fill up the chunk size limit. If a single piece is already larger than the chunk size, it’s kept as-is. It’s fast and cheap, but completely ignores the meaning of what it’s splitting.

Recursive text splitter is an upgrade on the above. Instead of one separator, you give it a priority list: try splitting by paragraph first, then by sentence, then by word if needed. If a chunk is still too large after the first split, it recursively applies the next separator. This is the default in most RAG frameworks like LangChain for a reason: it respects document structure without being expensive.

Document-specific splitting goes one step further by understanding the file format itself. A PDF splitter knows about pages and columns. A Markdown splitter knows about headers and code blocks. An Excel splitter knows about rows and sheets. Use this when your source documents have rich structure that a generic splitter would destroy.

Semantic chunking is where it gets interesting. Instead of splitting by characters or structure, it splits by meaning. Here’s how it works: every sentence gets encoded into a vector, then sentences are compared against each other using cosine similarity. Sentences above a similarity threshold (commonly the 70th percentile) get grouped into the same chunk. Sentences that diverge in meaning start a new chunk. The result is chunks that are semantically coherent rather than just the right size. The tradeoff is cost: you’re running an embedding model during ingestion, not just doing string operations.

Agentic chunking is the most accurate and the most expensive. You pass the raw text to an LLM with a prompt that says “divide this into logical chunks.” The LLM reads the content with full comprehension and makes chunking decisions the way a human editor would. Reserved for high-stakes use cases where quality matters more than cost.

In practice, production pipelines rarely use any of these in isolation. Libraries like unstructured.io combine multiple strategies under the hood depending on what type of content they’re processing.

Step 3: Embedder

Once your chunks are ready, the next step is to convert each one into a vector, a list of numbers that captures the semantic meaning of that text.

Here’s the intuition: words and phrases that mean similar things end up close to each other in vector space. “Dog” and “puppy” will have vectors that point in nearly the same direction. “Dog” and “quarterly earnings” will not. This is how the retrieval step finds relevant chunks later: it converts the user’s query into a vector and looks for chunks whose vectors are nearby.

The quality of your embedder directly affects the quality of your retrieval. The MTEB leaderboard is the standard benchmark for comparing embedding models. As a reference, OpenAI’s two main options are:

• text-embedding-3-small: 1,536 dimensions by default, configurable to 512 or 1,024. Good balance of quality and cost.

• text-embedding-3-large: 3,072 dimensions by default. More expressive, captures finer semantic nuance, but more expensive.

More dimensions generally means richer representation, but it also means more storage and slower search. For most applications, text-embedding-3-small is sufficient to start.

Step 4: Vector DB

The final step of ingestion is storing your vectors somewhere they can be searched at query time. This is the vector database.

Choosing the right one mostly comes down to your infrastructure preferences and how far along you are in building:

• Pinecone: Fully managed, API-based. Zero infrastructure to maintain. Good choice if you want to move fast and don’t want to think about ops.

• Qdrant: Open source and self-hostable. Good if you want control over your data and are comfortable running your own infrastructure.

• pgvector: A Postgres extension that adds vector search to your existing database. The best choice if you’re already on Postgres and want to keep your stack simple.

• ChromaDB: Extremely easy to set up, runs locally. The go-to for prototyping and experimentation.

• FAISS: A Meta library for efficient similarity search. Lightweight, runs in-memory, great for local development or when you need raw speed without a database server.

With step 4 complete, ingestion is done. Your knowledge base is now chunked, embedded, and stored, ready to be searched.

4. The retrieval pipeline

Ingestion was the one-time setup. The retrieval pipeline is what runs live, every single time a user asks a question. It has three steps.

Step 1: Query

The user types a question in natural language. The very first thing the system does is convert that question into a vector, using the exact same embedding model that was used during ingestion.

This point is easy to miss but critical: it has to be the same model. Your chunks were embedded into a specific vector space. Your query needs to land in that same space for similarity search to make sense. Using a different model is like translating a sentence into French to compare it with text in German and expecting the words to align.

Step 2: Retriever

Now that the query is a vector, the retriever takes it and searches the vector DB for the chunks whose vectors are closest to it. The closeness is measured using cosine similarity, which looks at the angle between two vectors:

cosine similarity = (A · B) / (|A| × |B|)

The smaller the angle between two vectors, the more semantically similar the underlying text is. With modern normalized embedding models, the denominator is always 1, so similarity essentially reduces to a dot product between the two vectors.

The retriever returns the top-k most similar chunks. These are the pieces of your knowledge base most likely to contain the answer to the user’s question.

Step 3: LLM generation

The retrieved chunks are combined with the original user query and passed to the LLM as a single prompt. The structure looks something like:

“Here is some context: [chunk 1] [chunk 2] [chunk 3]. Using this context, answer the following question: [user query].”

The LLM now has everything it needs: the question and the relevant information to answer it. This is the augmentation step from our earlier definition, now made concrete. The model generates a response grounded in your actual data rather than just its training weights.

5. History aware retrieval

The three-step retrieval pipeline works well for a single question. But real users don’t ask one question and stop. They have conversations.

Consider this exchange:

User: “What is RAG?”
System: explains RAG
User: “How is it different from fine-tuning?”

That second question, sent to a basic RAG system, is ambiguous. “It” refers to RAG, but the retriever has no idea. It sees the words “different from fine-tuning” and may retrieve completely irrelevant chunks. The conversation history exists in the UI, but the retrieval system is stateless.

History-aware retrieval solves this by adding a query rewriting step before retrieval. Here’s how it works:

The system stores the full conversation, every question and every answer. When a new question comes in, it doesn’t go straight to the retriever. Instead, the system passes the chat history along with the new question to the LLM and asks it to rewrite the query into a fully self-contained standalone question.

So “How is it different from fine-tuning?” becomes “How is RAG different from fine-tuning?” before it ever reaches the retriever. Now the retrieval is accurate regardless of how many turns deep the conversation is.

Once the answer comes back, the new question and answer are appended to the chat history, and the loop continues.

It adds one LLM call per query, but it’s a small cost for a significant improvement in multi-turn accuracy.

6. Production ingestion pipeline using unstructured.io

Everything we covered in the ingestion pipeline so far was conceptually clean: take your documents, chunk them, embed them, store them. In practice, it’s messier.

Real-world documents are not plain text files. A company’s knowledge base might include PDFs with embedded tables, Word docs with inconsistent formatting, slide decks, spreadsheets, and scanned images. A plain text splitter handed a PDF with a financial table will produce garbage chunks that destroy the meaning of that table entirely.

This is the problem unstructured.io solves. It’s an ETL (extract, transform, load) library built specifically for the chaos of real-world documents. It handles the complexity of mixed content types so your chunking and embedding steps receive clean, structured input. Here’s what a production ingestion pipeline looks like when built with it.

Step 1: File service

Raw documents are uploaded to a file service before any processing begins. S3 is the standard choice here. This gives you a durable, scalable store for source files that’s decoupled from your processing infrastructure, so you can reprocess documents at any time without needing to re-upload them.

Step 2: Queue

Document processing is slow. A large PDF can take several seconds to partition, chunk, and embed. In a production system you never want that blocking a request thread.

Instead, each uploaded file drops a message into a queue. Workers pick up jobs from the queue and process them asynchronously. This keeps ingestion non-blocking and makes the system resilient: if a worker crashes mid-job, the message stays in the queue and gets retried.

Step 3: Partitioning

This is where unstructured.io earns its place. Partitioning is the step that breaks a complex document into its atomic elements: paragraphs of text, tables, images, headers, captions, footers.

This is fundamentally different from chunking. Chunking is about size. Partitioning is about structure. Unstructured reads the document layout first, identifies what type of content each section is, and classifies it before any splitting happens. A table is recognised as a table, not just “some text with a lot of whitespace.” An image is extracted as an image, not skipped.

Step 4: Chunking

Once the document is partitioned into atomic elements, unstructured applies a chunking strategy called “chunk by title.” It keeps all content under the same heading together as a unit, which preserves the semantic relationship between a heading and the paragraphs that follow it.

Images and tables get special treatment here because they can’t just be chunked as text:

• Images are stored as base64 encoded data, but the original image file is also retained.

• Tables are stored as text chunks, but the original table structure is also retained.

This distinction matters for the next step.

Step 5: Agentic chunking for multimodal chunks

For chunks that contain multimodal content like tables or images, unstructured uses agentic chunking. An LLM is asked to generate a natural language summary of what the chunk contains.

The reason for this split is that retrieval and generation have different requirements. Retrieval needs something compact and semantic, a dense summary that embeds well and surfaces reliably when a user asks a related question. Generation needs the full original content, the actual table with all its rows and values, or the image with all its detail.

So the pipeline stores both, and in LangChain’s document model this maps cleanly to two fields:

• page_content stores the LLM-generated summary, used for embedding and retrieval

• metadata stores the original content, passed to the LLM at generation time

The result is a pipeline that retrieves accurately and generates completely, even when your knowledge base is full of charts, tables, and images.

7. Advanced retrieval techniques

Basic retrieval gets you surprisingly far: embed the query, find the most similar chunks, pass them to the LLM. For simple use cases it works fine. But as your knowledge base grows and your users ask more complex questions, the cracks start to show. You get irrelevant chunks, redundant context, and queries that miss relevant information simply because of how they were phrased.

Here are four techniques that meaningfully improve retrieval accuracy in production.

Scored retrieval

The simplest upgrade and the one you should add first.

By default, a retriever returns the top-k chunks regardless of how similar they actually are. If a user asks about your refund policy and your knowledge base doesn’t contain anything about it, basic retrieval will still return the top-k chunks, they’ll just be loosely related noise. The LLM then tries to answer using that noise and either hallucinates or produces a vague, hedged response.

Scored retrieval adds a minimum similarity threshold. Any chunk that scores below it gets dropped, even if it was technically in the top-k. The most common threshold is 0.3. It’s a small change that prevents low-quality context from polluting the prompt and nudging the LLM toward bad answers.

MMR : Max Marginal Relevance

Scored retrieval filters out irrelevant chunks. MMR solves a different problem: redundancy.

Imagine asking “What are the benefits of RAG?” and your top-5 retrieved chunks are all slight variations of the same paragraph from five different pages of the same document. They’re all relevant, but they’re all saying the same thing. You’ve wasted most of your context window passing duplicate information to the LLM.

MMR balances relevance with diversity. It retrieves a larger candidate set first, then selects the final chunks by iteratively picking the one that is both relevant to the query and maximally different from what’s already been selected. Three parameters control this:

• top_k: how many chunks to ultimately send to the LLM

• fetch_k: how many chunks to retrieve as the initial candidate pool before filtering

• lambda_mult: the diversity dial. 0 means maximum diversity, 1 means maximum relevance (equivalent to standard retrieval)

One caveat: MMR is not the right choice when you need precise, factual answers where every piece of context must be directly on-point. Introducing diversity can occasionally pull in chunks that are semantically interesting but not strictly relevant to the question.

For factual Q&A, stick with scored retrieval.
For broader research-style queries, MMR shines.

Multi-query retrieval

Vector search is sensitive to phrasing. A user asking “How do I reduce LLM hallucinations?” might miss chunks that were written around “improving factual accuracy” or “grounding model outputs,” even though they cover the exact same concept.

Multi-query retrieval addresses this by using the LLM to generate several different phrasings of the original question before running retrieval. Each variation gets its own retrieval pass, and the results are merged into a single deduplicated set of chunks. The LLM then generates an answer from this richer, more comprehensive context.

It costs more, one extra LLM call upfront plus multiple retrieval passes, but for complex or ambiguous queries it can significantly improve recall.

Reciprocal Rank Fusion (RRF)

Multi-query retrieval creates a new problem: you now have multiple ranked lists of chunks coming back from different query variations, and you need a principled way to combine them into one final ranking.

For example if you generated 5 queries from 1 user query and fetched 5 chunks with each generated query, you now have 25 chunks, these 25 chunks need not be unique and you can only pass top 5 chunks to the LLM for generation. A reranker helps here by identifying the top 5 relevant chunks from this set of 25 chunks.

RRF is the standard solution. For each chunk, it looks at its rank position across all the retrieval results it appeared in and computes a combined score:

RRF score = Σ 1 / (k + rank)

A chunk that ranked highly across multiple query variations ends up with a high RRF score. A chunk that only appeared in one result list, or ranked low, gets a lower score. The constant k is typically set to 60, which softens the penalty for lower-ranked chunks so that a chunk ranked 10th doesn’t get completely written off just because another chunk ranked 1st.

RRF is retrieval-method agnostic, which makes it especially useful in hybrid search setups where you’re combining results from both vector search and keyword search, which we’ll cover next.

8. Hybrid search

Semantic search is powerful, but it has a blind spot: precision.

Ask a semantic search system for “invoice INV-2024–00892” and it might return chunks about invoicing processes, billing cycles, or payment terms. Conceptually related, but completely wrong. Semantic search looks for meaning, and meaning doesn’t help you when the user is looking for an exact string.

This is where keyword search fills the gap. And hybrid search is the approach that combines both, using each where it’s strongest.

The tradeoff looks like this: semantic search understands intent and handles paraphrasing well, but struggles with exact matches. Keyword search is precise and reliable for specific terms, but falls apart when a user phrases their query differently from how the document was written. Hybrid search gets you both.

BM25 : Best Matching 25

The most widely used keyword search algorithm in RAG systems is BM25. It scores chunks based on two complementary signals:

TF (Term Frequency) measures how often the searched term appears in a given chunk. A chunk that mentions “pgvector” five times is likely more about pgvector than one that mentions it once in passing.

IDF (Inverse Document Frequency) corrects for common words. If a term appears in almost every chunk in your knowledge base, finding it doesn’t tell you much. IDF down-weights frequent terms and up-weights rare ones, so a search for a specific product code or a niche technical term gets ranked above chunks that just happen to contain common words like “the” or “data.”

Together, TF and IDF give BM25 its precision: it finds chunks where the exact term appears and ranks them higher when that term is meaningfully rare across the knowledge base.

Ensemble retriever

To combine semantic and keyword search, you use an ensemble retriever. It runs both retrievers in parallel and merges their results using RRF, which we covered in the previous section.

What makes the ensemble retriever flexible is weighting. You can control how much you want to lean toward semantic results versus keyword results. A knowledge base full of technical documentation with specific product codes might warrant a higher keyword weight. A conversational FAQ might lean more semantic. The weights feed directly into the RRF scoring as numerators, so tuning them shifts which retriever has more influence over the final ranking.

In production

A production-grade retrieval setup combines everything from the last two sections: multiple query variants, run across both semantic and keyword retrievers, with RRF merging the results.

Even then, you’re often left with more chunks than you want to pass to the LLM. Running multi-query hybrid search can surface 20 to 30 candidate chunks. Passing all of them increases cost, latency, and the risk of noisy context degrading the answer quality.

This is where reranking comes in: a final filtering step that scores the merged candidates more carefully and cuts the list down to the top 5 or top 10 before anything reaches the LLM. We’ll cover that in the next section.

9. Reranking

After hybrid search and RRF, you might still have 20 to 30 candidate chunks. Passing all of them to the LLM would bloat the context window, increase cost and latency, and reintroduce the exact noise problem RAG was designed to solve. In production you want to pass the top 5 to 10 chunks, no more.

But how do you decide which 5 to 10 are the best ones? The similarity scores from vector search are good approximations, not precise relevance scores. This is where a reranker comes in.

A reranker is a model that sits between the retriever and the LLM. It takes every candidate chunk, scores it carefully against the query, and reorders the list so you can confidently cut it at the top.

The two stage process

It helps to think of the full retrieval pipeline as two distinct stages with different jobs:

Stage 1, embeddings: fast and broad. You cast a wide net using vector similarity. This is intentionally an approximation: you’re finding chunks that are roughly in the right direction, not necessarily the most relevant ones. Speed is the priority here.

Stage 2, reranker: slow and precise. For each chunk returned in stage 1, the reranker takes the query and the chunk together, encodes them jointly, and produces a precise relevance score. This is repeated for every candidate, then the list is reordered by these scores and trimmed.

The two-stage design is deliberate. Running a precise reranker over your entire vector DB on every query would be too slow. Running only fast vector search gives you speed but imprecision. Together, you get the best of both.

Bi-encoder vs cross-encoder

Understanding this distinction explains why reranking is more accurate than embedding alone.

An embedding model is a bi-encoder. It encodes the query into a vector and each chunk into a vector separately, then compares them after the fact. It’s fast because the chunk vectors can be precomputed and stored. But encoding them separately means the model never gets to see how the query and chunk interact with each other. It’s comparing two things in isolation.

A reranker is a cross-encoder. It takes the query and the chunk concatenated together as a single input and encodes them jointly. Because the model sees both at the same time, it can capture much richer signals: does the chunk directly answer the question, or just share some keywords? Is the relevance explicit or just implied? This joint encoding produces a relevance score that’s meaningfully more accurate than cosine similarity.

The tradeoff is that cross-encoders can’t precompute anything. Every query requires a fresh pass over every candidate chunk, which is why you use the bi-encoder to filter down to a manageable set first. Cohere’s reranker is one of the most widely used options in production.

Where to go from here?

This post covered the full architecture of a RAG system, from the ingestion pipeline and chunking strategies to hybrid search, RRF, and reranking. By now you should have a solid mental model of how a production RAG system is actually built.

But building the pipeline is only half the story. Once it’s running, a new set of questions takes over:

What actually breaks in production, and how do you diagnose it? How do you ensure reliability when your knowledge base changes? What does hallucination look like in a RAG system specifically, and how do you reduce it? How do you evaluate whether your RAG system is good, not just working?

These are the questions that separate a RAG proof-of-concept from a production system you can trust.

I’ll be covering each of them in the next part of this series.

Subscribe now