How to Make AI Answer Questions About Your Documents: A Practical RAG Guide

A 2,000-page handbook cannot fit into a model's context window in a usable way. Even when it fits, models lose attention to material in the middle of long inputs and may answer confidently from the wrong section. The solution is a step before generation: find the few paragraphs that answer the question and pass only those to the model. That step is retrieval, and the full technique is Retrieval-Augmented Generation: search first, then generate.

Retrieval means locating relevant information, much like checking a textbook index before reading a chapter. Augmentation means adding that retrieved text to the prompt, supplementing the model's built-in knowledge with specific context. Generation means the model writes its answer while the retrieved evidence sits in the conversation, so the answer is grounded in the actual data rather than recalled from training. In one sentence: find the right chunks, put them in the prompt, and let the model answer from them.

Search tools such as Elasticsearch, Kendra, and SharePoint search have located relevant passages for decades. The retrieval step is not new. What RAG adds is a language model at the end of the pipeline, which reads the retrieved evidence and writes an answer instead of returning a results page for the user to read. RAG can be understood as enterprise search with a language model at the end.

The example uses PineRidge Solutions, a fictional company whose six onboarding documents cover the employee handbook, benefits, leave policy, expenses, engineering onboarding, and IT security. The system should answer questions like 'how many vacation days do I get?' by finding the right section. The build uses Kiro IDE and Amazon Bedrock, but the same pattern works with any embeddings model, local or cloud, such as Ollama, OpenAI, or Cohere. The full code is in the author's GitHub repository.

Documents are split into paragraph-based chunks so that retrieval returns a relevant section rather than an entire file. Chunk size involves a trade-off: too large reintroduces the context problem, too small can cut an answer in half. Overlap matters: if a sentence like 'These days do not carry over' is split from the sentence defining which days, the chunk loses its referent. Including one paragraph of overlap from the preceding chunk keeps each chunk self-contained. The chunking function loops over markdown files, splits on blank lines, and stores each chunk's text with its source filename for citations. The example yields roughly 150 chunks.

An embedding converts text into a vector, a list of numbers placed in a space where distance reflects meaning. Similar chunks land near each other even when their wording differs, so 'parental leave top-up' and 'salary during maternity leave' are numerically close. The embeddings model is separate from the generation model: it is small and fast and only produces searchable numbers. The example calls Amazon Titan Embeddings V2 through Bedrock to produce 1024-dimensional vectors. For this demo the vectors are cached to a local file; production systems can use Amazon S3 Vectors, OpenSearch Serverless, pgvector, or Bedrock Knowledge Bases.

At query time the question is embedded with the same model, placing it in the same number space as the chunks. The retrieve function computes cosine similarity between the question vector and every chunk vector, where 1 means identical and 0 means unrelated, sorts the scores, and returns the top three chunks. This is semantic search: a query about 'working from home rules' matches a handbook section titled 'remote work policy' because the meanings are close.

The generation function labels each retrieved chunk with its source file, builds a prompt instructing the model to use only the supplied context and to say so if the answer is absent, and calls Claude through Bedrock's Converse API. In the example, asking about the RRSP matching policy returned a grounded answer: dollar-for-dollar matching up to 5% of base salary, starting after 90 days, with a vesting schedule, all traceable to specific chunks that serve as citations. The pipeline is chunk, embed, retrieve, generate, in about 60 lines of Python.

Retrieval can succeed while the answer fails. Asking 'How many vacation days do I get as a senior engineer?' retrieved the correct vacation table, but the model could not map 'senior engineer' to a job level, because that mapping lived in a different chunk. The model was not hallucinating; it honestly reported missing evidence. When a RAG system gives a bad answer, the first question is which chunks were retrieved, not why the model is wrong.

For newcomers: RAG lets AI answer questions about your documents without pasting them into a chat. It searches first, then answers from what it finds, in three steps of chunking, embedding, and retrieval, and the model never sees the full document. For builders: RAG is a pipeline whose chunking strategy, embedding model, retrieval method, and generation model each independently affect quality, and different models can serve different jobs, such as Titan for fast cheap search and Claude for generation. The next post addresses chunks that are too small, answers spread across sections, and retrieval that succeeds while answers still fail, covering the strategies that make retrieval reliable.