RAG Implementation: 10 Lessons a Yellow Engineer Learned Building Retrieval-Augmented AI Systems

Today, we are interviewing Ivan, a software development engineer who has been building AI features at Yellow for the last four years, and a big chunk of that time went into retrieval-augmented systems: internal knowledge assistants, document search tools, support copilots, and a few projects that taught him things the hard way.

We’ll discuss some of the practical and sometimes annoying lessons he picked up while building retrieval-augmented generation systems that hold up in production.

Document Preprocessing That Improves RAG Retrieval

“What is one document preprocessing step that significantly improved your RAG system's retrieval accuracy? Why do you think it made such a difference?”

— Cleaning up document structure before chunking. We had a knowledge base full of PDFs exported from older systems, and they came with broken layouts, footnotes bleeding into paragraphs, that kind of mess. Early on, we just dumped the raw text in and hoped the embeddings would sort it out. They didn't. Retrieval kept pulling fragments that were technically related but practically useless.

So we built a preprocessing step that rebuilt the logical structure: detect real headings, keep section context attached to the text under it, strip the junk. Retrieval accuracy jumped more than any model change we tried that month.

“What preprocessing step turned out to be unnecessary or overengineered in your case?”

— Aggressive text normalization. We spent a week building this elaborate pipeline that lowercased everything, expanded abbreviations, removed special characters, normalized whitespace into oblivion. Felt very thorough, but it barely moved the needle, and in a couple of cases, it actually hurt. Stripping out things like product codes and casing removed the signal that the embedding model was happily using. We rolled most of it back. These days, I keep normalization light unless I have a clear reason not to.

Chunking Strategies That Work Better Than Defaults

“Can you share one chunking strategy that worked better than standard approaches for your use case? What characteristics of your data influenced this?”

— Fixed-size chunking with overlap is the usual default, and for the data on one of the projects, it was mediocre. Our documents were mostly structured policy and procedure content, so we switched to section-aware chunking. Each chunk respects a logical boundary, like a full subsection, and only splits further if it gets too long.

The reason it worked comes down to how people ask questions. Users asked about the whole procedure, not random sentence fragments. When a chunk contained a complete idea, the answer quality went up noticeably. Among the RAG chunking strategies we tested, this was the clear winner for our content.

“What was one chunking mistake that significantly hurt performance early on?”

— Chunks that were too small. We thought tiny chunks would make retrieval more precise. Instead, they killed context. A user would ask a question, the system would retrieve a fragment that mentioned the right keyword, but the surrounding explanation lived in a different chunk that never got pulled. We doubled the chunk size, added overlap, and a lot of half-correct responses disappeared.

Choosing the Embedding Model

“What is one challenge you faced with embedding model selection for your RAG system?”

— Benchmarks lie. Or rather, they don't lie, they just don't describe your data.

On one of our projects, we picked an early model partly because it topped a public leaderboard. On our actual domain content, it underperformed a smaller, less hyped model. The benchmark documents looked nothing like the client’s internal jargon-heavy material. That gap between leaderboard performance and real performance caught me off guard, and it's the main challenge with embedding models for RAG. You can't outsource the decision to a ranking someone else made on someone else's data.

Source: Market.us Scoop

“How did you evaluate and choose the right model eventually?”

— We built a small evaluation set from real user questions paired with the documents that should answer them. Maybe 150 query-document pairs, labeled by hand. Then we ran each candidate model against that set and measured how often the right document showed up in the top results. So, test on your own data. That's the whole tip.

Handling Irrelevant Retrieved Context

“How do you handle a scenario where your RAG system retrieves irrelevant context? What filtering or re-ranking approach do you use?”

— We add a re-ranking step after the initial retrieval. The vector search pulls a wider set of candidates, say the top twenty, then a re-ranker scores each one against the actual query and keeps only the strongest few. This can catch a lot of near-misses. We also set a score threshold, so if nothing clears the bar, we'd rather pass less context than feed the model noise.

“What is the most common cause of irrelevant retrieval?”

— Confusing queries, that’s for sure! When someone asks something vague like "how do I update it," the system has no idea what "it" refers to, so it grabs whatever looks statistically close. The fix isn't always better retrieval. Sometimes it's asking the user what they really mean.

Keeping RAG Knowledge Fresh

“What is one technique you've used to improve the freshness of information in your RAG system?”

— Incremental re-indexing tied to the source systems. Instead of rebuilding the whole index on a schedule, we listen for document changes and re-embed only what changed. When a policy doc updates, that section gets re-chunked and re-indexed within minutes.

“How do you detect when your knowledge base is outdated? How do you manage document updates?”

— We attach metadata to every chunk: source, last-updated timestamp, version. That lets us flag content that hasn't changed in a long time and surface it for review.

For updates, we treat the source system as authoritative. The RAG index never becomes its own source of truth, which is a mistake I've seen teams make. The moment your index and your real documents disagree, you've got two versions of reality and users trusting the wrong one.

Optimizing Vector Search Performance

“Can you describe one way you've optimized vector search performance in your RAG implementation?”

— We once switched the index to an approximate nearest neighbor setup with tuned parameters instead of exact search. Exact search was accurate and slow. Painfully slow once the collection grew.

The trick with vector search optimization is that you trade a tiny bit of recall for a large speed gain, and for most RAG use cases, that trade is worth it. We tuned the index parameters against our own evaluation set, so we'd know exactly how much recall we were giving up. It wasn't much.

“What was your biggest bottleneck in scaling vector search?”

— Memory, honestly. As the index grew, keeping everything in RAM became expensive. We hit a point where the box just couldn't fit it. We solved it partly with quantization, which shrinks the vectors with a small accuracy cost, and partly by being more disciplined about what we even indexed. A lot of content didn't need to be searchable. Cutting that down helped as much as any clever optimization. Sometimes the best scaling move is indexing less.

Hybrid Search Approaches

“What is one hybrid search approach you've tried that combines vector and keyword search?”

— We ran vector search and keyword search in parallel, then merged the results with a weighted scoring step. Vector handles meaning, keyword handles exact terms. A classic hybrid search RAG setup, and it earned its place.

Source: Precedence Research

“What advantages did this provide over pure vector retrieval?”

— Exact terms stopped slipping through the cracks. When a user searched for a specific product code or an error string, vector search alone sometimes wandered off toward semantically similar but wrong results. Keyword search nailed those instantly.

Evaluating RAG Systems Beyond Retrieval Metrics

“What is one evaluation metric beyond standard retrieval metrics that you use to assess your RAG system's quality?”

— Faithfulness. Basically, does the generated answer actually stick to the retrieved context, or is it just improvising? We score a sample of answers on whether every claim traces back to a retrieved source. Retrieval metrics tell you whether the right documents showed up. They say nothing about whether the model used them honestly. That's the gap faithfulness fills, and it's one of the RAG evaluation metrics I now trust most.

“What is one misleading metric you initially relied on?”

— Raw retrieval recall at a high cutoff. The right document can show up in the top ten almost every time. The problem is that the model doesn't read all ten with equal attention, and stuffing ten chunks into the prompt diluted the good context with mediocre context. High recall can hide a quality problem. Once you look at end-to-end answer quality instead, the picture gets more reliable.

Prompt Design for RAG Systems

“Can you share one lesson you learned about prompt design specifically for RAG systems? How does it differ from prompting without retrieval?”

— Be explicit about where the answer should come from. With plain prompting, you mostly shape tone and task, but with retrieval, you also have to tell the model to treat the provided context as the source and not its own memory. You can see clear gains just from instructing the model to answer only from the supplied context and to say it didn't know if the context was missing. Good RAG system design leans heavily on these boundaries.

“How do you prevent the model from ignoring retrieved context?”

— A few things together. We label the context clearly in the prompt so it's obviously separate from the question. We ask the model to cite which chunk it used. And we keep the context tight, because an overloaded prompt makes it easier for the model to drift.

UX Patterns for Low-Confidence RAG Responses

“What is one user experience pattern you've implemented to handle cases where your RAG system has low confidence in its answer?”

— When confidence is low, the interface says so plainly and shows the sources it did find, instead of presenting a guess as fact. Users trust software more when it’s honest with them and doesn’t pretend.

“Do users respond better to fallback answers or clarifying questions? Why so?”

— From my experience, clarifying questions, in most cases. When the system asks a short follow-up instead of guessing, users finish their task more often and get better answers. I already mentioned vague queries, well, this is the fix. The clarifying question feels like the system is actually trying to help. People could feel the difference, even if they couldn't name it.

To Sum Up

Most RAG problems aren't model problems. They're preprocessing, chunking, and evaluation problems hiding in plain sight. A clean RAG architecture with boring, careful data handling beats a flashy setup built on messy inputs almost every time.

Test on your own data, relentlessly. Benchmarks, default settings, and famous models are just starting points. The teams that succeed with RAG implementation are usually the ones willing to do the unglamorous work: label a real evaluation set, watch where retrieval fails, and stay honest about what the system actually does versus what the dashboard claims.