Chatbots built on RAG architecture were supposed to be a breakthrough in access to company knowledge – fast answers for employees, customers or partners, without digging through dozens of folders and systems. And, as usual, reality verified those plans rather brutally. At Web Systems we have been designing AI solutions for business for years and we keep seeing the same pattern: a promising rollout turns into a source of frustration, because the chatbot starts telling people things that have no basis in the documents. Hallucinations in RAG systems do not come out of nowhere. They are the consequence of specific architectural mistakes that can be tracked down and eliminated before the system ever reaches users. Below we go through the six most common reasons why a chatbot trained on company documents lies through its teeth – and show how to deal with them.
Spis treści
No knowledge base curation – garbage in, garbage out
The most common mistake we see in RAG implementations? The “let’s dump everything in and see what happens” approach. Companies load the entire contents of network drives, SharePoint or the intranet into the knowledge base without any selection. The result is usually grim. The chatbot answers based on an outdated policy from three years ago, or quotes a draft version of a document that was never approved. Every duplicate, every obsolete procedure, every scan without proper OCR is a ready-made source of hallucinations.
But the problem gets really nasty when the knowledge base contains internally contradictory documents. Imagine two versions of the leave policy – one from 2021, the other current. The model does not know which one applies, so it may pick either. Or, worse, combine information from both. The user gets an answer that sounds credible but contains wrong data. And that is it. Trust in the system collapses after a single incident like that, and rebuilding it takes months.
“You need the best possible semantic search over a carefully curated knowledge base to make sure the retrieved information is relevant to the query. If the retrieved data is irrelevant, the generated answer may be grounded in context but off topic – or simply wrong.”
Tip: Run a proper document audit before indexing. Throw out duplicates and drafts, update obsolete material, merge fragmented files into coherent wholes. Test OCR quality on scans – if the text is unreadable for you, it will be useless for the model too. The knowledge base has to be treated as a product that needs regular care. Loading it once is not enough.
The wrong document chunking strategy
Splitting documents into fragments looks like a technical detail, yet it can decide whether the whole rollout succeeds. Chunks too large? The model gets an excess of information and loses what matters in the noise of side data. Too small? You tear sentences out of context and strip them of meaning. Finding the right granularity requires understanding the structure of the documents and the way people actually ask questions.
A frequent problem is ignoring the natural structure of a document while splitting it. I have tested this many times – tables broken up into single rows lose their meaning, procedure step lists split across fragments become useless, and headings detached from their content mislead the model. The chunking algorithm has to respect the semantic boundaries of the text. Mechanically counting tokens or characters is not enough.
Different document types call for different splitting approaches. Here are the most commonly used strategies and the situations where they work best:
- Chunking by headings – ideal for technical documentation and manuals, where sections are self-contained units of information
- Chunking with overlap – works well for narrative texts, where context flows smoothly between paragraphs
- Semantic chunking – best for complex legal documents and contracts, where related clauses can sit in different places
- Recursive chunking – the universal choice when the knowledge base holds a mix of formats and content types
- Table chunking – essential for financial documents, reports and summaries, where tabular data has to keep its structure
In our projects we test several strategies on a representative sample of documents before making the final call. The cost of such experiments? Minimal compared with the risk of rolling out a system that hallucinates regularly because of badly split sources.
A weak semantic search mechanism
You can have a perfectly prepared knowledge base, but if the search mechanism cannot find the right fragments – what is the point of all of it? Simple vector search based purely on similarity search is a starting point. Not a production solution. Without rerankers, metadata filters and hybrid mechanisms, the system too often returns fragments that look semantically similar but miss the substance.
Hybrid search – combining vector search with classic keyword search – significantly improves result accuracy. I have verified this across many projects. Embeddings handle paraphrases and synonyms well, but they fail on specialist terminology, document numbers or product names. This is where keyword search comes in and fills that gap perfectly. Skipping this approach is one of the more common mistakes we observe even in mature technology organizations.
Configuring the top-k parameter and similarity score thresholds calls for considerable judgment. A top-k value that is too high floods the model with irrelevant fragments, dilutes the context and increases the risk of hallucination. Too low? There is not enough information for a complete answer, and the model fills the gaps with its own knowledge. Because that is how it works – it does not like saying “I don’t know”. In practice the optimal values differ depending on the domain and the type of questions. A universal default rarely holds up in production.
A reranker as a second layer of result scoring can dramatically improve retrieval quality. The first search stage returns candidates quickly but imprecisely. The reranker analyzes them more deeply in the context of the query and sets the right order. This two-stage architecture increases precision without sacrificing speed, which matters enormously in a production environment.
No evaluation and no answer quality metrics
Rolling out a RAG chatbot without a system for measuring answer quality is like driving a car with the gauges covered up. You do not know how fast you are going, how much fuel is left, or whether the engine is overheating. At Web Systems we treat evaluation as an integral part of the architecture, not an optional extra. Metrics such as groundedness, faithfulness and relevance let you judge objectively whether the system works properly – instead of relying on the impressions of a few people who “clicked around and it looks fine”.
Modern evaluation platforms offer whole sets of metrics for assessing generated text. Coherence measures the logical consistency of an answer, fluency rates how natural the language is, groundedness verifies that it is based on sources, and instruction_following checks compliance with the system instructions. Each of these metrics reveals a different aspect of potential problems and points to what specifically needs fixing. Without them, iterative improvement of the system rests on guesswork. Not on data.
The RAG Ops approach assumes a continuous cycle of measurement, analysis and optimization – analogous to DevOps, but focused on the quality of generated answers. After every configuration change, knowledge base update or prompt modification you run a set of tests and compare the results with the previous ones. That discipline lets you improve quality systematically instead of changing things blindly and keeping your fingers crossed.
Tip: Build a set of at least 50 test questions with expected answers, covering the typical usage scenarios. Run it automatically after every change to the system. And add trick questions where the correct answer is a refusal – that is the best way to catch hallucinations before they reach end users.
A careless prompt and no grounding instructions for the model
A language model without a clear instruction to “answer solely on the basis of the provided context” behaves like a talkative guest at a party who hates admitting they do not know something. When the retrieved context does not contain the answer to a question, the model reaches for general knowledge and produces an answer that sounds confident but has nothing to do with the company’s documents. The user cannot tell such a hallucination apart from a correct answer. And that is exactly why it is so dangerous.
A refusal mechanism is the element missing from most implementations. The system should be able to say “I did not find enough information in the available documents” instead of making things up. Designing that behavior requires precise instructions in the system prompt, defining confidence thresholds and escalation scenarios. A generic “answer based on the context” template is not enough. The prompt has to account for the specifics of the domain, industry terminology and typical traps.
The difference between a prompt tailored to a specific field and a generic template can be enormous. In projects for law firms we add instructions on precision when citing legislation; in medical systems, rules on caution when interpreting results. Every domain has its own rules, and breaking them undermines the credibility of the entire solution. Fine-tuning the prompt to the company’s specifics is an investment that pays off quickly – in answers users can rely on.
The system prompt also acts as an arbiter in ambiguous situations. When the retrieved fragments contain seemingly contradictory information, a well-designed prompt tells the model how to settle such conflicts. Does the newer document take priority? The document with a higher level of approval? Or should the system present both versions and ask the user to make the question more specific? This has to be designed up front.
Ignoring system maintenance after go-live
Company documents live their own lives. Policies change, procedures evolve, new products appear, old ones leave the range. A RAG chatbot whose knowledge base is not updated regularly starts serving outdated information within a few weeks of launch. It is not a question of whether it will go stale – it is a question of when. Without an automated update pipeline the system becomes less trustworthy with every passing day.
Monitoring user queries matters just as much. Because without analyzing what people actually ask, you do not know where the chatbot falls over. Maybe it regularly struggles with questions about one particular product, because that product’s documentation is full of holes. Maybe users phrase questions in a way the system does not understand. A feedback loop between users and the team maintaining the RAG system is the foundation of long-term quality. Without it you are flying blind.
Maintenance costs for a RAG system are often underestimated at the planning stage. Reindexing documents after updates, regenerating embeddings when the model changes, versioning the knowledge base, monitoring performance and quality – all of it takes resources and processes. In our implementations we always design the software architecture with post-launch operations in mind. Because a system nobody maintains degrades faster than it was built.
Practice shows one thing – companies that treat RAG as a project with an end date instead of a product that needs continuous care lose the money they invested within a quarter. Versioning the knowledge base lets you roll back changes when a new batch of documents lowers answer quality, and automatic alerts flag a drop in metrics before users start complaining.
FAQ
Can a RAG chatbot eliminate hallucinations completely?
No. No system based on language models guarantees one hundred percent accuracy. You can, however, reduce the risk of hallucination to a level that is acceptable in a business environment. It takes a multi-layered approach – careful curation of the knowledge base, a precise prompt with grounding instructions, a refusal mechanism for low-confidence cases, and continuous quality evaluation. Each of these layers reduces the probability of error, and together they produce a system users can trust day to day. One more thing – the chatbot should communicate the confidence level of its answers transparently. No pretending it knows everything.
How many documents does a company need for RAG to make sense?
The number of documents matters less than their quality and topical coverage. We have seen effective implementations built on twenty carefully prepared documents that answered 80% of user questions. Seriously – twenty. A minimum viable knowledge base should cover the most frequent query scenarios in a given organization. It is better to start with a narrow, well-covered area and gradually widen the scope than to index thousands of files straight away with no quality control. That iterative approach delivers value quickly and builds trust, instead of risking disappointment right at the start.
Summary
Each of the six mistakes described here is a design decision, conscious or not. Not a random accident. No data curation, careless chunking, weak search, no evaluation, a generic prompt and neglected maintenance – these are architectural elements that can be designed correctly from the start or fixed in an existing system. RAG is not a plug-and-play solution you connect to your documents and forget about. It requires a well-thought-out architecture, systematic testing and continuous operational care.
At Web Systems we approach RAG chatbot implementations like any other engineering project – with a plan, metrics and clearly defined success criteria. If you are planning to launch a chatbot on your company documents, or your current system produces answers users do not trust, get in touch with us. We will help you design a RAG architecture that resists hallucinations, or audit your existing solution and point to the specific places that need fixing.
