When yet another client asked whether an AI chatbot really handles questions based on company documentation, we decided: enough talking, time to check. At Web Systems we have been building web applications since 2006. The natural next step? Test it on ourselves. On our own internal knowledge base, project procedures and technical documentation. For three months we put a chatbot built on the RAG architecture through its paces and wrote everything down – every success and every failure. This article is an honest report from the trenches. No embellishment, no invented case studies. Concrete numbers, the problems that surprised us, and the conclusions we now share with clients considering a similar rollout.
Spis treści
What RAG is and why we chose this pattern instead of fine-tuning
Retrieval-Augmented Generation is an architecture in which a language model does not answer purely from its training memory. First it searches an external knowledge base, and only then generates an answer based on the fragments it found. In practice? A combination of a smart search engine and a text generator. The user asks a question, the system finds the most relevant documents, and the LLM formulates a coherent answer grounded in facts. And, importantly, there is no need to retrain the model every time someone updates the documentation.
“The retrieval mechanism in RAG is critically important. You need the best semantic search over a carefully prepared knowledge base to ensure that the information retrieved is relevant to the query. If the retrieved data turns out to be inadequate, the generated answer may be correctly grounded yet completely off topic.” – Google / Gemini Enterprise Agent Platform
Fine-tuning? We considered it at the beginning, but dropped it quickly. It requires preparing thousands of question-answer pairs, an expensive training process and repeating the whole exercise with every knowledge update. RAG lets you simply drop a new document into the base – the chatbot sees it immediately. For a software house whose procedures change every quarter, that flexibility turned out to be decisive. On top of that, the cost of a RAG rollout was many times lower than fine-tuning of comparable quality.
- Knowledge base – a set of company documents in PDF, DOCX and Markdown formats, cleaned up and split into fragments
- Embedding model – converts text into numerical vectors that make semantic search possible
- Vector database – stores the embeddings and handles fast similarity queries
- Large language model (LLM) – generates answers based on the retrieved context
- Chat interface – the front-end layer through which users ask questions and receive answers
This modular architecture gives you a lot of freedom. Want to swap the language model? You do not have to rebuild the search engine. Updating the knowledge base? The interface stays untouched. You scale the whole thing incrementally as the organization grows.
How the rollout went step by step – from raw documents to a working chatbot
We started by collecting and cleaning up the documentation. We had roughly 400 files – project procedures in PDF, proposal templates in DOCX, retrospective notes and fragments of our internal wiki. Parsing went more smoothly than we expected. But OCR scans? A disaster. Polish diacritics were mangled at every turn. Every document had to be verified by hand to see whether the extracted text was readable for the embedding model at all.
Chunking, that is splitting documents into smaller fragments – this is where we made the most mistakes. And avoidable ones at that. At first we cut the text mechanically every 500 tokens, ignoring section and paragraph boundaries. The result? Grim. The chatbot’s answers mixed threads from completely different contexts, because a single chunk contained the end of one chapter and the beginning of the next. Only once we moved to a split that respected headings and logical blocks of content did answer quality jump dramatically.
“Beyond a simple vector database replacement, watsonx Discovery offers out-of-the-box NLP enrichments, including entity extraction, sentiment analysis, emotion analysis, keyword extraction, category classification and concept tagging.” – IBM, watsonx documentation
We chose an embedding model that supports Polish and a vector database that could handle fast search even with tens of thousands of fragments. We tested search quality on a sample of questions before connecting the generative model – and that was an excellent decision. Tip: Before you put a chatbot into production, prepare a set of 50-100 test questions with expected answers and check whether the semantic search returns the right fragments – this lets you catch chunking problems early, before users start reporting wrong answers. That one simple step saved us weeks of post-rollout fixes. Seriously.
What surprised me positively – answer accuracy and time savings
The quality of answers grounded in company documentation beat our expectations. The chatbot could point precisely to the procedure for launching a new project, the technologies we use for specific types of rollouts, and where the NDA template lives. But it was not only about factual correctness. The answers sounded natural – not like fragments copied out of a document, but like an explanation from an experienced colleague.
The biggest surprise? Onboarding new employees. Instead of tying up a senior for several hours a day during the first week, a new team member could put questions to the chatbot. Instant, contextual answers. The time needed to learn company procedures dropped by around 40 percent. Sure, the chatbot did not replace a mentor entirely, but it took over the most repetitive questions. “Where do I find the template?” “What is the code review process?” “Who do I report an infrastructure problem to?” That sort of thing.
The system handled multi-part questions better than we expected. A user could ask about the difference between two procedures, and the chatbot would compare fragments from separate documents and build a synthetic answer. In the first month it handled over 1,200 queries. 78 percent required no escalation to a human. The rest concerned mainly topics missing from the knowledge base, or questions that required a business decision – and the chatbot rightly did not make those decisions on its own.
An extra bonus we did not see coming – identifying gaps in the documentation. Analyzing the questions the chatbot could not answer showed us specific holes that needed filling. That unexpected side effect justified the investment in the project all on its own.
What surprised me negatively – the traps nobody writes about
The first disappointment came quickly. The semantic search sometimes returned fragments that looked related to the question but were completely off in the given context. A question about “deployment to production” brought back fragments about “project rollout” in the organizational rather than the technical sense. The model then produced an answer that was linguistically correct but missed the asker’s intent entirely. Those subtle semantic differences could confuse even advanced embedding models. Frustrating.
“Thanks to tuning or prompt engineering aimed at generating text solely on the basis of retrieved knowledge, RAG helps minimize contradictions and inconsistencies in the generated text. This significantly improves the quality of the produced content and the user experience.” – Google / Gemini Enterprise Agent Platform
But even with that approach the hallucinations did not disappear entirely. In roughly 5 percent of cases the model ignored the context it was given and went its own way, generating answers from its training knowledge. Sometimes correct, sometimes completely made up. The worst cases were those where the context was only partly on point. The model filled in the missing information with its own confabulations and produced answers that looked credible. And were false.
- Duplicates in the knowledge base – the same document in different versions caused contradictory answers, because the model received conflicting fragments
- Badly split fragments – chunks that cut a thought off mid-sentence produced incomplete or distorted answers
- Missing up-to-date data – outdated documents still showed up in search results and the chatbot treated them as current
- Overly general user queries – questions like “how does it work” with no context led to random answers, because the search had too little signal for a precise match
- Conflicts between sources – when two documents described the same procedure differently, the model picked at random or mixed both approaches
Each of these problems needed its own solution. Deduplicating documents, versioning the knowledge base, adding a reranking layer that re-scored the relevance of results before passing them to the model. There is no single magic fix here.
Metrics and evaluation – how we measured the quality of the RAG chatbot
The “let’s ship it and see” approach collapsed in the first week. Without systematic measurement we could not tell a subjective impression from a real improvement. We moved to a RAG Ops model – iterative refinement based on concrete metrics. Every change to chunking, prompts or search configuration was measured and compared with the previous result. More work? Yes. But it eliminated the dead ends we used to charge into.
We built an internal evaluation platform that scored answers along several axes – coherence (does the answer hold together logically), linguistic fluency, grounding in sources (does every sentence have backing in the supplied context) and safety (does the answer mislead). Each of these metrics gave a separate signal about what was failing. Grounding alone does not guarantee quality – an answer can be correctly rooted in the sources yet unreadable or incoherent.
The most important metric turned out to be groundedness – the percentage of statements in an answer that have direct backing in the supplied fragments. At the start? 71 percent. That meant nearly every third sentence contained information from outside the context. Not great. After three iterations of prompt optimization and a rebuild of the chunking strategy we raised it to 89 percent. In parallel we improved search accuracy by 23 percentage points – and that translated into a clearly better user experience.
Comparing quality before and after optimization showed something worth remembering. The biggest jump came from improving the source data, not from modifying the model. Better document parsing, removing duplicates, adding metadata to fragments – that delivered more than experiments with different LLMs. A lesson for anyone planning a rollout: the quality of the knowledge base determines the quality of the answers more than the choice of model does. I tested that the hard way.
FAQ – the most common questions about AI chatbots with RAG
How much does a RAG chatbot rollout cost in a mid-sized company?
It depends on the size of the knowledge base and the integration requirements. A simple MVP with a few hundred documents and one channel (web chat) can be launched on a budget of 15,000-40,000 PLN. More advanced solutions – with CRM integration, multilingual support and advanced reranking – run to 60,000-150,000 PLN. On top of that come operating costs: hosting the vector database, fees for the language model API and ongoing updates to the knowledge base. All together, from several hundred to several thousand PLN a month.
Can a RAG chatbot replace a customer service department?
It should not. But it does take real load off. From our experience, a well-implemented RAG chatbot handles 60-80 percent of repetitive queries. The ones whose answers sit in the documentation, the terms of service or the FAQ. Complex cases, complaints that call for empathy, non-standard situations – those still need a human. A hybrid model works best. The chatbot handles the first line and escalates harder matters to a consultant along with the full conversation context.
How long does a RAG chatbot rollout take from idea to production?
Realistically? 4-8 weeks for an MVP, 3-4 months for a full production solution. The first week is an audit and preparation of the knowledge base. The next two go on building the document processing pipeline, configuring the search and integrating with the language model. The last week of the MVP is spent on quality testing and iteration. And the most common reason for delay? Underestimating the time needed to clean up and structure the source documentation. Companies regularly discover that their knowledge base is far messier than anyone suspected.
Conclusions after three months – is RAG worth implementing in a company
Three months of testing gave us a far more nuanced picture than the usual marketing promises of AI vendors. RAG works. It delivers real benefits. But on one condition – continuous work on the quality of the knowledge base. This is not an “install and forget” solution. It is a living system that requires regular document updates, metric monitoring and a response to new patterns in user queries.
The biggest lesson? The success of a RAG rollout depends 70 percent on the quality of the input data and only 30 percent on the technology. Companies with well-organized documentation, clear procedures and a coherent knowledge base will get good results quickly. Organizations with scattered, outdated or contradictory sources have to invest in tidying up their content first. The paradox is that the very process of preparing data for a chatbot forces information hygiene. And that pays off regardless of AI.
RAG works best where there is a large volume of repetitive questions whose answers are buried in documentation – customer service, employee onboarding, technical support, product knowledge bases. But if your queries are mainly creative, require decision-making or concern data outside the documentation? A simpler solution – a well-organized FAQ or a ticketing system – may turn out to be enough. And cheaper to maintain.
If you are considering an AI chatbot rollout in your company, integrating RAG with your existing systems, or want to start with an MVP that verifies the potential of this technology in your business context – get in touch with us at Web Systems. We are happy to share our experience and help you avoid the mistakes we made ourselves at the start.
