Rolling out an AI-based application rarely ends the way it looks in startup movies. No bang, no drama, no single fatal decision. In practice, AI projects sink slowly. You lose weeks on fixes, the budget dissolves into costs nobody anticipated, and after six months the client has a working demo instead of a working product. We, the Web Systems team, a software house from Łódź operating continuously since 2006, have been through hundreds of rollouts: web and mobile applications, B2B systems, API integrations, automations, e-commerce and, for a few years now, increasingly often solutions based on artificial intelligence. And that gives us one advantage. We see where a project will get stuck before the client has even managed to call it a problem.
Let me start with a claim that should weigh on every conversation about budget: most rollout failures do not come from “not having the right model”. Language models today are surprisingly good and cheap. The problem sits elsewhere – in the decisions made before anyone writes the first line of code. Who owns the data? What happens when the model returns an error? How will you calculate the cost of one query times a thousand users a day? Because if those questions only come up at the production stage, it is already too late. And the fixes then cost many times more than a well-designed foundation.
The topic of a software house in Łódź and a real AI application rollout sounds like a marketing slogan, but at its core it is about something very down to earth. About engineering. AI is not a magic add-on that you simply “plug into” an existing system. It is another layer, and it has its own requirements – architecture, data, security, maintenance. A company that treats artificial intelligence as a feature to tick off in a specification will sooner or later hit a wall. Either at the scaling stage, or when the inference bill arrives, or when the application starts giving confident but wrong answers.
In this article we break down the five mistakes that most often derail AI projects in Polish software houses. We are not writing this as a neutral market observer, but as a contractor who knows these problems inside out. Because we made some of them ourselves at an early stage and drew concrete conclusions from them. All these mistakes have a common denominator: they come from confusing an impressive prototype with a finished, maintainable product. If you are looking for an AI application contractor, this text will give you a concrete list of questions worth asking before you sign a contract. And if you build such solutions yourself, treat it as a checklist that will spare you a few painful lessons. So let’s get to it. From a demo that works only during the presentation, all the way to having no plan for what happens the day after the rollout.
Spis treści
Mistake 1: confusing the demo with production – AI that only works at the presentation
The most common trap looks innocent. Someone opens the model provider’s playground, pastes in a neat prompt, gets a great answer and announces that “the application basically works already”. Well, it does not. A working prompt is not an application – it is a single, isolated experiment under ideal conditions. Between it and production lies an entire layer of engineering that is invisible at a presentation, and that decides whether the solution survives contact with real users. What is missing is the data layer, error handling, request limits, token cost control, monitoring. And that invisible part is roughly 80 percent of the actual work.
Think about what happens when the model responds slowly or does not respond at all. In the playground you simply click “try again”. In production you need a defined strategy: timeouts, retries with exponential backoff, a fallback mechanism for when the provider is down, a sensible message for the user and event logging so it can be analyzed later. Without that, the first serious traffic on the application ends in a series of errors nobody anticipated. A demo never tests such scenarios. Because a demo by definition shows only the path that works.
The second face of the same mistake is architecture. Or rather the lack of it. In the rush, all the logic lands in a single endpoint: fetching data, building the prompt, calling the model, parsing the response, saving – all in one function several hundred lines long. Code like that looks great at a hackathon and becomes a nightmare to maintain. Every change threatens to break something else, testing a single part is practically impossible, and onboarding a new developer onto the project takes weeks. And a healthy split into layers – user interface, business logic, data access – is not an academic luxury. It is the condition for being able to develop the application for longer than one sprint.
The third dimension is money. In the playground the cost of a query is invisible. In production, with thousands of users, every token translates into a concrete invoice. An application that sends whole documents to the model without control, or repeats the same queries without caching, can generate a bill many times higher than assumed. Inference cost control – limits, response caching, choosing a cheaper model for simpler tasks, capping context length – has to be designed from the start. Not bolted on after the first shock on the billing statement.
What can you do about it in practice? Before signing a contract with a contractor, it is worth moving from the level of promises down to the level of specifics.
Tip: before you sign a contract for an AI application, ask the contractor to describe two things in writing – the plan for handling model errors (what happens when the answer does not arrive or is incorrect) and the mechanism for controlling inference costs. If you hear generalities instead of specifics, that is a warning signal.
At Web Systems we treat the demo phase as a stage for validating the idea, never as a product. After showing a working prototype there is always a conversation about what separates it from the production version: about scale, costs, failures and data. A less spectacular moment than the presentation itself, true. But it is exactly then that the decisions are made which determine whether we deliver the project on budget or spend the following months rescuing it. Confusing the demo with production is mistake number one, because it opens the door to all the others.
Mistake 2: weak retrieval in RAG – the application answers confidently, but off-topic
RAG architecture, that is retrieval-augmented generation, has become the default way of building AI applications based on company knowledge. The idea is simple and sound: instead of counting on the model “knowing” everything, we first search the knowledge base for the fragments relevant to the question, and then ask the model to answer solely on their basis. Except that the quality of the whole solution depends not on the model but on that first step – on the search. When the retrieval mechanism hands the model irrelevant fragments, you get an answer that sounds confident, is written correctly, and completely misses the point.
The industry documentation puts it unambiguously:
The retrieval mechanism in RAG is critically important. You need the best semantic search on top of a curated knowledge base to make sure that the retrieved information is relevant to the query. If your retrieved information is irrelevant, your generation could be grounded but off-topic or incorrect.
That sentence should hang above the desk of everyone who builds RAG. “Grounded but off-topic” is the most dangerous kind of error, because the user has no way of catching it. The answer looks credible, the model cites some source, the tone is professional. Only a domain expert notices that the content is about something adjacent. And in B2B use cases, where the application advises on legal, technical or financial matters, such a silent error can cost far more than an open outage.
Where do weak search results come from? Most often from three failures at the data level. The first is the lack of knowledge base curation – everything the company owns gets dumped into it, together with outdated document versions, duplicates and internally contradictory material. The model has no way of deciding which version of the policy is in force, so it mixes them together. The second is bad chunking, that is splitting documents into fragments. If you cut the text mechanically every few hundred characters, you break a sentence in the middle of a thought and destroy the context the model needs. The third is poor parsing – misread tables, skipped headings, columns glued together from a PDF. Garbage in, garbage out works here without mercy.
But the most important lesson concerns measurement. Judging RAG quality “by eye”, by clicking through a handful of sample questions, gives a false sense of control. The professional approach means measuring concrete metrics – above all the groundedness of answers in the sources, fluency of language, and the extent to which the answer actually relates to the question. As a piece on evaluation platforms puts it:
Implementing these evaluations gives you a baseline and lets you optimize RAG quality by configuring the search engine, curating the source data, improving the parsing of source layout or the chunking strategy, or refining the user’s question before retrieval. A metrics driven approach lets you gradually hill climb toward high quality generation.
In other words: RAG quality is not a one-off effect but a process of iterative tuning. You measure, change one parameter, measure again. Without a baseline and metrics, every “improvement” is plain guesswork.
Tip: if you are commissioning an application such as a knowledge chatbot or a document assistant, agree with the contractor right at the start how answer quality will be measured. Ask for a set of test questions with expected answers and for a report with groundedness and relevance metrics. That turns the vague promise of “it works well” into something you can verify and hold someone accountable for.
In our RAG projects we treat the knowledge base as a product in its own right – with an update process, version control and regular evaluation. Less spectacular than the model itself, granted. But this is exactly where it is decided whether the AI application really helps or merely gives the impression of helping.
Mistake 3: no architecture and no single source of truth for data
The third mistake is more fundamental than the others, because it concerns not the AI layer but the entire way the application is built. Artificial intelligence changes nothing here. On the contrary – it exposes architectural weaknesses faster and more painfully. An AI application operates on data, states and flows that are easy to turn into chaos if engineering discipline is missing. And that discipline starts with one principle which sounds trivial and in practice saves projects: separation of concerns.
The architectural documentation calls it by name:
The most important principle is separation of concerns: separating your app into methods, classes, files, packages, modules and layers that have clearly defined responsibilities and boundaries. It’s a common mistake to write all your code in a single component.
“All the code in a single component” is exactly the same disease we described under mistake one, only here we are looking at it from a bird’s eye view. When business logic, database access, AI model calls and interface handling are all mixed in one place, the application becomes a monolith in which nothing can be changed without risk. Every layer should have one clearly defined task and communicate with the others through defined boundaries. The minimum healthy split is a presentation layer, a business logic layer and a data layer – and in more complex systems an intermediate layer with reusable use cases is added.
The second pillar is the single source of truth, that is the principle that for each type of data there is exactly one owner that may modify it. The rest of the system only reads that data in immutable form and reports changes through events. It sounds formal, but the consequences are thoroughly practical. As the source material describes it, such a pattern centralizes all changes to a given type of data in one place, protects the data from external manipulation and makes changes easier to trace, so bugs are easier to catch. In an AI application, where the same piece of state may be read by the interface, modified by the logic and enriched by the model, the absence of a single data owner leads to bugs nobody can reproduce afterwards.
The single source of truth goes hand in hand with unidirectional data flow. State flows in one direction, usually from the data layer toward the interface, while user events flow the opposite way until they reach the data owner. Thanks to that, at any moment you know where a given value came from and what could have changed it. It is the foundation of an application that is meant to scale – both in terms of traffic and in terms of the number of developers working on it in parallel.
What exactly do you lose by giving up architecture in the name of a “faster start”? The list of consequences is predictable and expensive:
- Difficult tests – when network, data and presentation logic are intertwined, you cannot test a single part in isolation, so most often there are no tests at all.
- Long onboarding – a new developer needs weeks to understand code in which everything depends on everything, instead of days with a readable split into layers.
- Conflicts in the code – several people editing the same overloaded file means constant merge conflicts and changes that cancel each other out.
- Growing technical debt – every shortcut today is a many times greater cost tomorrow, when the codebase grows and the foundations have to be rebuilt.
- No scalability – an application that worked with a hundred users starts falling apart at ten thousand, because nobody thought about boundaries of responsibility.
Good architecture is not free – it requires thinking at the start. But it is an investment that pays off at the first larger change of scope. And in AI projects such changes are the rule, not the exception. Models change, requirements evolve, new data sources appear. A system with clean boundaries absorbs those changes. A system without architecture cracks. At Web Systems we design the layers before we write the AI logic, because we know one thing: artificial intelligence is only as good as the foundation it stands on.
Mistake 4: ignoring integrations, data and security
An AI application does not exist in a vacuum. That is one of the most frequently overlooked facts in conversations about rollouts – the client imagines an elegant chat with a model and forgets that the real value only emerges when that intelligence connects with the systems the company actually works on. A CRM with customer history, an ERP with order and warehouse data, an e-commerce platform, internal B2B systems, document repositories. AI cut off from those sources is like an adviser who has been denied access to the files – they may speak nicely, but they know nothing specific about your company.
Integrations are usually the most labor-intensive and most underestimated part of a project. Every system has its own API, its own data formats, its own performance limits and its own quirks. Synchronizing data between the model and the ERP in near real time, handling situations when the source system is unavailable, mapping fields that are named differently in different systems – all of that is real engineering that has to be planned. Not assumed to “somehow plug in”. As a software house carrying API integration experience since 2006, we know that this is exactly where projects most often slip in time.
It is also worth understanding that a modern AI application is far more than a vector database with similarity search. The source material shows well where the additional value lies:
Going beyond a simple vector database replacement, the solution offers ready-made NLP enrichments including entity extraction, sentiment analysis, emotion analysis, key phrase extraction, category classification and concept tagging.
This matters, because it shows that real business value is born out of data enrichments rather than out of search alone. Entity extraction lets you automatically pull company names, amounts and dates out of a document. Category classification routes a ticket to the right department. Sentiment analysis lets you catch a dissatisfied customer before they leave. These NLP layers turn raw text into structured knowledge on which automations can be built – and they are often what decides whether an AI rollout really saves time or is just a pretty gadget.
The most serious area, however, is security. Treated as an afterthought in AI projects surprisingly often. When sending data to an external model, you have to know precisely what flows out and where to. A poorly thought-through rollout can send sensitive data to the model provider – customers’ personal data, trade secrets, information covered by confidentiality agreements. There are several risks and each of them requires a conscious decision.
- Data leaking into the model – sensitive data pasted straight into the prompt may end up outside the controlled environment; you need anonymization, masking or a privately hosted model.
- No access control – the AI application has to respect user permissions, otherwise it will return data to someone who should not see it, bypassing the safeguards of the source systems.
- Sensitive data in prompts and logs – what goes into the prompt usually ends up in the logs as well; without a thought-through logging policy you create a second, uncontrolled copy of confidential information.
These issues have to be settled at the design stage, because they touch on GDPR compliance and the company’s real legal liability. The choice between a cloud model and a locally hosted one, the provider’s data retention policy, encryption, mechanisms for masking sensitive data before sending – these are architectural decisions, not details to be tied up at the end. In projects involving personal data or trade secrets, we treat security as the first topic of the conversation, not the last. Because the difference between an AI application that builds a company’s advantage and one that creates a leak risk lies exactly here – in the way data is handled, integrated and protected.
Mistake 5: zero maintenance plan – the rollout treated as the end, not the beginning
The fifth mistake is perhaps the most costly in the long run, because it stems from a fundamental misunderstanding of the nature of AI software. The client receives a working application, signs the acceptance protocol, celebrates the end of the project – and treats the moment of rollout as the finish line. And for an application based on artificial intelligence, the rollout is only the starting line. What happens afterwards decides whether the solution still delivers value a year later or quietly degrades into a tool nobody trusts any more.
Let’s start with the cost of inference, because it is the most tangible aspect of maintenance. Unlike a traditional application, where after the rollout you mainly pay for the server, an AI application generates a cost with every use. The more popular it is, the higher the bill. Without monitoring token usage, without prompt optimization, without caching repeatable queries and without a conscious choice of model for the task, the cost can grow faster than the value the application delivers. Maintaining AI means, among other things, constantly watching the economics of that process.
The second area is the ground shifting under your feet. Model providers regularly release new versions, retire old ones, change prices and API behavior. The model you based your application on may become unavailable within a year or start answering differently. Without a plan for updates and without an abstraction layer separating your application from a specific provider, every such change is a sudden emergency project. With a well-designed architecture, it is a controlled migration.
The most insidious problem, though, is quality degradation over time. An application that answered excellently on rollout day may answer worse a few months later, even though the code has not changed. The reasons vary: the knowledge base has aged, new types of questions have appeared that the system was not prepared for, the model’s behavior changed after an update at the provider. Without metrics you will not notice this until the complaints start coming in. And here the lesson about measurement returns, the one we learned with RAG – without numbers you cannot improve quality, because you do not even know whether and where it dropped.
That is why mature AI rollouts include what is increasingly called RAG Ops or LLM Ops – the operational discipline of maintaining quality. Regular evaluation on a fixed set of test questions, monitoring of groundedness and relevance metrics, alerts when quality drops, a process for updating the knowledge base, tracking costs and model versions. This is not an optional extra for the largest players. It is the minimum if the application is to genuinely serve the company for years rather than merely pass a successful demo at launch.
Tip: plan the budget for maintaining an AI application from day one, not after the first outage. Assume that operating costs, monitoring and regular tuning are a fixed item, not a one-off expense. A company that knows maintenance awaits it after the rollout makes better architectural decisions right at the start – because it designs for the whole life cycle, not for the moment of acceptance.
At Web Systems we treat maintenance as an integral part of every AI project, not a service tacked on after the fact. We discuss it at the quoting stage, we design the application so that it can be monitored and updated, and we tell the client plainly what the life of the product after the rollout requires. Because an AI application without a maintenance plan is not a finished project. It is a deferred problem.
FAQ: the most common questions from clients looking for an AI application contractor
We have collected three questions that come back in almost every conversation with a client considering an AI application rollout. We answer them the way we do in meetings – without jargon and without promising things that cannot be kept.
Do I need my own model, or is integrating a ready-made LLM enough?
In the vast majority of business cases, integrating a ready-made large language model is the right and cheaper choice. Training your own model from scratch is a project counted in hundreds of thousands of zlotys and requiring enormous data sets – it rarely pays off. Real value for a company arises not in the model itself but in the layer around it: in a well-prepared knowledge base, in the RAG architecture, in integrations with your systems and in data enrichments. If you have a specific, repeatable task and a lot of data, fine-tuning an existing model can sometimes make sense, but that is an optimization for a later stage, not a starting point. We usually start with the integration of a ready-made model and focus on what really differentiates the solution.
How much does maintaining an AI application really cost after the rollout?
There is no single number, because the cost depends on the scale of usage, the chosen model and the complexity of the solution, but it can be broken down into three components. The first is the cost of inference, that is the fees for queries to the model – they grow with the number of users and can be significantly reduced through prompt optimization and caching. The second is infrastructure and hosting, similar to a classic application. The third, most often overlooked, is the operating cost of maintaining quality: monitoring, updating the knowledge base, reacting to model changes and regular tuning. An honest quote accounts for all three. We encourage you to think of it like the cost of owning a car rather than a one-off purchase – maintenance is a fixed item worth planning for from the start.
Where do I start if I have data but do not know whether it is suitable for AI?
With a data audit and a small pilot project. Before you invest in a full rollout, it is worth checking the state of your data – whether it is complete, up to date, consistent and whether it can be sensibly cut into fragments for search. It often turns out that the greatest value of the initial stage is precisely the tidying up and curation of the data, because that is what determines the quality of the whole solution. We suggest starting with a limited scope – one specific use case, on a selected slice of data – measuring answer quality with metrics, and only then scaling. Such a pilot costs a fraction of the full project and lets you make a decision based on numbers rather than assumptions.
Summary and contact: how not to repeat these mistakes
The five mistakes we have discussed share one common denominator – all of them come from putting an impressive start above solid foundations. Let’s recap them briefly, because this is a list worth keeping at hand in every conversation about an AI rollout. First, confusing the demo with production – a working prompt is not an application until there is a data layer, error handling and cost control underneath it. Second, weak retrieval in RAG – an application that answers confidently but off-topic is more dangerous than one that is openly wrong. Third, no architecture and no single source of truth – without clean boundaries between layers the project turns into a monolith that can neither be tested nor developed. Fourth, ignoring integrations, data and security – AI without a connection to the company’s systems and without control over sensitive data creates risk instead of value. Fifth, no maintenance plan – the rollout is the start, not the finish, and without metrics and RAG Ops quality quietly degrades over time.
Note that none of these mistakes concerns the model itself. All of them concern the decisions around it – architectural, data-related, maintenance-related. And that is no coincidence. Artificial intelligence is today the easiest piece of the puzzle. What is hard is building around it a solution that works in production, integrates with existing systems, respects data security and can be maintained for years. That is exactly why the topic of a software house in Łódź and a real AI application rollout ultimately comes down to engineering and experience, not to access to a fashionable model.
Web Systems has been that kind of partner since 2006. Over those years we have built web and mobile applications, B2B systems, API integrations, automations and e-commerce platforms, and today we combine that experience with AI solutions. We do not promise magic – we promise a sensible, technical approach in which we first understand your problem, your data and your systems, and only then choose the tools. We work both on projects from scratch, starting with the first MVP, and on the modernization of existing systems that need a fresh breath or a layer of intelligence.
If you are planning an AI application, an integration, an automation, an MVP of a new product or the modernization of a system that has stopped keeping up with your company – let’s talk. We will gladly look at your idea, point out the real risks before they become problems, and propose an approach tailored to your data, budget and goals. Get in touch with the Web Systems team and see what it is like to work with a contractor who thinks not only about the launch, but about the entire life of your solution.
