Building an application based on artificial intelligence (AI) looks simple on slides today. In production – rarely. As the Web Systems team, a software house from Łódź operating continuously since 2006, we have seen dozens of projects in which the thing that failed was not the language model, but the way the contractor approached the system as a whole. An AI application does not end with an impressive demo that produces nice answers during a presentation. It is a product that has to be maintained for years, integrated with the company’s tools, secured and scaled for real traffic. And that is precisely why the relationship between a software house in Łódź and building an AI application is so often settled at the level of the contractor’s engineering competence, not the list of models it likes to show off.
The difference between a demo and a rollout is like the difference between a prototype and a car approved for the road. The first one is meant to impress. The second one has to drive every day, in rain and frost, for several years. In AI projects that gap tends to be even deeper, because the unpredictability of models, inference costs and data quality can wreck a budget that looked perfectly reasonable on paper. A client looking for a contractor rarely sees those risks until they surface mid-project. And by then it is usually too late for a cheap course correction.
In this article we walk through the five mistakes we see most often, the ones that really decide whether a software house delivers a working AI application or just another prototype for the drawer. These are not abstract warnings from a guidebook. They are technical, financial, integration and maintenance problems we know from our own projects. All of them share one common denominator: the absence of systems thinking. When a team treats AI as a feature glued onto an application rather than as a system with an architecture, data and a life cycle, the outcome is predictable. It works at the demo, and after the rollout it starts falling apart.
Before we get to the specifics, one caveat. The point is not to scare anyone with technology or to argue that AI is harder than it really is. The point is for the client to ask the contractor the right questions before signing a contract. A good technical partner raises these topics on their own: architecture, retrieval, maintenance costs, data security, quality metrics. And if they do not raise them? That does not mean the problems are not there. It only means the client will be the one carrying them. Usually at the least convenient moment.
Spis treści
Why AI applications fail because of the contractor more often than because of the technology
Language models and libraries for building AI solutions are within arm’s reach today. Access to the Claude API, OpenAI models or open models hosted locally is no longer any barrier to entry. And paradoxically, it is exactly that availability that creates the illusion that building an AI application comes down to plugging in a model and writing a few prompts. Meanwhile, the difficulty has moved elsewhere – from the model to the engineering around the model. Most of the project risk sits in the integration, data, monitoring and maintenance layers.
When we analyze failed rollouts, the model itself is rarely to blame. Far more often the failure lies with a contractor who did not treat the AI application as a fully fledged IT system. There is no layered architecture. There is no thought-through handling of input data. There is no plan for the situations in which the model returns a wrong or incomplete answer. A demo does not require any of this, because a demo controls the conditions. Production does not control them – so without solid engineering it falls apart on first contact with real users and real data.
For us, a software house from Łódź with almost two decades of experience in web applications, mobile applications, B2B systems and API integrations, AI is one more layer that has to be embedded in proven engineering discipline. The same principles that have decided software quality for years – separation of concerns, a single source of truth, testability, cost control – apply here as well. An AI application is not an exception to the rules of good engineering. It is a particularly demanding case of them, with the added unpredictability of the model on top.
So the relationship between a software house in Łódź and building an AI application comes down to a single question: does the contractor understand that they are building a system, not a flashy toy? A client who does not verify this is buying a promise, not a product. In the sections that follow we break down the five mistakes that most often separate one from the other. We describe each of them from the perspective of a contractor who knows what they cost, because we have more than once cleaned up after someone else’s project or deliberately avoided them in our own.
It is worth reading these points as a checklist for a conversation with a prospective supplier. Not in order to become an architecture expert in a single afternoon. But in order to recognize whether the team on the other side of the table thinks in terms of systems. That one distinction protects a budget more effectively than any list of technologies in a proposal. An AI application delivered all the way to the finish line is the sum of dozens of good engineering decisions, not a single choice of model.
Mistake 1: Treating AI as a gadget instead of a system with an architecture
The most common mistake is at the same time the most fundamental one. The team treats the AI feature as an ornament glued onto the application rather than as a system that requires a well-considered architecture. In practice it looks like this: all the logic lands in one place – the prompt, the model call, parsing the response and updating the interface, all in the same function or controller. This is exactly the antipattern that in the mobile world is called writing all your code in an Activity. At first it works. But it makes further development impossible.
A well-designed application, including one with AI, rests on separation of concerns. The interface layer displays data, the data layer holds the business logic and exposes information, and an optional domain layer encapsulates complex rules used in many places. Industry sources state this principle without mincing words:
The most important principle is separation of concerns: separating your app into methods, classes, files, packages, modules and layers that have clearly defined responsibilities and boundaries. It’s a common mistake to write all your code in an Activity.
The quote formally refers to mobile applications, but the principle is universal. When the model-call logic, context handling, retrieval and result presentation are all mixed together, the result can neither be tested nor developed. Every prompt change threatens to break the interface. Every interface change requires digging into the AI logic. And what emerges is a monolith nobody wants to touch, because every modification carries the risk of a regression somewhere unexpected.
The second symptom of this mistake is ignoring the principles of a single source of truth and unidirectional data flow. In an AI system the state tends to be complex: the conversation history, the context retrieved from the knowledge base, the model parameters, the call status. If several components modify that state independently, you get bugs that are impossible to reproduce afterwards. The single source of truth pattern brings order here – it designates one owner for a given type of data, which exposes it in immutable form and accepts changes only through defined events.
The benefits of this approach are concrete and measurable. Here is what a project gains when the architecture is thought through from the start:
- Testability – the retrieval and model-handling logic can be tested in isolation, without spinning up the whole interface.
- Team scalability – clear boundaries between layers let several people work in parallel without constant conflicts in the code.
- Easier debugging – when data changes are centralized, bugs are easier to track down, because you know where to look.
- Model interchangeability – a properly separated AI layer lets you swap the model provider without rewriting the entire application.
- Onboarding – new team members understand the project faster, because the structure is consistent and predictable.
The consequence of neglecting architecture is always the same. The prototype works and impresses at the presentation, but it cannot be developed further or tested in any sensible way. And when the client asks for a new feature or an integration with yet another system, it turns out to be cheaper to write the project from scratch than to extend the existing one. That is the most expensive scenario there is. And its source is a decision made in the first week – treating AI as a gadget instead of a system with an architecture. At Web Systems we start with layers and boundaries of responsibility precisely so that this scenario never has a chance to happen.
Mistake 2: Weak retrieval and a disorganized knowledge base in RAG solutions
Most corporate AI applications are in practice RAG solutions – the model generates answers based on knowledge pulled from company documents, a product database or a ticket history. And this is where a mistake appears that can ruin an entire project despite the use of the best model: underestimating the search mechanism. Teams focus on choosing the language model and treat retrieval as an obvious thing that can be bolted on at the end. That turns the priorities upside down, because it is the quality of the retrieved context that determines the value of the answer.
The logic here is merciless. The model generates an answer based on whatever the search mechanism hands it. If semantic search over a disorganized knowledge base returns irrelevant fragments, the model will build on them an answer that is confident in form but false in substance. Industry sources put it plainly:
The retrieval mechanism in RAG is critically important. You need the best semantic search on top of a curated knowledge base. If your retrieved information is irrelevant, your generation could be grounded but off-topic or incorrect.
That sentence should hang above the desk of everyone who builds RAG. An answer that is grounded but wrong is more dangerous than no answer at all, because it sounds credible. The user has no way of recognizing that the model relied on the wrong document. And in B2B use cases, where an AI application advises on contracts, products or procedures, such a silent mistake can cost real money or the client’s trust.
Good retrieval is not a single parameter. It is a whole chain of engineering decisions, each of which affects the quality of the final answer. In RAG projects we watch at least these elements:
- Chunking – the way documents are split into fragments. Chunks that are too large lose precision, ones that are too small lose context, and badly chosen ones cut sentences in the middle of a thought.
- Layout parsing – correctly reading the structure of documents, tables, headings and lists before they reach the index. A badly parsed PDF poisons the knowledge base at the source.
- Data curation – removing duplicates, outdated versions and contradictory information that confuse the search and lead to inconsistent answers.
- Quality evaluation – systematically measuring the relevance of the retrieved fragments, rather than eyeballing it after a handful of test queries.
The last point takes us to the heart of the matter. Without metrics there is no control over hallucinations. Model evaluation platforms today assess the generated text and the retrieved fragments using indicators such as groundedness, coherence or question answering quality. These metrics provide a point of reference that lets you optimize RAG deliberately – by configuring the search engine, tidying up the source data, improving layout parsing and the chunking strategy, or refining the user’s query before the search.
A RAG Ops, metrics driven approach like this will help you hill climb to high quality RAG and grounded generation.
The absence of such metrics means the team is flying blind. It does not know whether a change to the chunking strategy improved quality or made it worse, because there is nothing to compare against. Every modification becomes guesswork, and hallucinations appear and disappear without explanation. That is why at Web Systems we treat retrieval and evaluation as the core of a RAG project, not an add-on. The best model on a weak, unmeasured knowledge base will produce a worse result than an average model on a well-curated and regularly evaluated one. It is an investment that pays off in the quality of every single answer.
Mistake 3: Leaving out the costs of maintenance, data and API integrations
The third mistake is about money, but it only reveals itself after months. Quite a few proposals for building an AI application really price only one thing: the cost of access to the model and the time needed to build a demonstration. And the model is often the smallest and most predictable item on the whole bill. The real costs sit in inference under real traffic, in monitoring, in maintenance, in integrations and in working with data. A proposal that says nothing about them is not actually cheaper. It is simply incomplete.
Integrations are particularly often underestimated. An AI application rarely lives in a vacuum. It has to talk to the client’s B2B system, to the e-commerce platform, to the CRM, to existing automations and databases. Every such API integration is concrete engineering work: handling authentication, mapping data, handling errors, request limits, versioning. In projects where this element was treated as an afterthought, the integrations later consumed more time than the AI layer itself – because the client’s systems turned out to be older and less consistent than someone had assumed.
For a conversation about the budget to make sense, you have to put on the table the items that simply do not exist in naive proposals. Here are the real costs we account for when planning an AI application:
- Inference at real volume – the cost of model calls grows linearly with traffic, and in RAG the cost of embeddings and vector database searches comes on top of that.
- Monitoring and observability – logging queries, tracking answer quality, alerts about anomalies and about costs that are getting out of hand.
- Maintaining the knowledge base – data ages, documents have to be updated, reindexed and curated, otherwise RAG quality drops month after month.
- Integrations and their upkeep – the APIs of external systems change versions, the client adds new sources, and every change requires work and testing.
- Model updates – providers retire older models and introduce new ones, which forces migrations, prompt retesting and renewed quality evaluation.
- Security and compliance – access policy reviews, data audits, responding to regulatory changes concerning the processing of information by AI.
Each of these items is a recurring operating cost, not a one-off implementation expense. An AI application resembles a living organism more than a finished product sitting on a shelf. It requires observation, corrections and feeding with fresh data. If the contractor does not communicate this, the client will discover these costs on their own – usually at the moment when the inference invoice turns out to be many times higher than the forecast, or when answer quality starts dropping for no obvious reason.
Tip: Always ask the contractor not about the cost of the rollout, but about the expected total cost of owning the AI application over a twelve-month horizon. Ask for a breakdown into inference, monitoring, knowledge base maintenance and integrations. If all you hear is a price for building the system, and the subject of maintenance is brushed off with a generality, that is the most serious warning signal in the whole conversation. A contractor who thinks in terms of systems will raise the question of operating costs before you get a chance to ask.
At Web Systems we show these items from the start, even if it makes the initial quote look less attractive. We prefer to talk about the full picture of costs, because an AI project settles its accounts not on the day of the rollout, but throughout its entire life. A client who knows the real cost of maintenance makes an informed business decision. And a client lulled by the low price of a demonstration sooner or later pays the difference with interest – and often loses trust in the technology itself, even though the fault lay with the quote, not with AI.
Mistake 4: No strategy for security, data privacy and scalability
The fourth mistake is the most dangerous, because its effects can be irreversible. Quite a few AI applications are built without a thought-through strategy for security and data privacy. In the rush, the team sends sensitive company data to external models – contracts, customer data, internal documentation – with no access policy and no control over what exactly leaves the company’s infrastructure and where it ends up. Convenient while building a demo. In production it means real legal, reputational and business risk.
Security in an AI application starts with the question of which data may leave the client’s environment at all, and which has to be processed locally or anonymized before being sent to the model. You need an access policy defining who may query the system and to what extent, and how those queries are logged. Without it the company loses control over its own information and cannot answer an auditor’s basic question: where and how is its sensitive data processed?
The second neglected issue is scalability. An application that handles ten queries a day beautifully can collapse at a thousand. A production environment has its own resource constraints and variable operating conditions that a demonstration never reveals. Industry material on architecture is a reminder of this nature of the runtime environment, and of how components should handle resources and concurrency:
Mobile devices – even large screen devices – are resource constrained, so at any time, the operating system might stop your app process to give its resources to other processes. Types should be main-safe, meaning they’re safe to call from the main thread without blocking it.
The quote describes mobile devices, but the principle runs deeper and applies to any system under load. Resources are limited, and the environment may take them away from a process at any moment. Types are responsible for their own concurrency policy and must not block the main thread with long-running operations. And in an AI application, the long-running operation is precisely the model call or the vector database search. If those operations block request handling, the system stops responding under load and users see a freeze instead of a result.
That is why a scalable AI application needs well-considered concurrency handling, queueing of model requests, limits and mechanisms protecting against overload. You have to plan what happens when the model provider returns an error or exceeds a limit, how the system degrades gracefully instead of crashing, and how it grows along with traffic. These are not ornaments added at the end. They are the foundation that decides whether the application survives its first day of real load.
Security and scale are areas in which we position Web Systems as a partner who is aware of the risks. Almost two decades of building B2B systems, integrations and web applications have taught us that data and resilience under load are not optional features but a condition for admitting a system to production. We approach AI projects with the same discipline – with an access policy, control over data flow, a scaling plan and failure handling designed before we write the first line of business logic. A client who receives an application without that layer receives risk wrapped in a pretty interface.
Mistake 5: Zero quality evaluation and no measurement of results after the rollout
The fifth mistake closes the circle. The team rolls out the AI application and treats launch day as the end of the project. There is no quality baseline, no metrics for the model’s answers, no way to tell whether the system is working better or worse than it did a week ago. The application goes out into the world and lives a life of its own, and the only signal that something is wrong becomes user complaints. That is quality management in the dark. And with unpredictable language models it is especially risky.
Because the quality of an AI solution is not constant. It changes as the provider updates the model, as the data in the knowledge base ages, as users change the way they phrase their questions. Without measurement you will not catch any of those changes in time. Only by establishing a point of reference, a baseline, and regularly measuring indicators such as groundedness, coherence or question answering quality can you manage the system deliberately instead of putting out fires.
The right approach is RAG Ops – treating quality as an iterative process rather than a one-off rollout. You measure, find the weak points, improve the search configuration, curate the data, tune the chunking or refine the queries, measure again and check whether the change actually helped. It is a methodical climb toward higher quality, in which every decision rests on data rather than on impressions. A one-off deploy without that cycle freezes quality at the level it had on rollout day. And from there it can only go down over time.
Without metrics you also cannot have an honest conversation with the client about results. Does the application actually take work off the support team? Are the answers accurate? Did last month’s fix achieve anything? Those questions can only be answered with numbers. A team that does not collect them relies on anecdotes and hope – and that is not enough to develop a product into which the client has put real money and real expectations.
How do you tell whether a software house will actually build a working AI application?
Pay attention to what the contractor brings up on their own initiative. If they raise the subject of layered architecture, retrieval quality, metrics such as groundedness, twelve-month maintenance costs and data security policy themselves – that is a sign they are thinking about a system, not a demonstration. If, on the other hand, the entire conversation revolves around how impressive the answers the model produces at the demo are, while questions about maintenance, scale and quality measurement are waved away with generalities, you are dealing with a team that builds prototypes, not products. Ask as well for a description of the process after the rollout – a good partner will talk about evaluation, iteration and monitoring as part of the service, not about a one-off handover of the code and the end of the cooperation.
At Web Systems, measuring quality is a natural extension of the engineering discipline we have applied since 2006 in every type of project. An AI application without evaluation is like a system without tests – it may work, but nobody can guarantee it or improve it deliberately. That is why we treat the baseline, the metrics and the iterative raising of quality as part of the rollout rather than a costly extra. They are what turns a one-off effect into a product that gets better month after month instead of worse.
Summary: how to choose a contractor who will deliver an AI application
The five mistakes described here look like different problems, but they share one common denominator: the absence of systems thinking. Treating AI as a gadget instead of a system with an architecture, disregarding retrieval and the knowledge base in RAG, leaving out maintenance and integration costs, having no strategy for security and scalability, and zero quality evaluation after the rollout – all of it grows from the same attitude. The contractor looks at an AI application as an impressive demonstration rather than as a fully fledged IT system that has to be designed, secured, integrated, priced honestly and measured throughout its life cycle.
The antidote is easy to name and harder to implement: good architecture. It is what improves quality, makes the system testable and lets you scale both the product and the team working on it. Industry sources leave no doubt about the benefits of a well-considered structure:
Having a good architecture improves the maintainability, quality, and robustness of the overall app. It lets the app scale – more people and more teams can contribute to the same codebase with minimal code conflicts – and it is easier to test.
The same principles – separation of concerns, a single source of truth, testability, control over resources and costs – decide the success of an AI application exactly as they have decided the quality of web applications, mobile applications and B2B systems over the past dozen or so years. AI invalidates nothing in that equation. It only adds a layer of unpredictability that makes engineering discipline even more important here, not less. So when choosing a contractor, judge them not by the list of models in the proposal, but by whether the team thinks in terms of systems, data, costs and life cycle.
Web Systems is a software house from Łódź that has been designing and delivering web and mobile applications, B2B systems, API integrations, automations, e-commerce and AI solutions since 2006. We approach artificial intelligence projects the way we approach every serious system – with an architecture, measurable quality and a plan for security, scale and maintenance. We do not promise magic. We promise sound engineering and a conversation about the full picture of costs and risks before we build anything. That is how we understand the role of a sensible technical partner for companies that want their AI application to actually work in production, not just at the demo.
Are you planning an MVP, an AI application, an integration with existing systems, process automation or the modernization of an older solution? Let’s talk. Tell us about your idea and your challenges, and we will show you how to build it systematically, securely and in a way that can be maintained and developed for years. Get in touch with the Web Systems team and see what it is like to work with a contractor who thinks about the entire life cycle of your application.
