At Web Systems we have been building web applications, B2B systems and integrations for almost twenty years. When generative AI started shaking up the market, we said to ourselves: let us find out first hand. Not from someone else’s case studies, not from conference reports. On our own software house in Łódź, on a living organism. For three months we tested AI tools in three departments: customer service, marketing and development. Each department had different pain points, but the common denominator was simple – piles of repetitive tasks eating up the time of people who should be doing something smarter. What came out of it? A lot and a little at the same time. On some processes the time savings reached several dozen percent. But the road to those savings? Data preparation, team training, endless tinkering with prompts. I am writing this so you do not have to repeat our mistakes. If you are wondering whether AI will actually cut costs in your company, here you will find answers from the field, not from a sales brochure.
Spis treści
Why we picked exactly these three departments for the AI tests
Before we ran any model in a production environment, we had to decide where to test. We set three criteria: plenty of repetitive tasks, a decent volume of text to generate or process, and the ability to measure the effects with hard KPIs. Customer service, content marketing and the development team – all three fit perfectly. Consultants answered dozens of similar enquiries every day. Copywriters churned out articles and descriptions that required research. Developers wrote unit tests and technical documentation – tedious work, but someone has to do it.
Setting up the test environment took two weeks. We measured the baseline metrics: average response time to a customer ticket, how many hours a blog article swallows, how much time a developer burns on tests and documentation. Those figures became our “before”, so that after three months we would have a solid comparison. The departments worked in parallel but independently of each other, so that the results of one would not distort the other.
We selected the tools to match the nature of the tasks in each area. Specifically:
- Customer service – a RAG chatbot based on the Claude API with a vector database of company knowledge, integrated with the ticketing system
- Content marketing – GPT-4o as a copywriter’s assistant for generating drafts, outlines and headline variants with human verification
- Development team – GitHub Copilot for generating code and unit tests, plus the Claude API for code review and writing technical documentation
- Analytics layer – our own dashboard for tracking productivity and quality metrics in real time
Thanks to that we compared not only performance but also the cost of the individual APIs – how much each department’s budget really goes on them.
Customer service – a RAG chatbot instead of canned replies
Classic chatbots built on decision trees and ready-made templates? For years they have annoyed both customers and support teams. We know that from experience. We went with RAG – Retrieval-Augmented Generation – which means searching our knowledge base combined with answer generation by a large language model. We loaded in technical documentation, FAQs, the history of resolved tickets and internal procedures. More than two thousand documents sliced into chunks, turned into embeddings and pushed into a vector database.
And that is where the real work began. Because the quality of the retrieval mechanism is the foundation of the whole solution. If the system pulls the wrong chunks, the model produces an answer that is smooth in language but off the mark in substance. Completely off. That is why we spent most of the time cleaning up the source data: standardizing formats, removing duplicates, tagging by topic, testing different chunking strategies. We measured coherence, relevance and answer quality, and then improved iteratively. No guesswork.
The retrieval mechanism in RAG is critical. You need the best semantic search over a carefully prepared knowledge base so that the retrieved information is relevant to the query. A metric-driven approach – RAG Ops – based on measuring the coherence, relevance and groundedness of answers lets you iteratively optimize the quality of generated content.
– Google Cloud, Gemini Enterprise Agent Platform documentation
The results after three months? On simple queries, better than we expected. The average first response time dropped by 62%. Consultants got rid of roughly 40% of repetitive questions about statuses, procedures and typical technical problems. But – and this is an important “but” – complex cases still required a human. The RAG chatbot did not replace an experienced specialist. What it did was free that specialist up for the cases that genuinely need thinking.
Tip 1: The quality of the knowledge base determines the effectiveness of RAG more than the choice of language model. Before the rollout, spend at least a week auditing, deduplicating and structuring your source documentation – investing in data pays off many times faster than experimenting with different LLMs.
Marketing and content – generating text under human control
In our marketing team AI came in as an assistant, not a replacement. And that distinction turned out to matter enormously. Copywriters started using GPT-4o for first drafts of articles, for building outlines from a brief and for generating headline variants for A/B tests. The content creation process itself did not change structurally – research at the start, editing at the end – but the individual stages sped up. Research in particular. AI can gather and synthesize in a dozen seconds information that would burn an hour of a person’s time.
The numbers from our measurements: time to produce a first draft, minus 45%. Topic research, minus 55%. Preparing headline variants and meta descriptions, which used to take a copywriter twenty minutes, dropped to three minutes including verification. All in all, the content marketing team got back an average of twelve working hours a week. We were able to increase content output by 60% without hiring anyone new. But, to be honest, raw model output always needed a human edit. Always.
And that is where the traps come in. The model produced convincing-sounding statistics that turned out to be invented once we checked them. Seriously, they looked perfectly credible. Quotes attributed to real people? Fabricated. The tone of the text without a good instruction in the prompt? Generic and stripped of any brand character. Every article had to be combed through for figures, proper names and references to sources. Without that stage, publishing AI content is asking for trouble – both for your image and for your credibility.
Tip 2: Always verify AI-generated data before publication, especially statistics, quotes and references to research. Language model hallucinations are particularly dangerous because they sound credible. Introduce a mandatory fact-checking stage in your editorial process, no matter how good the generated text looks.
Development team – AI in code review and documentation
Our developers approached AI with the most scepticism of all three departments. And here is the paradox – they are exactly where we saw the most stable time savings. We rolled out GitHub Copilot for everyday coding and the Claude API for automating code review and technical documentation. Where did AI deliver the most? In the tasks that experienced developers consider tedious but necessary. Writing unit tests, refactoring repetitive patterns, docstrings for existing code. That kind of work.
Generating unit tests – here the models do surprisingly well. After reading a function signature and a few existing tests, Copilot suggested coverage for edge cases the developer had not thought of straight away. Time spent writing tests fell by about 35%, and code coverage went up, because the barrier to entry – hammering out test boilerplate – practically disappeared. Technical documentation with the Claude API? It needed smaller corrections than the marketing content. Probably because technical context is unambiguous and less prone to hallucinations.
But it has to be said plainly where AI does not cope. Architectural decisions, choosing design patterns, complex business logic that requires understanding the domain context – in those areas the models’ suggestions were shallow or outright misleading. We checked it. A developer who tried to hand microservice architecture design over to AI got solutions that were syntactically correct but strategically naive. The models do not grasp the trade-offs between scalability and complexity, they do not know the client’s infrastructure and they will not predict the maintenance consequences of their own proposals. They simply will not.
Tip 3: Treat AI like a capable junior – fast and effective on simple, well defined tasks, but requiring review on every bigger challenge. Never merge AI-generated code without a review by an experienced developer, especially in the areas of security, authorization and data operations.
Hard numbers – a summary of the savings after three months
After a quarter of testing we gathered the data from all the departments and compared it with the baseline metrics. The results confirmed what we felt intuitively – AI brings real savings, but their scale depends heavily on the nature of the tasks and the quality of the prepared data. Here is the breakdown:
- Customer service: first response time cut by 62%, escalations of simple enquiries down by 40%, savings of around 30 working hours per month per consultant, customer satisfaction measured by NPS up by 8 points
- Content marketing: time to draft an article cut by 45%, research sped up by 55%, 12 working hours a week recovered across the team, content output increased by 60% without an extra headcount
- Development team: time spent writing unit tests cut by 35%, technical documentation produced 50% faster, test coverage up by 15 percentage points, savings of an average of 8 hours a week per developer
And the costs? API subscriptions (Claude API, GPT-4o, GitHub Copilot), time spent integrating with existing systems, team training. Around 2,800 PLN a month in total for a twelve-person team. Given the value of the working time saved, the investment paid for itself in the second month. Sounds great, right? But that calculation leaves out the hidden costs, which are easy to miss at the start.
The indirect costs are engineers’ time spent building integrations, regularly updating the RAG knowledge base, constantly refining prompts and putting out fires after the model providers change their APIs (yes, that happens more often than we would like). Maintaining a RAG system requires continuous care for the data. Every new procedure, change in the offer, documentation update has to reach the vector database. Otherwise the chatbot gives outdated answers. These costs are not dramatic, but they grow with the scale of the rollout and you must not leave them out when calculating ROI.
The most common mistakes when rolling out AI in a company
Over three months we collected enough slip-ups to fill a decent list of warnings. Mistake number one, and the most serious – launching AI tools without defining success metrics first. Without a reference point you cannot objectively judge whether the rollout is delivering value or just burning money. Measure the current state before you switch anything on: task completion time, number of errors, unit cost of the process. Then compare. Many managers skip that step and later cannot justify the project to the board. I have seen it more than once.
Mistake number two – trying to automate everything at once. A company that rolls out AI in five departments simultaneously usually ends up with five unfinished pilots instead of one successful one. A better strategy? One area with the highest return potential. You get it to stable results. Only then do you scale. We started with three departments at once because we had the engineering resources – most companies do not have that comfort and should start with a single pilot.
The third trap is data security and GDPR. Sending customer data to external language model APIs without a legal review? That is a risk that can cost more than all the savings from AI put together. Check the data processing terms of every provider, consider anonymizing sensitive information before sending it to the model, talk to a data protection lawyer. And the fourth problem is the lack of a human in the loop, meaning blind trust in AI output. Every model hallucinates. Every one makes mistakes. The only question is whether someone catches them before it is too late.
FAQ
How much does an AI rollout cost in a small or medium company and where should you start?
It depends on the scale and the approach. The language model APIs alone come to around 200-800 PLN a month for a small team – the Claude API and GPT-4o bill per token, GitHub Copilot costs about 80 PLN per user. The bigger item tends to be integration with existing systems and data preparation – with a RAG solution that can be anywhere from a few days to a few weeks of engineering work. How to start? A pilot in one department. Pick the area with the largest number of repetitive tasks, measure the baseline metrics, roll out a single tool and assess the results after a month. If the pilot confirms the savings, expand. This approach minimizes financial risk and lets you learn on a small scale before you put serious money into full automation.
Summary
Three months of testing AI at Web Systems showed one thing beyond any doubt – AI is a powerful tool, but not a magic wand. Real savings appear when you have properly prepared data, clearly defined metrics and a person keeping an eye on the whole thing. The RAG chatbot took the load off our consultants, but weeks of treating the knowledge base were the precondition. The copywriter’s assistant sped up content production, but every text still passed through human hands. AI in the development team? It handles tests and documentation brilliantly. But architectural decisions? That is still the domain of experienced engineers. And it will be for a long time yet.
Do not ask whether AI pays off. Ask where in your company repetitive tasks are eating up the time of people who could be doing something more valuable. Start the pilot there, measure the effects and only then decide about scaling on the basis of data. Companies that approach this methodically – a pilot in one department, gradual expansion – achieve better results than those trying to automate everything at once.
If you are considering an AI rollout in your company, whether that means a chatbot for customer service, content marketing automation or support for your development team, get in touch with us at Web Systems. We will help you plan a pilot tailored to the specifics of your business, integrate AI tools with your existing systems and measure the real savings. Since 2006 we have been building solutions that work in production, not on presentation slides.
