DW.
•AI

What I Learned: From AI Chat Demos to Real AI Products

What I Learned: From AI Chat Demos to Real AI ProductsPhoto on Unsplash

In 2025, I built quite a few demos using Gemini and Kendo UI, building small chat experiences using chatbots combined with Conversational UI for quick prototypes, taking a piece of text (sometimes an image) and sending back a nice-looking answer.

These demos were fun and useful to learn. But at some point I realized something important: getting a chatbot to answer once is not the same as building a reliable AI product that people can trust every day.

I want to share what I learned about the difference between quick demos and real AI features, when not to use LLMs, when they actually make a difference, and a few core concepts you really need.

AI Is More Than Calling a Chat API

My “AI journey” started like many others: I created a chatbot using Gemini or another LLM with the SDK, where you could paste some text or upload an image, and the bot would reply with something that looked smart.

The first time you see it working, it feels magical. You think: “Nice, I’m building AI now.”

But when I tried to move these ideas into a real product, the problems appeared very quickly. The same prompt sometimes produced slightly different answers. Some responses were great, others were confusing or incomplete.

That’s when I understood the key point:

It’s not just about calling a model and getting one answer back. When you integrate AI into a product, you are working with a probabilistic system. You have to think about:

  • how the model behaves in different situations,

  • how you will measure quality, and

  • how you will improve the system over time.

In other words, using the chat API is just the first step. Building an AI feature means designing the whole system around the model so it behaves consistently in the real world.

“But I’m Already Working With AI… Right?”

One thing I quickly learned is that “working with AI” can mean very different things. Simply calling an API that wraps around a model like ChatGPT or any other LLM is not the same as truly integrating AI into your product. There’s a big difference between consuming a model’s API to get a stream of data and actually embedding AI-driven functionality into the core of your application.

In other words, using tools like Cursor, Copilot, or any AI-based coding assistant to write code faster doesn’t mean the AI is part of your product. It simply means you’re using AI to become more efficient as a developer.

When you move from just using AI as a coding assistant to actually building AI-powered features into your product, you’re designing systems that combine models, tools, and data to solve real-world problems. That’s the real distinction.

Not Every Problem Needs an LLM

It’s easy to try to put an “AI” flavor on every product, but another thing I learned quickly is that not every problem needs AI or an LLM.

If you have a simple FAQ or a strict flow, a rule-based system or a decision tree is usually enough. There is no point in adding complexity or cost with an LLM when a simpler solution works better.

Likewise, if you need a fully deterministic system—where the same input must always produce the same output for legal, compliance, or business reasons—an LLM is simply not the right tool. LLMs naturally introduce variation and creativity, which is great for some tasks but terrible when you need strict guarantees.

So the first lesson I learned is this: if you would feel uncomfortable with “sometimes slightly different answers” for the same input, then you probably need a deterministic system, not an LLM.

When LLMs Make Sense

On the other hand, LLMs shine when the problem is fuzzy, language-heavy, or full of messy data. They are great for:

  • transforming one representation into another,

  • classifying text that is informal, noisy, or multilingual, and

  • answering questions grounded in your own documents or data.

Using an LLM for classifying logs and answering contextual questions makes sense because the data is varied and not always clean. The LLM can handle that kind of complexity in a way that simple rules cannot.

Concepts You Actually Need

Let me put it this way: I’m not a machine learning expert, not even close. But over time, I realized that just knowing how to call an API and handle a stream of responses wasn’t enough.

I had to learn why model size matters, the trade-offs between speed and cost, how to manage context windows, and how to think about latency.

One of the key things I had to understand was how LLMs generate tokens. They produce text token by token, and that process is probabilistic. So I had to learn about sampling strategies—why top-k and top-p matter, and how adjusting the temperature changes the output. Lower temperatures mean more stable, predictable answers, while higher temperatures add creativity (and sometimes chaos).

I also had to understand that when you use an LLM, you pay for tokens—both the input tokens (your prompt and context) and the output tokens (the model’s response). Long prompts and long answers mean higher costs, so designing efficient prompts isn’t just nice to have; it’s essential for managing cost and performance.

Resources Gen AI

If you want to go beyond “just calling an API” and really understand how Gen AI works and where it makes sense, these are the resources I recommend:

Conclusion

So, let’s wrap this up. It’s easy to stop at building a simple chatbot that just calls the Gemini API (or any other AI API) and feel like you’ve done the job. But the real takeaway is to go beyond that.

We should focus on learning how LLMs can truly help us make decisions and design systems that solve real problems, not just create another demo.

In other words, the main point is to move past the idea of just calling an API and think about how LLMs fit into the bigger picture. Let’s focus on building products where models, tools, and data come together to create something truly useful.

That’s the real shift: from just playing with AI to truly integrating it into your product.

Real Software. Real Lessons.

I share the lessons I learned the hard way, so you can either avoid them or be ready when they happen.

No spam ever. Unsubscribe at any time.

Discussion