When an AI Recommendation Is Wrong, Don't Blame the Model First

A user asked the travel webapp for a five-day trip with a $5,000 budget.

The system recommended Beijing for five days at roughly $2,633, with Delhi and Athens as alternatives.

At first glance, this looked like a recommendation-quality problem.

It wasn't.

I broke the complaint into individual failure modes and traced each one through the product pipeline. What looked like “AI giving bad recommendations” was actually a combination of dead product logic, broken data connections, prompt ambiguity, and unsupported model-generated claims.

That investigation changed how I think about AI product quality:

When an AI output is wrong, debug the system before debugging the model.

One bad answer can have many causes

The five complaints initially looked like one recommendation problem:

• The budget was significantly underused.
• The trip was shorter than requested.
• Singapore holiday overlap wasn't working.
• No multi-city option was offered.
• Delhi was labelled as a winter destination despite having no winter data.

Tracing them separately revealed very different causes.

Budget / duration:

A duration optimiser already existed, but it was dead code. The function that should have scaled trip duration according to budget was never connected to the constraint-building pipeline.

Holiday logic:

The holiday functionality existed, but the real generation call sites weren't passing the user's country. The feature was effectively dead during actual generation.

Multi-city:

This wasn't a recommendation or database problem at all. Multi-city was correctly gated by trip duration, and there were enough reachable East Asian destinations to support it.

Vibe / activity grounding:

The UI exposed 21 fixed interest categories, while destination data used a completely different vocabulary across interest_tags, vibe_keywords and seasonal_events. The model was being asked to reason over a taxonomy that wasn't properly connected to the product's interface.

Unsupported prose:

The model was also capable of turning sparse destination information into specific-sounding narrative claims that weren't grounded in the underlying data.

Five symptoms. Five different failure mechanisms.

The lesson: AI quality is a system property

This is where AI products differ from how we often talk about AI.

When an LLM produces a bad answer, the natural reaction is:

“The prompt needs improvement.”

Sometimes it does, but the model is only one component in the chain:

User input → constraints → data → retrieval → ranking → prompt → model → enrichment → validation → UI

A failure anywhere along that chain can appear to the user as an “AI problem.” That means an AI PM needs to ask: Where did the system first become wrong?

Not: Why did the model give the wrong answer?

That distinction changes the investigation completely.

Fix the right layer

In this case, two fixes belonged in engineering:

1. Wire budget-aware duration calculation into the actual generation pipeline, so the model receives a duration range that reflects the user's budget.
2. Fix holiday-country resolution, including Singapore and US holiday data, and ensure the correct country reaches every holiday calculation.

Another problem was deliberately not fixed immediately.

The destination-vibe problem required deeper research into the right calculation, retrieval and data-grounding architecture. Rather than prematurely implementing a solution, I turned it into an outsourced research brief specifying:

• what correctness and coverage require
• what the product already has
• what remains unknown
• what needs further research

That was an important product decision too. Not every discovered problem should immediately become an implementation task.

The more dangerous bug isn't the crash

One of the most interesting bugs was in the holiday resolver.

The system had different parts of the product passing country information in different formats: some passed full country names, while the generation pipeline passed two-letter country codes.

A naïve normalisation fix could have caused "SG" to be treated as an unsupported input and silently fall back to "US". Nothing would necessarily crash. The system would simply return valid-looking but incorrect holiday information. That's much more dangerous than an obvious error.

For AI products, I increasingly care about this distinction:

A failed request is visible. A plausible wrong answer is not.

This is why I prefer systems that fail honestly when information is unavailable rather than silently filling gaps with assumptions.

What this changed in my approach to AI PM work

The biggest lesson was about where product judgement is most valuable in AI. AI can write code, analyse logs and suggest fixes. But someone still needs to decide:

• What is actually broken?
• Which layer owns the problem?
• What evidence do we need?
• What should we fix now?
• What should we investigate first?
• What should we explicitly leave out of scope?

The PM's job is increasingly less about having all the answers and more about structuring the problem so the right answer can be found efficiently. Before improving the model, understand the system that surrounds it.