How I traced a “bad AI recommendation” back to deterministic code

As I did a few user persona walk throughs on my own, I formulated a detailed complaint about the travel webapp.

The recommendations:

• underused their budget
• ignored preferences like pools and kid-friendly options
• kept returning Bangkok and Delhi
• produced only two meaningful trips instead of three
• and optimised holiday dates even though that setting was switched off

At first glance, this looked like a model quality problem; the AI wasn't listening. Despite so, before changing the prompt, I wanted to answer a simpler question:

Did the model actually have the right options to choose from?

That turned out to be the important question.

I traced the recommendation backwards

The system doesn't send every possible destination to the model. There is a deterministic candidate-reduction step before ranking. I inspected the real session's 68 hard-filter survivors and traced them through the pipeline logs.

The result was the root cause: Southeast Asia had 14 valid candidates.

But reduceCandidate() was configured to keep only one destination per region.

So:

14 candidates → Bangkok

South Asia had four candidates:

4 candidates → Delhi

By the time Claude received the candidates, most of the diversity had already disappeared. The model wasn't repeatedly choosing Bangkok from 14 Southeast Asian options.

Bangkok was the only Southeast Asian option it was allowed to choose from. That completely changed the diagnosis.

The fix wasn't a better prompt

I reproduced the exact user request against the real production model and prompt twice before changing anything.

Both times:

Bangkok + Delhi

That gave me enough evidence to separate the problem from model randomness.

I then changed the candidate reduction logic from:

One destination per region

to:

Allocate candidates proportionally according to regional representation.

I also changed amenity fit (pool, gym, kid-friendly, co-working) so it influenced whether a destination survived the candidate cut.

Previously, these preferences existed largely as information in the ranking prompt. Now they influenced the actual candidate pool.

I also found a completely separate bug

While tracing the "travelling with children" preference, I discovered something unexpected - the UI was correctly collecting the information but the backend never read it when creating the session.

So the preference effectively disappeared between:

UI → API → ranking

The model couldn't respect a constraint that never reached it.

This reinforced another principle I now use when debugging AI products:

Trace the data, not just the prompt.

A user-facing AI failure can originate anywhere between the interface and the model.

I deliberately didn't reach for randomisation

One obvious way to break the Bangkok/Delhi repetition would have been to randomise the candidate selection but I chose not to because randomness would create variety, but it wouldn't create personalised variety.

Instead, I made candidate survival sensitive to the user's actual preferences.

That means two users can receive different candidate pools because their constraints are different, while the system remains deterministic and explainable.

That felt like the better product decision.

Variety should come from relevance, not randomness.

The result

The difference was visible in the actual pipeline.

Before

Southeast Asia → 1 candidate

Bangkok, repeatedly.

After

Southeast Asia → 8–10 candidates

The same user profile then produced:

Bali + Chiang Mai + Hanoi, with roughly 48–60% budget utilisation, rather than the repeated Bangkok + Delhi combination.

That was a promising live signal, but I wouldn't call it proof as only one clean live run exercised the entire new path; another run hit an unrelated pre-existing prose QA failure.

So the honest conclusion is:

The fix worked in live reproduction, but we still needed a larger sample to establish that it generalised.

The second bug: when fixing one problem reveals another

A few hours later, another user persona test exposed a different failure. The product returned five trips, despite a documented maximum of three. Again, I started with the real session log rather than guessing.

The pipeline showed:

Round 1 → 3 candidates

One failed post-enrichment budget validation. That correctly triggered the existing round2-fallback. Round 2 then returned another three.

The system concatenated the successful results: 2 + 3 = 5

There was simply no final cap. The same failure also explained a major latency problem: 143 seconds total, because the system had run another full ~66-second cycle.

The immediate fix was deliberately small:

Cap the final merged result at three.

I could have redesigned the fallback to request only the missing number of candidates, which would also reduce latency, but that was a separate optimisation. The immediate product requirement was to restore an existing invariant: Never return more than three trips.

So I fixed that first and left the deeper latency optimisation as the next experiment.

What I took away

These sessions reinforced something I think is particularly important for AI PMs:

Don't assume an AI problem is an AI problem.

When a user says:

“The AI keeps recommending the same places.”

There are many possible causes:

Prompt problem?

Model problem?

Retrieval problem?

Data problem?

Candidate-generation problem?

Deterministic filtering problem?

Validation problem?

Fallback problem?

The right response isn't immediately:

“Let's improve the prompt.”

It's:

“Let's trace the system and find where the user's intent is actually being lost.”

In this case, the model was blamed for a decision it barely had the opportunity to make. The system had already collapsed the candidate space before Claude ever saw it.

The PM lesson

The most useful debugging question I took from this wasn't:

“Why is the model doing this?”

It was:

“At what point did the system stop giving the model the opportunity to do the right thing?”

That distinction matters, because sometimes the best AI improvement isn't a better model, a better prompt, or more sophisticated reasoning. Sometimes it's simply not throwing away the right answer before the AI gets to see it.