The PM Decisions Behind the Investigation

01 - Instrument before optimising

I refused to assume the model was the bottleneck. I instrumented the full pipeline and used traces to establish where latency was actually being spent.

02 - Remove low-risk waste before trading quality for speed

I first targeted optional API calls and unnecessary generation budget, achieving a 27% latency reduction without changing recommendation logic.

03 - Test architecture before downgrading the model

I separated decision-making from prose generation and moved enrichment off the critical path. The experiment didn't improve total latency, so I rejected the hypothesis rather than forcing the solution.

04 - Treat model quality and validator quality as separate problems

When Sonnet also failed QA, I investigated the validator rather than assuming the model was at fault. This exposed both genuine model failures and false-positive validation.

05 - Optimise successful task completion, not individual model calls

The biggest emerging opportunity was reducing retries, fallbacks and duplicated inference. A faster model is irrelevant if it requires more attempts to produce an acceptable result.

Diagnosing latency in an AI travel-planning system

The travel webapp had a problem that looked straightforward: trip generation was taking 60–120 seconds.

The ambitious target was to bring the typical experience below 30 seconds, without sacrificing recommendation quality. My first assumption was that the model was simply too slow, but it turned out to be much more interesting.

After instrumenting the system and running controlled experiments, I found that latency was not determined by any single component. It emerged from the interaction between LLM generation, external APIs, validation, retries, fallbacks and the architecture around the model.

In several cases, making an individual model call faster actually made the overall system slower because the faster model failed validation and triggered more work.

This became an investigation into a broader AI product question:

How do you optimise an AI system when quality failures themselves become a latency problem?

---

1. Start with the trace, not the model

The first step was to make the generation pipeline observable. The system roughly looked like:

User constraints → deterministic filtering → AI ranking → enrichment → validation → results

The AI ranking stage selected destinations and generated narrative content, while the backend handled deterministic calculations such as costs, dates, visa information and leave-day calculations. The initial baseline was approximately:

• ~90s typical generation
• ~120s estimated P95
• 40–60s for the main Claude ranking call
• 20–40s for downstream enrichment

Rather than immediately replacing the model, I added timing instrumentation to the major pipeline stages and pushed the measurements into Langfuse. This changed the question from: “Why is the AI slow?” to: “Where is the user's 90 seconds actually being spent?”

That distinction proved important.

---

2. First intervention: remove the obvious waste

The first optimisation cycle deliberately focused on low-risk changes.

Optional restaurant context

Restaurant context was useful to the model but was not a hard ranking requirement.

When the system was already consuming too much of its latency budget, I allowed the request to skip the expensive OSM restaurant lookup rather than blocking the ranking call.

Output token ceiling

The ranking call was also producing around 5,800 tokens in typical three-candidate cases, while its maximum was 12,000. I reduced the ceiling to 9,000 (still leaving substantial headroom) to avoid unnecessarily large generation budgets.

Instrumentation

Most importantly, I added phase-level timing so future decisions could be based on traces rather than intuition.

The result:

Metric	Before	After
Average	~90s	66s
Median	~90s	63.9s
Minimum	~80s	51.4s
Maximum	~120s	82.7s

That represented approximately a 27% reduction in average latency, but the system was still nowhere near the 30-second target. The traces pointed towards the next question: Should we change the model?

---

3. Before changing models, change the architecture

The ranking call was doing two fundamentally different jobs:

1. Decide which destinations to recommend
2. Write the user-facing explanation

At the same time, the backend was performing enrichment work that did not fundamentally depend on the prose.

So I tested a different architecture:

Decision → [Prose + Enrichment concurrently] → Merge

instead of:

Decision + Prose → Enrichment

The decision call returned destinations and stops. The prose call generated the narrative. Backend enrichment ran concurrently with prose, meaning expensive enrichment could potentially disappear from the critical path.

This was an important AI systems principle:

Don't just optimise individual operations. Look for work that can be removed from the critical path.

And the mechanism worked. In successful runs, enrichment took 14–28 seconds, but almost all of that work was completely hidden behind the prose call, but the overall system did not become faster. The split introduced a new ~17–18 second decision call, while prose still had substantial variance. Across four successful runs, the split averaged 69.9s, compared with the previous 66s baseline.

The architecture was technically sound. It simply wasn't the right latency lever. That was a useful result. An optimisation that works locally can still fail at the system level.

---

4. The model switch produced a more interesting discovery

The next hypothesis was straightforward:

If the model is the dominant critical-path component, use a faster model.

I moved the prose generation to Haiku. A single Haiku prose call was indeed fast - approximately 15 seconds in production sampling.

But there was a problem. The QA pass rate was effectively zero in the initial sample.

The model generated prose that failed validation for things such as:

• missing required vibe information
• invented dollar figures
• incomplete required content

The system responded by retrying. So the real flow became:

15s Haiku → QA failure → 15s retry → fallback → 35–50s more work

A faster model had not made the product faster. It had made the failure path faster, while triggering the failure path more often.

That was the critical insight.

Model latency cannot be evaluated independently from model reliability.

---

5. I initially blamed the model. The traces proved me wrong.

The natural conclusion was: Haiku is too weak for this task. I tested that assumption by running the prose path on Sonnet. However, Sonnet also failed QA on 5/5 first attempts in the investigation sample. Every run recovered on a retry.

So the problem was not simply:

Haiku = bad quality

It was:

Generation → validator → retry architecture

The validator itself was contributing to the latency. Some of its failures were genuine model problems. Others were false positives.

For example:

• A vibe tag was present in the generated rationale, but the validator only searched specific fields.
• “George Town” was used instead of “Penang”, but the validator did not understand the alias.
• Some “hallucinated destination” checks correctly identified invented comparison destinations.
• Other cases suggested the validator was incorrectly matching a destination's own name as a hallucination.

This changed the remediation strategy again.

Instead of asking:

“How do we make the model generate perfect prose?”

I asked:

“Which failures should the model be responsible for, and which should the system handle deterministically?”

---

6. The retry system itself became a product problem

A particularly useful discovery came from a 173-second trace. It initially looked like an extremely slow fallback. Tracing the actual calls revealed something different: three sequential calls to the same model.

A bug in the fallback guard was reading the wrong model configuration variable, causing the system to believe a fallback was available when it wasn't.

The fix introduced a single source of truth for the prose model. Five subsequent real generations confirmed that the erroneous third call had disappeared. This was a good reminder that AI observability needs to track more than: “LLM call = 30 seconds.”

It needs to tell you:

• Which model?
• Which attempt?
• Why was it retried?
• Was it actually a fallback?
• What validation rule failed?
• Did the retry change anything?

Without that context, a trace can lead you to the wrong conclusion.

---

7. Then came the 20-persona test

At this point, I stopped relying on a handful of manual generations. I built a fixed evaluation harness covering 20 personas and 26 real end-to-end sessions.

The result was sobering:

• 53.8% of sessions succeeded
• Median latency: 76.5s
• Only 23% succeeded cleanly on the first attempt
• Range: 1.0–178.8s

This was valuable because it exposed failure modes that were difficult to see from individual traces. One new issue was particularly important. The round2-fallback decision call could itself violate its output schema. The system then retried the identical prompt twice.

In one case: Invalid schema → retry identical request → same invalid schema → 502

The system had spent tens of seconds trying the same thing without changing the conditions that caused the failure. That is not a model-speed problem. It is a recovery design problem.

---

8. The emerging architecture principle

Across the investigation, the same pattern kept appearing.

Bad recovery

Failure → repeat the same expensive operation

Better recovery

Failure → determine whether the failure is deterministic → repair locally or change the operation

For example:

Failure	Better response
Missing deterministic bullet	Render it from structured data
Invented dollar figure	Strip/repair rather than regenerate everything
Validator false positive	Fix the validator
Invalid destination	Reject deterministically before expensive downstream work
Schema violation	Retry with a corrected constraint, not the identical request
Optional API timeout	Skip or use fallback data
Expensive enrichment	Move it off the critical path where dependencies allow

This led to one of the most important conclusions from the project:

The highest-leverage latency optimisation may not be reducing the duration of an LLM call. It may be reducing how often the system needs to make that call again.

---

9. What I learned as an AI PM

This investigation changed how I think about AI performance.

1. Latency is a system property

A 15-second model can be slower than a 50-second model if the 15-second model fails twice and triggers a fallback.

2. Observability is a product tool

Langfuse was not just monitoring infrastructure. It helped answer product questions:

Where is latency coming from? Which failures create it? Which intervention is worth pursuing?

3. AI systems need deterministic boundaries

The backend already handled hard facts such as dates, costs and visa rules deterministically. The same principle should extend to output validation. If the system already knows the correct answer, don't ask the model to reproduce it perfectly.

4. Validators are part of the product

A validator isn't automatically ground truth. A poorly designed validator can create false failures, retries and latency while making a perfectly good AI response look broken.

5. Experiments should be allowed to fail

The call-splitting experiment did not improve latency. The prompt-trimming experiment did not improve latency and introduced a correctness regression. The Haiku experiment exposed a QA problem rather than solving latency.

These were not wasted experiments. Each removed a hypothesis from the search space.

---

10. The real optimisation target

The original target was simple: Bring generation from ~90 seconds to ≤30 seconds.

I now think about the target differently.

The system should optimise for: Low latency × high first-pass success × low unnecessary computation, rather than latency alone.

The proposed product-level success criteria remain:

• P50 ≤30s
• P95 ≤45s
• P99 ≤60s
• Error rate <1%
• No material deterioration in recommendation quality

But I would also add operational AI metrics:

• First-pass success rate
• Retry rate
• Fallback rate
• Average LLM calls per successful generation
• Validator false-positive rate
• Percentage of latency spent in recovery paths

These metrics tell us why the system is slow, not merely whether it is slow.

---

The takeaway

I started with a performance problem:

“Why does generating a trip take 90 seconds?”

I expected the answer to be model latency. The investigation produced a more useful answer. The product was slow because multiple parts of the AI system amplified one another:

Model generation → imperfect output → validation → retry → fallback → more model generation

The fastest model in isolation was not necessarily the fastest product, the cleanest architecture in isolation was not necessarily the fastest product, and a validator designed to improve reliability could itself become a source of unreliability and latency.

The broader lesson is:

In AI products, performance is not just about making inference faster. It is about designing the entire system so that expensive inference is necessary as few times as possible.

That is the lens I now use when debugging AI product performance:

Trace the system. Form a hypothesis. Change one variable. Measure the result. Disprove your assumptions. Then optimise the failure path.