When Quality Became the Latency Problem

The challenge

The travel webapp generates personalised destination recommendations through a multi-stage AI pipeline.

Recommendation generation was taking approximately 88 seconds. My target was under 40 seconds.

The initial instinct was to optimise the AI generation itself. Instead, I first wanted to understand where the user's waiting time was actually being created.

1. Find the bottleneck

I instrumented the recommendation pipeline and broke the request into its major stages.

Stage	Latency
Setup, authentication, hard filtering, prompt construction	3.7s
Claude decision	5.8s
Claude writing	77.2s
Validation + save + response	1.1s
Total	~88s

Enrichment took approximately 10 seconds, but it was already running concurrently with the writing stage. Since writing was slower, enrichment did not add to the user's wall-clock wait.

The first conclusion was therefore straightforward:

The recommendation pipeline wasn't broadly slow. The Claude writing stage was the critical-path bottleneck.

2. Look inside the bottleneck

The next question was not "How do I make Claude faster?"

It was:

"Why is the writing stage taking 77 seconds?"

An LLM call can consume time through several mechanisms. I then traced the actual model behaviour rather than treating "Claude latency" as one number.

!Source: Gemini

Source: Gemini

3. The 77-second call wasn't actually one call

The writing stage was using a fast-first model strategy:

Haiku → quality check → retry → quality check → Sonnet fallback

In the observed production path, Haiku was failing the quality gate and triggering another attempt before eventually falling back to Sonnet.

The baseline experiment confirmed the problem:

0 of 3 baseline Haiku runs passed the quality check cleanly on the first attempt.

This changed the diagnosis.

Haiku looked like a faster model at the individual-call level, but it was not faster at producing a successful user-facing response.

The system was effectively doing:

~15s Haiku

→ failure

→ ~15s Haiku

→ failure

→ ~35–50s Sonnet

→ success

The result was a writing stage that could easily consume 65–80 seconds.

The key insight

Model latency and user-perceived latency are not the same thing.

A faster model that frequently fails can create a slower product.

4. Quality was creating latency

The next step was to understand why Haiku was failing. Two recurring QA failures emerged:

Failure 1: Missing vibe-tag bullets

The model was expected to reproduce vibe tags that were already known elsewhere in the system.

Failure 2: Invented dollar figures

The model occasionally generated cost figures that were not grounded in the supplied data, triggering a quality guard.

Both failures had something in common:

They were deterministic requirements being delegated to a probabilistic model.

The system already knew the correct vibe tags. The system already knew which financial figures were valid. There was no reason to ask the LLM to reliably reproduce or invent them.

5. Move deterministic work out of generation

The solution was therefore not simply:

"Make Haiku better."

It was:

"Reduce what the LLM is responsible for."

Instead of asking the model to generate the entire recommendation including deterministic facts:

Data + facts + tags → LLM → complete response

the architecture should become:

Verified data → deterministic rendering

and:

Recommendation context → LLM → connective prose

For example, vibe tags can be rendered directly from the user's selected preferences and destination data.

The writing model doesn't need to remember them.

Similarly, cost data should not be presented as something the model needs to calculate or reproduce. Verified figures can be inserted by the application rather than generated by the model.

This turns the LLM task into a much narrower problem:

Take an already-decided recommendation and explain it clearly.

6. Remove wasted computation before trying to make it faster

This led to a broader optimisation principle:

The fastest model call is often the one you don't have to make.

A retry is computation that produced no user value. In this case, the best latency optimisation was therefore not to make retries faster or run them concurrently. It was to remove the conditions causing the retries.

That changes the execution path from:

Generate → fail → regenerate → fail → fallback → succeed

to:

Generate → succeed

A single successful generation is faster by definition than multiple failed attempts followed by a successful one.

7. Revisit model choice only after fixing the task

This doesn't mean Haiku is inherently unsuitable.

The initial Haiku experiment was testing it against a relatively broad writing task with requirements that created unnecessary failure modes.

Once the writing task is reduced to:

• use the provided recommendation
• use verified facts
• generate concise connective prose
• avoid making decisions
• avoid inventing numerical information

it becomes a fundamentally easier task.

That creates a new hypothesis:

A smaller/faster model may perform well once the task has been properly constrained.

So model selection becomes the next experiment, rather than the first solution.

8. The optimisation strategy

The resulting approach is:

1. Remove Haiku's negative production path

Return to a reliable model while the failure mechanism is being fixed.

↓

2. Remove deterministic requirements from the LLM

Render known facts and tags programmatically.

↓

3. Simplify the writing prompt

Make the model responsible only for language generation, not decision-making or factual reconstruction.

↓

4. Re-test Haiku

Measure whether it can now pass the quality gate reliably.

↓

5. Measure the real product outcome

Compare:

• generation latency
• first-pass success rate
• retry rate
• model cost
• output quality

The target is not simply "make Claude faster."

It is:

Produce a high-quality recommendation in under 40 seconds with minimal unnecessary computation.

What I learned

The biggest lesson was that AI latency is not purely a model-performance problem.

My initial framing was:

The AI is slow.

Tracing changed that to:

The writing stage is responsible for almost all of the user's waiting time.

Deeper investigation changed it again:

The writing stage is slow because the system is repeatedly asking models to perform work that is either unnecessary or better handled deterministically.

And the most important insight was:

Quality architecture can directly become latency architecture.

A strict quality gate is valuable only if the system is designed so that the model can reliably satisfy it on the first attempt.

Otherwise, every quality failure becomes another model call, another set of tokens, and another period of waiting for the user.

For AI products, quality, reliability, cost and latency are often coupled - not independent optimisation problems.