A Practical Framework for Diagnosing AI Product Latency

When an AI product feels slow, the instinct is often to optimise the model: use a smaller model, reduce tokens, or lower reasoning effort. But latency is a systems problem, not simply a model-speed problem. The first step is to understand where the user's waiting time is actually being created and whether that computation is necessary in the first place.

1. Find the bottleneck before changing anything

Start with a latency waterfall. Break the request into its sequential stages:

Input → retrieval → model → tools → validation → response

!image.png

Measure the time spent in each. The goal is to identify: which stage is actually contributing to wall-clock latency?

Without this, optimisation becomes guesswork.

2. Separate sequential work from parallel work

Not every operation contributes equally to total latency.

If three operations take: 5s + 10s + 20s, and run sequentially, the user waits 35 seconds.

If they are independent and run concurrently, the user waits approximately 20 seconds.

Therefore:

Total latency is driven by the critical path, not by the total amount of computation performed.

This is one of the first things to check in any AI pipeline.

3. When an LLM is the bottleneck, decompose the call

An LLM call can be slow for fundamentally different reasons.

Context

The model may be processing more information than necessary.

Question:

Are we giving the model unnecessary context?

Reasoning

The model may be performing more internal reasoning than the task requires.

Question:

Does this task actually require deep reasoning?

Output

The model may be generating excessive amounts of text.

Question:

Are we asking the model to produce more than the product needs?

Retries

The system may be making multiple model calls for one user request.

Question:

How many model attempts does a successful user request actually require?

This decomposition prevents a common mistake: treating "LLM latency" as one undifferentiated problem.

4. Match computation to the job

Different parts of an AI product require different kinds of computation.

Use deterministic systems for things they are good at:

• filtering
• calculations
• hard constraints
• validation
• data retrieval

Use models where qualitative intelligence adds value:

• interpreting ambiguous preferences
• making trade-offs
• generating natural language
• handling unstructured information

The principle is:

Don't use expensive model computation to solve problems your system can solve deterministically.

5. Optimise the critical path, not everything

Once the bottleneck is known, optimise the bottleneck first.

If your pipeline looks like:

Setup: 4s

Decision: 6s

Writing: 60s

Validation: 2s

spending hours reducing setup from 4 → 2 seconds is unlikely to meaningfully change the user experience. Reducing writing from 60 → 20 seconds does.

6. Optimise for successful user requests

One model call's latency is not necessarily the same as the user's experience.

Consider:

Haiku: 10s → failure

Haiku: 10s → failure

Sonnet: 20s → success

The system's model latency may look reasonable at the individual-call level. The user's experience is 40 seconds. Therefore, an AI product should consider: time-to-success, not just time-per-model-call. This makes retries, fallbacks and reliability part of the latency equation.

7. The optimisation hierarchy

When an AI feature is slow, work through this sequence:

1. Locate

Where is the latency?

↓

2. Understand the critical path

What is actually blocking the user?

↓

3. Decompose

Is the bottleneck caused by context, reasoning, output, orchestration or retries?

↓

4. Remove unnecessary work

Can the task be simplified?

↓

5. Reduce computation

Can we use less context, less reasoning, fewer tokens or a smaller model?

↓

6. Preserve quality

Does the optimisation maintain the product's quality bar?

The final goal is not:

minimum latency

It is:

minimum latency for the required level of product quality and reliability.

The broader AI product principle

Latency optimisation is ultimately a resource-allocation problem.

Every additional token, reasoning step, model call, retry and external request consumes computation.

The product manager's job is not to eliminate computation indiscriminately.

It is to ask:

Where does additional computation create meaningful user value, and where is it simply redundant?

That is the difference between making an AI system faster and designing an AI system that doesn't need to do unnecessary work in the first place.