Building an AI product changes what “good analytics” means.

Traditional product analytics can often stop at questions like How many people clicked? or How many users returned? For an AI product, that is not enough. A feature can execute successfully, return within its latency target, and still produce an answer users do not trust or find useful.

For my travel webapp, I therefore designed the MVP measurement framework around a simple chain:

System reliability → AI quality → User value → Retention

The goal is not to instrument every possible interaction. It is to understand whether the AI system is producing outcomes that users value, and diagnose where it fails when it does not.

1. Start with the product's value, not the events

The first mistake I wanted to avoid was building an analytics system around button clicks to track users actions. A click tells me that something happened, but it does not necessarily tell me that the product worked.

For a travel planning product, the core journey is:

Visit → Plan → Receive recommendations → Evaluate → Save → Act → Return

That led me to define the primary MVP value metric as:

Planning Session → Saved Trip Rate

The percentage of completed planning sessions that result in at least one saved trip.

This is a much stronger signal than DAU.

DAU tells me people are using the product. A saved trip tells me that the recommendation was compelling enough for someone to keep.

DAU therefore remains an important health metric, but not the product's primary value metric.

2. Measure the funnel, not isolated events

The core product funnel becomes:

Visit → Plan Started → Plan Completed → Results Viewed → Saved → Downstream Action → Returned

Each stage answers a different question.

Stage	What it tells me
Visit	Are people arriving?
Plan started	Are they interested enough to try?
Plan completed	Can the product successfully produce an answer?
Results viewed	Did the system technically deliver?
Saved	Did the recommendation create enough value to keep?
Downstream action	Did the user want to do something with it?
Returned	Did the experience create lasting value?

This also makes debugging much easier.

If plan completion is high but save rate is low, I don't have an acquisition problem. I likely have a recommendation quality or product experience problem.

If plan start is high but completion is low, the problem may instead be UX, latency, reliability, or funnel friction.

3. AI products need a separate quality layer

One of the most important distinctions in the framework is:

A technically successful AI response is not necessarily a good AI response.

For the travel webapp, I split plan quality into three layers.

Layer 1 - Technical quality

Did the system work?

• Plan generation success rate
• API failure rate
• Timeout rate
• P50/P95 latency
• Results rendering success
• Retry rate

Layer 2 - Constraint correctness

Did the system follow what the user asked for?

• Budget respected
• Dates valid
• Requested number of trips generated
• Leave constraints respected
• Required preferences satisfied
• Required amenities satisfied

This produces a Constraint Satisfaction Rate:

Percentage of generated plans satisfying all required constraints.

Layer 3 - User-perceived quality

Did the user actually like the result?

• Save
• Thumbs up/down
• Regenerate
• Regeneration reason
• Itinerary engagement
• Calendar
• Share
• Booking intent

This creates a useful progression:

Did it run? → Did it follow instructions? → Did the user value it?

4. Treat regeneration as a quality signal

AI products have a particularly useful behavioural signal that traditional products often do not: the user can ask the system to try again.

For the travel webapp, users can request:

• Different location
• Cheaper options
• Different vibes
• Too boring
• Find a better fit
• Start over completely

I don't want to treat these as generic button clicks. They are diagnostic feedback about the recommendation system.

For every regenerated plan, I therefore capture:

• Original plan ID
• Parent plan ID
• Regeneration reason
• Generation sequence
• Resulting plan
• Whether the new plan was saved
• Whether the user regenerated again
• Whether the user abandoned the experience

This allows a much more interesting metric:

Regeneration Improvement Rate

Percentage of regenerated plans that produce a better downstream outcome than the original.

For example:

Original → Different Vibes → New Plan → Save

is evidence that regeneration successfully corrected a mismatch.

But:

Original → Different Vibes → New Plan → Regenerate Again → Abandon

suggests the system still failed to understand the user's intent.

This turns regeneration from a UX interaction into an AI evaluation signal.

5. Measure recommendations at the object level

A travel plan contains multiple destinations. If I only measure plan-level save rate, I lose information about which individual recommendations are actually working.

So I also track: Destination Impression → Click → Itinerary → Save → Booking

This allows me to calculate:

• Destination CTR
• Destination save rate
• Click → save rate
• Destination dismissal rate
• Destination booking engagement

This creates a feedback loop for the recommendation system.

For example:

Destination A is repeatedly surfaced but rarely clicked.

versus:

Destination B is clicked frequently and disproportionately saved.

Those are fundamentally different signals for improving ranking and destination selection.

6. Activation is more meaningful than sign-up

Another important distinction is between registration and value realisation. A user signing up doesn't mean they experienced the product.

For the MVP, I define activation around successful product usage:

User successfully generates a trip and takes a meaningful downstream action.

The strongest version is: Generate plan → Save trip

I also track:

Time to First Value

• First visit → first successful plan
• First visit → first saved trip

This helps distinguish a product that merely attracts users from one that gets users to value quickly.

7. Retention should be connected back to value

I track:

• D1
• D7
• D30 retention

But aggregate retention isn't enough.

The more useful question is:

Do users who experience value return more often?

So retention should be segmented by:

• Activated vs non-activated
• Saved vs non-saved
• Generated a plan vs did not
• Signed-in vs anonymous

This allows me to test the fundamental product hypothesis: Users who find useful recommendations should be more likely to return.

8. Build identity correctly before trusting the metrics

Analytics quality is only as good as identity resolution.

The MVP supports both:

• Authenticated user_id
• Anonymous anonymous_id/webID

A returning anonymous user should not suddenly become a "new user" because they have not signed in yet. Likewise, when an anonymous user signs up, their previous activity should ideally be associated with their authenticated identity.

This matters particularly for:

• Retention
• Activation
• Conversion funnels
• New vs returning analysis

I therefore treat anonymous ID persistence and identity merging as analytics infrastructure, rather than product KPIs.

9. Standardise event context

Rather than creating a unique event for every button, I use common semantic events with contextual properties.

For example:

plan_start

with:

entry_point = top_nav

or:

entry_point = home_banner

or:

entry_point = results_regenerate

The event schema should consistently capture:

• User
• Session
• Page
• Component
• Entry point
• Plan
• Destination
• Generation type
• Regeneration reason
• Experiment/version

This makes the analytics system flexible enough to answer questions I haven't thought of yet.

10. The resulting measurement architecture

The final framework looks like this:

Product health

DAU / WAU / MAU

↓

Activation

Plan completion → First saved trip

↓

AI quality

Technical success → Constraint satisfaction → User feedback

↓

Product value

Save → Itinerary → Calendar / Share / Booking

↓

AI learning

Regeneration reason → New recommendation → Outcome improvement

↓

Long-term value

D1 / D7 / D30 retention

The important part is that these layers connect.

11. What I deliberately did not optimise for

A mature analytics system should also know what not to prioritise.

I don't want the MVP dashboard dominated by:

• Raw button clicks
• Profile page visits
• Number of UI interactions
• Generic session length
• Vanity engagement metrics

These can be useful diagnostics, but they are not evidence that the AI product is working.

The hierarchy should be: Value > quality > engagement > activity

A user saving one genuinely useful trip is more meaningful than a user generating five plans and rejecting all of them.

12. The principle

The framework ultimately comes down to one question:

Can I trace a user's behaviour back to the quality of the AI experience and forward to meaningful product value?

For an AI product, analytics should not end at: "The model returned successfully."

It should allow me to follow the entire chain:

System worked → AI followed constraints → User found it useful → User acted on it → User came back.

That is the measurement architecture I want to build into the travel webapp from MVP rather than retrofitting it after launch.

The objective isn't to collect more data. It's to make every important metric answer a product question.