The demo looked great. The real thing didn't.
Your product team ships an AI feature, maybe a "smart" onboarding checklist, a churn-risk score on the customer dashboard, an in-app assistant that's supposed to nudge users toward the next best action. It works beautifully in the demo, because the demo runs on three test accounts with clean, deliberate data. Then it goes live on your real 4,000 accounts and the recommendations are wrong half the time. Support tickets show up asking why the assistant told someone to "complete step 2" when they finished step 2 three weeks ago. Your team's instinct is to blame the model. It's rarely the model. It's that nobody built clean usage data for the AI to read from in the first place.
This is the pattern we see over and over with SaaS founders: the AI feature isn't the hard part. Getting usage data structured enough for AI to trust it is.
Why this happens
Most SaaS products didn't start with a data plan. They started with a feature list. Someone added mixpanel.track('button_click') in year one. Someone else added a different event name for basically the same action in year two, because they didn't know the first one existed. Support logs live in Intercom. Billing state lives in Stripe. Product usage lives in your app database, maybe partially mirrored into Amplitude or Segment, maybe not.
None of these systems agree on what a "user" is. Your app database has a user_id. Stripe has a customer_id. Intercom has its own contact ID that only sometimes maps back correctly, especially after someone changes their email or a teammate gets added to an account. If you're multi-tenant, this gets worse. An event might be tagged to a user but not reliably tagged to the account or workspace they belong to, which means you can't even answer "what does this account do with our product" without a manual join that half the time returns nulls.
Nobody sat down and decided any of this. It accumulated, feature by feature, integration by integration. That's normal. It's also exactly the condition an AI feature cannot tolerate.
Why the usual fixes don't work
The first instinct is usually to hire a data scientist or bring on an AI vendor and hand them the raw event stream. This fails almost every time, for a boring reason: the model can only be as good as the labels and identity resolution underneath it, and nobody cleaned those up. You end up with a very expensive person building a very sophisticated model on top of a user_id column that's wrong 15% of the time.
The second instinct is to buy an off-the-shelf AI layer: a plug-in recommendation engine, a churn-prediction SaaS tool, an LLM wrapper that promises to "just work" on your existing analytics. These tools are usually fine. The problem is what you feed them. If your event names are inconsistent (signup_complete vs SignupCompleted vs onboarding_finished), the tool either ignores half your signal or double-counts it. If timestamps aren't normalized to one timezone, the model can't tell whether an action happened before or after another one. Vendors will tell you their tool handles messy data. It handles messy data the way a translator handles a language they don't speak: badly, and confidently.
The third instinct, and the most common one we see, is to retrofit analytics after the AI feature ships. Ship first, instrument later. This means your first three months of AI output are built on incomplete data, users form an opinion about the feature during those three months, and that opinion is usually "this thing doesn't know what it's talking about." You don't get many chances to earn back trust in an AI feature once it's told a customer something obviously wrong.
What "clean usage data" actually means
This phrase gets thrown around loosely, so here's the concrete version. Clean usage data for an AI feature means four things are true:
- One identity, everywhere. A single canonical ID for a user and a single canonical ID for the account/workspace they belong to, and every system (app, billing, support, analytics) resolves back to those same two IDs. Not "mostly resolves." Reliably.
- One event taxonomy. A documented list of what events exist, what they're named, and what properties they carry, enforced somewhere (a schema, a tracking plan, a lint check in CI) rather than left to whoever's writing the code that week.
- Timestamps and tenancy on everything. Every event carries when it happened and which account it belongs to, in a consistent format, with no silent nulls.
- A single place it all lands. Not four analytics tools each with a partial view — one warehouse or database that the AI feature actually reads from, even if the raw events still flow through multiple tools upstream.
If you can't currently write one SQL query that tells you, accurately, what a specific account did in the last 30 days across product, billing, and support — you don't have clean usage data yet. That's the honest test.
The tradeoffs, plainly
Fixing this before building the AI feature costs you time up front, usually four to eight weeks depending on how tangled your systems are, and it's not a glamorous project. Nobody gets excited pitching "we spent a month reconciling user IDs" to a board. The AI feature is the fun part; the data plumbing is not.
But skip it and you pay in a worse currency: a shipped feature that erodes trust, an expensive rebuild six months later, and a team that's now gun-shy about AI because "we tried that and it didn't work." We've watched founders spend $40-60k on an AI feature that had to be quietly turned off within a quarter, not because the model was bad, but because the data feeding it never was reliable enough to act on.
The honest tradeoff: slower start, durable result, versus fast start, feature you might have to retract. For a customer-facing AI feature, slower and durable wins almost every time. For an internal, low-stakes experiment nobody outside your team will see, moving fast on messy data is a reasonable bet. Know which one you're making before you start.
A practical checklist before you build
Before your team writes a line of AI feature code, walk through this:
- Pick your canonical IDs, one for user, one for account, and confirm every system (app, Stripe, Intercom, support) can resolve to them today, not "in theory."
- Write down your event taxonomy in one document. If two events mean the same thing, merge them. If an event has no clear owner or use case, cut it.
- Route events into one place your AI feature will actually query — a warehouse, a Postgres schema, whatever fits your scale. Segment or RudderStack can handle the routing; the destination is what matters.
- Backfill at least 60-90 days of history where you can, so the AI feature isn't starting cold with zero context.
- Only then define what the AI feature does: recommend, predict, summarize. Now you know what data it can actually see.
Do these in order. Teams that reverse steps 4 and 5 end up building a feature spec around data they don't actually have, then scrambling to make the data fit the spec.
Where AI genuinely helps once this is done, and where it doesn't
Once usage data is clean, an in-product AI assistant gets genuinely useful fast. It can tell a customer success rep which of their 200 accounts are actually at risk, based on a real drop in login frequency plus a support ticket spike plus a billing downgrade. Not a guess. An actual pattern across connected data. It can summarize what a specific account has done in the last month without someone manually pulling three reports.
What it still won't do well is compensate for a broken process. If your onboarding flow is confusing, an AI assistant that explains the confusing flow more articulately doesn't fix the drop-off — it just narrates it. If your product has no clear "aha moment" defined, no amount of usage data will let an AI model tell you what that moment is; that's a product decision, not a data problem. AI is very good at finding patterns in data you've made legible. It's not good at deciding what matters in data you haven't organized yet, and it's not a substitute for fixing a workflow that's broken regardless of who, or what, is running it.
Where to start
If you're staring at an AI feature roadmap item and a nagging feeling that your data isn't ready for it, that feeling is usually right. The fix isn't glamorous, but it's also not as large a project as it sounds once someone maps it out.
That's what a Process Teardown is for: a free 30-minute session where we look at one specific workflow or feature you're trying to build and show you, concretely, where the data is fragmented and roughly how many hours or how much rework it's costing you before you even get to the AI part. No pitch, no obligation. If you want to see what this looks like once it's done, our case studies walk through real SaaS and business teams whose scattered tools we connected into one system before layering AI on top.
0 Comment