Stop building data products no one needs
A product manager’s take on measuring value instead of vanity
“We delivered the dashboard on time”
“The pipeline ran successfully”
“Usage is up 37%”
These are all metrics of activity, but do they measure value?
The dashboard delivered on time may never be opened again. A successful pipeline might be deprecated next quarter. Increased usage could just mean higher computing costs, without driving any real impact like new customers or reduced risk. These are examples of signals and metrics that feel productive, but often drift into the category of vanity metrics; numbers that look good but don't mean anything unless tied to meaningful outcomes.
So how do we distinguish value from vanity?
In this article, we’ll explore how to measure the true value of a data product, and a few principles to guide better product management in the world of data and analytics.
Data Products tend to be misled through vanity metrics
Let’s consider a simple analogy. A construction project made up of three components: the building, the carpenter, and the hammer.
To evaluate the success of the project, we focus on the building. Was it completed within budget? Are there defects? Are the occupants satisfied? These are outcome metrics. They reflect the end impact of all the effort.
Now, consider the carpenter. If we want to evaluate the carpenter's performance, we might ask: Did they build according to spec? Did they complete the job on time? Was their craftsmanship living up to expectations? These are metrics tied to intermediate outcomes, directly contributing to the final result.
And then we have the hammer. How do we evaluate the success of a hammer in this context? We certainly don’t judge it by whether the building was completed. That’s too far removed. Instead we ask: Did it help the carpenter work faster, with less strain, more precision? Was it durable and usable?
Alright, let’s get to the point: data products are hammers.
Data products are not a building. They are rarely even a carpenter. Data products (regardless of if it’s raw data, an API or a dashboard) serve someone who is doing a job - like taking decisions, building models, or prioritizing risks. That “someone” is the data product’s user, and they are the closest equivalent to the carpenter in our analogy.
If we measure success only by the hammer’s durability or whether the handle is made of fine mahogany, we risk losing sight of its purpose. Likewise, the data product should not only be evaluated upon number of users or its uptime. The real question is: Did the data product help the user make a better decision? Drive a specific action? Avoid a costly mistake?
These are the metrics we desire to evaluate a data product’s value upon, but they are harder to figure out compared to metrics such as usage and adoption rates. So here’s a few principles that may be of help, when figuring out what metrics to evaluate your data product upon.
1. Start with the problem and the people
Before you build anything, clarify:
What problem exist?
What decisions are being made to remediate it?
What questions do those decision makers need to answer?
This aligns the data product to a clear user need. Teams that develop data products should deeply understand their users – their goals, workflows and frustrations just like any other product. Ask yourself: if the data product doesn’t help solve a problem, how else is it valuable?
2. Frame a clear value hypothesis
If [user] uses [data product], then [decision/action] will improve, resulting in [business objective].
Using a template like this one may seem silly, but it forces clarity. You’re not building for usage metrics. You’re building to affect a relevant business outcome. If this is hard to define, is there really value in the data product you’re planning to develop?
3. Define the Value Stream
To measure the success of a data product, you should first understand the value stream it supports: the series of activities that ultimately deliver value. This begins with mapping who the end-users are, what they’re trying to accomplish, and how your product enables that outcome.
A data product rarely delivers value on its own. It enables decisions, actions, or processes - often by improving access to information, automation, or insight. The key is to measure success based on how well the product contributes to that downstream value.
Rather than measuring output (dashboards built, pipelines deployed, queries ran), focus on measurable change in the user’s ability to do something valuable. For example:
Did the product help users decide or prioritize faster?
Did it result in decisions that improved performance?
Did it reduce risk, improve targeting, or unlock new business opportunities?
Your metrics should reflect outcomes, not just activity. And those outcomes should trace back to the value stream - what the end-user needed to achieve, and how your product helped make that happen.
In summary
Vanity metrics might help you feel productive. But they don’t help you improve. Data product teams should adopt a product mindset working towards real users, real problems, and real outcomes.
Because if it doesn’t change anything, it wasn’t worth building.
