When we as product managers launch a new product, our goal is for that product to be successful. We want users to get value from the product and to see the product meet its overall objectives. Otherwise, we would not spend the time developing product strategies, visions, and roadmaps.
But how do we know if our product is, in fact, successful? With data and analytics, we can create and monitor specific metrics related to a product, we can determine if the product is on track to meet its stated goals or if the product is falling short and we need to change course.
Data and analytics are topics at the center of many business discussions today including those in the world of product management. As such, product managers may find themselves asking what product data they should be concerned with, how they should be using this data, and so on. As an experienced product manager, I would argue that product managers should approach measuring products through two lenses: the ‘traditional’ user behavior and engagement metrics, as well as metrics related to a product’s stated objectives and outcomes. For purposes of this article, I will refer to the former as product analytics and the latter as outcomes-based analytics. A metric will refer to the specific measure being used for either.
Product Analytics
Most product managers are familiar with product analytics, which uncover end-user behavior and value.Product analytics focus on usage statistics and measure a product’s day-to-day performance as well as how end-users interact with the product. For example, product analytics allow product managers to answer the following questions:
How many users are engaging with the product regularly or in a meaningful way?
Can end-users use the product to complete the intended transaction?
Which features are the most popular?
Is the product growing/retaining/monetizing?
Product analytics are typically focused on measures of quantity and quality and will use metrics such as counts, rates, and dollars to measure end-user behavior or the performance of a specific feature.
For example, suppose you are the product manager for an online bike sharing application that connects individuals with available bicycles to individuals looking to rent bicycles. To measure end-user engagement you may measure the percent of rental transactions completed or the percent of users logging onto your app each week. To determine which features are most valuable to users you might measure page traffic or click rates of specific features. To measure growth or retention you might measure new or returning active users. The exact metrics used will be based on the product itself, but the goal of product analytics is to track these daily activities.
Outcomes-Based Analytics
Product managers will be familiar with the concept of developing product objectives, but measuring progress against these is something I have found to be less common.
Outcomes-based analytics measure if the product delivers its objectives. They are tied back to a product’s vision and are used to help determine whether a product is accomplishing its strategic objective. Outcomes analytics often look at data over time or make comparisons. Additionally, these metrics may be focused on the performance of the product’s end-user as the interest lies in measuring what end-users can achieve by using the product.
Again, using an online bicycle sharing application as an example, let’s say the objective of the product is to easily connect bike owners to nearby renters and popularize biking as a mode of transportation. One metric could be the average number of days a bike goes without being rented. If this number is small this could be evidence that the product is easily and efficiently connecting bike owners to renters, thus achieving its main objective. Another outcome we might consider measuring is the average distance between the bike owner and renter. If renters are having to travel a large distance to get to the bike they want to rent, then the product is not meeting its objective of connecting nearby owners and renters. Finally, a metric like the increase in bikes on the road in a specific city or the number of bike rides facilitated through the product could be used to determine if the product is meeting its objective of popularizing biking as a mode of transportation.
Product Analytics + Outcomes-Based Analytics
Together, product analytics and outcomes-based analytics track progress to success. Product analytics and outcomes-based analytics should be equally important because, as product managers, we are concerned with understanding how end-users engage with our product as well as if end-users can use the product to accomplish whatever the product was designed to do. Focusing only on product analytics neglects to measure a product’s overall impact on end-users, while focusing only on outcomes-based analytics neglects to measure how end-users are using the product.
If our goal is to see our products succeed in solving the business issues they were developed for, then we need to ensure we are tracking progress to success along the way, and the way we track success is through both product analytics and outcomes-based analytics.