Use product metrics to justify investment
The correct set of metrics, as well as aligning the product team on the right outcomes, is critical. It’s best to use product metrics to justify spend, terminate projects, or accelerate delivery to meet market opportunities. Furthermore, outcomes are influenced by backlog priorities. Product metrics are even more crucial for the team to align with a clear understanding of what to build first.
Many customer-centric design philosophies often omit the needs of the business; after all, priorities need to be balanced for the customer to succeed and the business to make money. Aligning customer-centric and business priorities is an essential step in establishing product metrics. Failure to align on these priorities results in products that appear to perform well through metrics, but often suffer negative consequences long term, such as, customer attrition.
Misaligned objectives
A great example is Groupon.com, a service that helps consumers save money through virtual coupons. Efficiencies are reached due to volume, and in theory, the platform allows the business to attract and retain a large number of new customers quickly. In theory, the influx of new customers makes up for the lost profits from offering products/services at a discounted rate.
Due to the popularity and viral nature of the platform, however, Groupon attracts (and partially created) a new type of bargain hunter. This new type of consumer does not demonstrate brand loyalty, and so participating businesses struggle to demonstrate ROI. While Groupon.com has evolved and adjusted its business practices, this is an excellent example of how keeping an eye on both customer success as well as the business objectives become critically important for long term growth.
Targeted outcomes and metrics
Product teams should establish outcomes that are specific, measurable, and achievable. Capturing 50% of total market share may be specific, but highly unlikely and thus not a great target outcome.
Targeting three to five outcomes is reasonable. Having a smaller set of outcomes allows the team to recall the target and recalibrate if necessary, quickly. Once the product goes through various releases, outcomes can adjust and evolve.
Example outcomes for Tier 1 bank
A lending group at the bank identifies outcomes for the rebuild of their legacy loan processing application. They include:
Speed. The average time the customer application takes to cash disbursement reduces from three weeks to five business days. This could be achieved through automated decisioning, improved collaboration, and simplified workflows.
Automation. 90% of the applications are approved or rejected using an automated decisioning engine based on historical evidence and data provided by underwriters.
Self-service. The product allows actuaries to design, build, and test new risk models within the platform. Since the legacy platform handles risk models as code in the application, this is a simple one to measure and achieve.
The power of quantitative and qualitative data
Product metrics combine quantitative and qualitative data to inform the product team throughout delivery. Quantitative data typically originates from a product analytics tool such as MixPanel, Adobe Analytics, or Google Analytics. Having quantifiable data helps the product team establish a baseline, monitor for trends, and inform the future roadmap. Leveraging the earlier case study as a reference, the healthcare provider uses quantitative data on a per-sprint basis to evaluate the adoption of the product.
Activity: percentage of operators that have logged into the platform
Activity: percentage of total users logged in
Activity: total downloads, downloads per week, delta
Adoption: number of messages exchanged through the platform, delta per week
Adoption: unique logins per week
ROI: open shifts created in the platform per week
ROI: shifts filled through the platform
These examples are just a subset of all variables monitored through time.
Qualitative metrics, on the other hand, provide the means for a dialogue with end-users and customers. Capture these metrics automatically through surveys, star ratings, reviews, feedback boxes, user interviews, or job shadowing. Acknowledging that all feedback is biased, reference multiple data sources, and critically evaluate the information to inform the product roadmap. Data gathered through more extensive usability studies trumps a vocal minority.
Bringing this all full circle, outcomes, and ROI from a product can only be recognized when the software is live and in front of users. This idea is the cornerstone of all agile methodologies, funding techniques, and organizational structures. The longer the product is stuck in development, the higher the risk, the less aligned it is to real market needs.