Go beyond velocity with advanced product metrics 

Configuring a comprehensive framework of metrics that drive results

Download white paper

Establish the framework

Selecting the right metrics, working toward the desired outcomes, drives adoption. A metrics framework fosters bidirectional communication for setting goals, attacking blockers, and communicating success. When designing the framework, consider the context and objectives of three distinct audiences:

  1. The working agile team (engineers and designers)

  2. The product manager (the person handling a single work stream)

  3. The portfolio owner (the CIO or CPO)

A metrics framework facilitates bidirectional communication. Avoid pointing fingers. Focus on enabling teams to succeed.

Classic project management techniques teach the project management triangle. Its rigid structure implies that scope, cost, and time are interdependent. In this model, increasing scope for a project increases either cost or time, or both. Strategically, this approach fails to consider the effectiveness of the product (Is the right thing being built?), the quality of the product, and the time to market. Blindly following the budget, time, scope paradigm, and so on compromises the results.

As an alternative, try a square construct to monitor the product’s quality, customer value, velocity, and organizational effectiveness.

Customer value establishes what’s important for users to inform and guide the product build. Customers can be external or internal. Use the build-measure-learn loop to keep the team aligned to outcomes.

Quality measures escaped defects, performance KPIs, and the level of debt accrued. Intentionally sacrifice go-to-market speed (e.g., intentional debt) or industrialize for a mass rollout and mission-critical applications, which in turn lowers overall velocity.

Velocity determines the product go-to-market and feature release speed. This facet helps the team and product leaders project and plan throughput in a given time period.

Organizational effectiveness tracks the health of delivery and people processes (e.g., retention, communication, and engagement). As a result, the enterprise achieves more through better processes, higher engagement from the team, less churn, and the like.

Use the factors outlining the product square as boundaries. A product team needs to decide which of the four variables to invest in, which to sacrifice, and which to appropriate depending on the maturity of the product.

CUSTOMER VALUE

METRICUSEPRIORITY
Business outcomeSets specific, measurable, achievable goals (e.g., loan approval reduces from two business weeks to one).High
NPSConsiders customer advocacy to understand what’s valuable for promoters.Low
Customer health scoreTracks renewal, churn, depth of usage, growth of an account, etc.Medium
Product debtLogs, prioritizes, and estimates the backlog product debt.Medium
Revenue channelsEvaluates the ability of the product to transact revenue through a channel.Medium

QUALITY

METRICUSEPRIORITY
Escaped defectsHelps the team understand the quality of the development output and testing strategy.High
Test coverage automation maturityDetermines the approach to test coverage, tools, methods, and acceptable coverage ratios.Medium
Technical debtMonitors the various types of technical debt accumulated over time.Medium
Application performanceCalculates page load times, concurrency limitations, job run times, etc.Medium
Downtime and volatilityIndicates the overall stability of the platform and volatility to help the team reflect on improvements made through infrastructure updates and the testing strategy.High

VELOCITY

METRICUSEPRIORITY
Sprint velocityDemonstrates the quantity of software a team ships within a sprint.Medium
Velocity trendingShows process improvement or degradation of team velocity (e.g., automation, DevOps, technical debt).High
Feature to production time (backlog aging)Assesses how quickly a feature can be productized and released to market.Low
Estimate accuracyRates how confident and realistic the team is when estimating backlog stories. High estimate volatility requires root cause analysis to help the team improve.Low

OPERATIONAL EFFECTIVENESS

METRICUSEPRIORITY
Maturity scoreNotes product best practices as well as technical maturity of the delivery framework.Medium
DevSecOps metricsMonitors uptime, security, infrastructure stability, environment build times, and others.Medium
Team healthEvaluates the team’s burnout, sentiment, and ability to succeed.Medium

Continue to:Product value metrics: Drive business outcomes