Establish the framework
Selecting the right metrics, working toward the desired outcomes, drives adoption. A metrics framework fosters bidirectional communication for setting goals, attacking blockers, and communicating success. When designing the framework, consider the context and objectives of three distinct audiences:
The working agile team (engineers and designers)
The product manager (the person handling a single work stream)
The portfolio owner (the CIO or CPO)
A metrics framework facilitates bidirectional communication. Avoid pointing fingers. Focus on enabling teams to succeed.
Classic project management techniques teach the project management triangle. Its rigid structure implies that scope, cost, and time are interdependent. In this model, increasing scope for a project increases either cost or time, or both. Strategically, this approach fails to consider the effectiveness of the product (Is the right thing being built?), the quality of the product, and the time to market. Blindly following the budget, time, scope paradigm, and so on compromises the results.
As an alternative, try a square construct to monitor the product’s quality, customer value, velocity, and organizational effectiveness.
Customer value establishes what’s important for users to inform and guide the product build. Customers can be external or internal. Use the build-measure-learn loop to keep the team aligned to outcomes.
Quality measures escaped defects, performance KPIs, and the level of debt accrued. Intentionally sacrifice go-to-market speed (e.g., intentional debt) or industrialize for a mass rollout and mission-critical applications, which in turn lowers overall velocity.
Velocity determines the product go-to-market and feature release speed. This facet helps the team and product leaders project and plan throughput in a given time period.
Organizational effectiveness tracks the health of delivery and people processes (e.g., retention, communication, and engagement). As a result, the enterprise achieves more through better processes, higher engagement from the team, less churn, and the like.
Use the factors outlining the product square as boundaries. A product team needs to decide which of the four variables to invest in, which to sacrifice, and which to appropriate depending on the maturity of the product.
CUSTOMER VALUE
| METRIC | USE | PRIORITY |
|---|---|---|
| Business outcome | Sets specific, measurable, achievable goals (e.g., loan approval reduces from two business weeks to one). | High |
| NPS | Considers customer advocacy to understand what’s valuable for promoters. | Low |
| Customer health score | Tracks renewal, churn, depth of usage, growth of an account, etc. | Medium |
| Product debt | Logs, prioritizes, and estimates the backlog product debt. | Medium |
| Revenue channels | Evaluates the ability of the product to transact revenue through a channel. | Medium |
QUALITY
| METRIC | USE | PRIORITY |
|---|---|---|
| Escaped defects | Helps the team understand the quality of the development output and testing strategy. | High |
| Test coverage automation maturity | Determines the approach to test coverage, tools, methods, and acceptable coverage ratios. | Medium |
| Technical debt | Monitors the various types of technical debt accumulated over time. | Medium |
| Application performance | Calculates page load times, concurrency limitations, job run times, etc. | Medium |
| Downtime and volatility | Indicates the overall stability of the platform and volatility to help the team reflect on improvements made through infrastructure updates and the testing strategy. | High |
VELOCITY
| METRIC | USE | PRIORITY |
|---|---|---|
| Sprint velocity | Demonstrates the quantity of software a team ships within a sprint. | Medium |
| Velocity trending | Shows process improvement or degradation of team velocity (e.g., automation, DevOps, technical debt). | High |
| Feature to production time (backlog aging) | Assesses how quickly a feature can be productized and released to market. | Low |
| Estimate accuracy | Rates how confident and realistic the team is when estimating backlog stories. High estimate volatility requires root cause analysis to help the team improve. | Low |
OPERATIONAL EFFECTIVENESS
| METRIC | USE | PRIORITY |
|---|---|---|
| Maturity score | Notes product best practices as well as technical maturity of the delivery framework. | Medium |
| DevSecOps metrics | Monitors uptime, security, infrastructure stability, environment build times, and others. | Medium |
| Team health | Evaluates the team’s burnout, sentiment, and ability to succeed. | Medium |