Metrics: Assessing Quality

Metrics enable better understanding of the business and are essential for decision support.  Here are some considerations when selecting metrics for use in assessing quality.

Enterprise quality metrics such as Net Promoter Score are important to the business to understand overall product and business performance quality.  Composite software test specific metrics are also helpful to understand the state of software quality.  For example, the organization may be interested in knowing the INC (internal non-conformance) rate compared to ENC (external non-conformance or customer identified defects).  Defect metrics get a lot of attention because they’re easy to report on and easy to understand.

Operation performance metrics are useful in assessing the ability to produce and deliver quality software on time.  The organization may want education to accompany the metrics to understand what the metrics mean and why they’re important.  For example, unit test coverage growth metrics can provide insight and understanding of the quality of the development process but some basic insight on what unit tests are would provide context on what the metrics mean.

The Scaled Agile Framework provides an excellent metrics model for the organization to use including:

1. Iteration level metrics (velocity, test cases, story acceptance, etc.)

2. Release level metrics (program velocity, automation rate, etc.)

3. Enterprise level metrics (velocity vs. capacity, feature delivery, escalation rate, etc.)

When assessing software release readiness, a weighted defect trend showing submit rates weighted by severity (impact) and priority (urgency) can be a leading readiness indicator.  If test activity remains at a steady state, scope has been maintained, support training is occurring, SRN (Support Release Note) candidates are being composed, and the defect submit trend is sustained downward, then it may be reasonable to conclude that readiness has been achieved or is approaching.  A defect submit double hump trend is a regular release cycle pattern after feature complete; it occurs as defects and secondary issues from the repairs are wrung out and is also a readiness indicator.

What metrics do you find are the most effective in assessing quality?