Defect Density

Owned by Natallia_Mendzel@epam.com
Last updated: Aug 01, 2022 by Anastasia Baranouskaya
4 min read

Purpose

Defect Density is the ratio of defects per software "size". Although there are known industry averages about typical defect densities per different programming languages and software frameworks, everyone would agree the less defects in the software the higher its quality. So, measuring it is the first step to improve the quality. 

Defect Density metric in PERF shows a ratio of defects to a software size. PERF provides several views of this metric:

  1. Defect Density per Story Point

  2. Defect Density per non-Defect Item

  3. Production Defect Density per release

How metric helps

Defect Density helps to understand a relative "level of quality" of the software. Being, by nature, normalized against the size of the delivered software, this metric can allow a comparison of different modules or even projects between each other. So, among other benefits it helps prioritize investments into quality improvements.  

In general, Defect Density can help with -

  • identify the areas for improvement (a piece with the biggest defect density requires more attention and resources for quality improvements)

  • compare quality of components/software modules

  • shows the ratio of defects logged by internal team per delivered items (Ndefects/Nitems)

  • shows quality of application

  • shows quality of testing

  • identifies what area should be treated with higher attention

  • shows how effective changes that were done (if any) in QA process

How metric works

Chart overview

Chart shows a Defect Density (Axis Y) over months (Axis X).

 

On hover over a column a hint appears with the following info:

  • Month - a calendar month a metric is calculated for;

  • Defects submitted - a number of defects both internal and external submitted in a considered month excluding defects satisfying criteria in Project Settings>Task Tracking System>Quality Management>Criteria for escaped defects;

  • SP delivered (w/o defects) - software size in Story Points excluding defects in a considered month;
    OR

  • Non-defects accomplished (w/o defects) - software size in items excluding defects in a considered month;

  • Defect density - metric value (rounded to 2 decimal places).

On hover over a column a hint appears with the following info:

  • Month - a calendar month a metric is calculated for;

  • External defects - a number of external submitted in a considered month satisfying criteria in Project Settings>Task Tracking System>Quality Management>Criteria for escaped defects;

  • Affected versions - number of released versions in a considered month.

  • Defect density - metric value (rounded to 2 decimal places).


Chart legend shows the following:

  • the last calculated metric value;

  • a difference with the previous month.

Top problems metric identifies 

  1. There is no Test Plan and/or Strategy

  2. There is no Regression testing in place, lack of Automated Testing

  3. Team misbalance (skills ratio)

  4. Low quality of testing

  5. Technical and Quality Debt is out of control

  • Some areas might need complete refactoring instead of building new functionality on the top of them

  • Definition of Done for features is not defined properly or missed

  • Limitations in testing, i.e. some complex flows cannot be tested from user perspective, or some test data cannot be generated

  • Sprint or / and Product goal or / and business context is not well known by the team

  • Requirements management process is not mature enough, so that improper A/C have been captured, implemented and tested

  • There are issues with code review

Calculation 

Defect Density per SP

Defect Density per item

Defect Density per release

Defect Density per SP

Defect Density per item

Defect Density per release

Defect Density per SP = (Ndefects  - Ninvalid)/SPitems - SPdefects closed,

where

Ndefects - a number of created defects in a considered month;

Ninvalid - a number of invalid defects in a considered month in any status;

SPitems - estimate in SP for all completed items in a considered month;

SPdefects closed - estimate in SP for all closed defects in a considered month.



Sub-items are not included into the calculation.

RAG thresholds: Red - metric value >= 3; Amber - 1 <= metric value  < 3; Green - metric value < 1.

Defect Density per Item = ( Ndefects  - Ninvalid)/Nitems - Ndefects closed,

where

Ndefects - a number of created defects in a considered month;

Ninvalid - a number of invalid defects in a considered month in any status;

Nitems - all completed items in a considered month;

Ndefects closed - all closed defects in a considered month.



Sub-items are not included into the calculation.

RAG thresholds: Red - metric value >= 3; Amber - 1 <= metric value  < 3; Green - metric value < 1.

Defect Density per release = Ntop external defects /Nreleases,

where

Ntop external defects - a number of external defects of a top priority (any status) in the versions released in a considered month, i.e. affected version a defect is assigned is released in this month. Top priority defects are defined in  Project Settings>Task Tracking System>Quality Management>Top-priority defects;

Nreleases- number of released versions in a considered month.



Sub-items are not included into the calculation.

RAG thresholds: no.

 

Data Source

Data for the metric can be collected from a task tracking system (Jira, TFS, Rally, etc.).