Defect Leakage

Owned by Natallia_Mendzel@epam.com
Last updated: Jan 18, 2022 by Anastasia Baranouskaya
2 min read

Purpose

Defect Leakage shows an amount of defects not detected by a testing team i.e. "leaked" to Production. PERF provides 2 kinds of this metric:

  • calculated for defects of a top priority (where "top priority" is specified in a project configuration; quite often it is a combination of Blockers and Criticals)

  • calculated for defects of any priority

How metric helps

Defect Leakage helps to evaluate the efficiency of a software testing process in the project: the lower the value of defect leakage, the better is the quality of software testing.

Some of the reasons why a defect leakage might happen:

  • Absence of stringent test cases review in a project

  • Differences between various Environments on the same project (e.g. production, pre-prod, integration, testing, etc.)

  • Missing test cases to cover appropriate conditions and/or edge cases

  • Poorly designed test cases due to misunderstood requirements

  • Incorrect deployment in UAT or production i.e. infrastructure/packaging issues

  • Use of incorrect test data

How metric works

Chart overview

Chart shows Defect Leakage as an amount of defects (Y axis) over months (X axis). 

 

On hover over a column a hint appears with the following info:

  • Month - a calendar month a metric is calculated for;

  • Escaped (external) defects - defect satisfying Criteria for Escaped defects;

  • Escaped (external) invalid defects - defect satisfying Criteria for Escaped defects and Criteria for Invalid defects;

  • Detected (internal) defects - a number of not external defects;

  • Detected (internal) invalid defects - a number of not external defects satisfying Criteria for Invalid defects;

  • Total - a number of defects both internal and external submitted in a considered month.

Chart legend shows the following:

  • the last calculated metric value

  • a difference with the previous month

By click on a column a pop up appears with the following information got from the task tracking system:

  • Defect ID

  • Type

  • Priority

  • Summary

Calculation

Defect Leakage = Next. defects  - Ninvalid ,

where

Next. defects - a number of created external defects in a considered month. External defect satisfies criteria set in Project Settings>Task Tracking System>Quality Management>Criteria for Escaped defects.

Ninvalid - a number of invalid defects in a considered month. Invalid defect satisfies criteria set in Project Settings>Task Tracking System>Quality Management>Criteria for Invalid defects.

RAG thresholds: Red - metric value >= 3; Amber - 1 <= metric value < 3; Green - metric value < 1.

Defect Leakage (top priority) metric is calculated in exactly the same way as Defect Leakage but the scope of defects is limited to top priority defects only. Top Priority Defects criteria is set in Project Settings > Task Tracking System (e.g. JIRA) > Quality Management > Top-priority defects.

Data Source

Data for the metric can be collected from a defect tracking system (Jira, TFS, Rally, etc.).