The impact of code review coverage and code review participation on Software quality: A case study of the Qt, VTK, and ITK projects

Shane McIntosh, Yasutaka Kamei, Bram Adams, Ahmed E. Hassan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

105 Citations (Scopus)

Abstract

Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.

Original languageEnglish
Title of host publication11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings
PublisherAssociation for Computing Machinery, Inc
Pages192-201
Number of pages10
ISBN (Electronic)9781450328630
DOIs
Publication statusPublished - May 31 2014
Event11th International Working Conference on Mining Software Repositories, MSR 2014 - Hyderabad, India
Duration: May 31 2014Jun 1 2014

Publication series

Name11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings

Other

Other11th International Working Conference on Mining Software Repositories, MSR 2014
CountryIndia
CityHyderabad
Period5/31/146/1/14

Fingerprint

Inspection
Students
Defects

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Software

Cite this

McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2014). The impact of code review coverage and code review participation on Software quality: A case study of the Qt, VTK, and ITK projects. In 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings (pp. 192-201). (11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings). Association for Computing Machinery, Inc. https://doi.org/10.1145/2597073.2597076

The impact of code review coverage and code review participation on Software quality : A case study of the Qt, VTK, and ITK projects. / McIntosh, Shane; Kamei, Yasutaka; Adams, Bram; Hassan, Ahmed E.

11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings. Association for Computing Machinery, Inc, 2014. p. 192-201 (11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

McIntosh, S, Kamei, Y, Adams, B & Hassan, AE 2014, The impact of code review coverage and code review participation on Software quality: A case study of the Qt, VTK, and ITK projects. in 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings. 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings, Association for Computing Machinery, Inc, pp. 192-201, 11th International Working Conference on Mining Software Repositories, MSR 2014, Hyderabad, India, 5/31/14. https://doi.org/10.1145/2597073.2597076
McIntosh S, Kamei Y, Adams B, Hassan AE. The impact of code review coverage and code review participation on Software quality: A case study of the Qt, VTK, and ITK projects. In 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings. Association for Computing Machinery, Inc. 2014. p. 192-201. (11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings). https://doi.org/10.1145/2597073.2597076
McIntosh, Shane ; Kamei, Yasutaka ; Adams, Bram ; Hassan, Ahmed E. / The impact of code review coverage and code review participation on Software quality : A case study of the Qt, VTK, and ITK projects. 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings. Association for Computing Machinery, Inc, 2014. pp. 192-201 (11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings).
@inproceedings{fb3807943c044775a555dedb22a89134,
title = "The impact of code review coverage and code review participation on Software quality: A case study of the Qt, VTK, and ITK projects",
abstract = "Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.",
author = "Shane McIntosh and Yasutaka Kamei and Bram Adams and Hassan, {Ahmed E.}",
year = "2014",
month = "5",
day = "31",
doi = "10.1145/2597073.2597076",
language = "English",
series = "11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings",
publisher = "Association for Computing Machinery, Inc",
pages = "192--201",
booktitle = "11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings",

}

TY - GEN

T1 - The impact of code review coverage and code review participation on Software quality

T2 - A case study of the Qt, VTK, and ITK projects

AU - McIntosh, Shane

AU - Kamei, Yasutaka

AU - Adams, Bram

AU - Hassan, Ahmed E.

PY - 2014/5/31

Y1 - 2014/5/31

N2 - Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.

AB - Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.

UR - http://www.scopus.com/inward/record.url?scp=84938797513&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84938797513&partnerID=8YFLogxK

U2 - 10.1145/2597073.2597076

DO - 10.1145/2597073.2597076

M3 - Conference contribution

AN - SCOPUS:84938797513

T3 - 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings

SP - 192

EP - 201

BT - 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings

PB - Association for Computing Machinery, Inc

ER -