An empirical study on the effects of code visibility on program testability

Lei Ma, Cheng Zhang, Bing Yu, Hiroyuki Sato

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an abstract property of programs, testability can be measured by various metrics, which are affected by different factors of design and implementation. In object-oriented software development, code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has some effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed a comprehensive empirical study to shed light on this problem. We first use code coverage as a concrete proxy for testability. We select 27 real-world software programs as subjects and ran two state-of-the-art automated testing tools, Randoop and EvoSuite, on these programs to analyze their code coverage, in comparison with that of developer-written tests. The results show that code visibility does not necessarily have effects on code coverage, but can significantly affect automated tools. Developer-written tests achieve similar coverage on code areas with different visibility, while low code visibility often leads to low code coverage for automated tools. In addition, we have developed two enhanced variants of Randoop that implement multiple strategies to handle code visibility. The results on Randoop variants show that different treatments on code visibility can result in significant differences in code coverage for automated tools. In the second part, our study uses fault detection rate as another concrete measurement of testability. We apply the automated testing tools on 357 real faults. The result of our in-depth analysis is consistent with that of the first part, demonstrating the significant effects of code visibility on program testability.

Original languageEnglish
Pages (from-to)951-978
Number of pages28
JournalSoftware Quality Journal
Volume25
Issue number3
DOIs
Publication statusPublished - Sep 1 2017

Fingerprint

Visibility
Testing
Concretes
Fault detection
Software engineering
Defects

All Science Journal Classification (ASJC) codes

  • Software
  • Safety, Risk, Reliability and Quality

Cite this

An empirical study on the effects of code visibility on program testability. / Ma, Lei; Zhang, Cheng; Yu, Bing; Sato, Hiroyuki.

In: Software Quality Journal, Vol. 25, No. 3, 01.09.2017, p. 951-978.

Research output: Contribution to journalArticle

Ma, Lei ; Zhang, Cheng ; Yu, Bing ; Sato, Hiroyuki. / An empirical study on the effects of code visibility on program testability. In: Software Quality Journal. 2017 ; Vol. 25, No. 3. pp. 951-978.
@article{0be675a80eae453585fb7617b490d4b4,
title = "An empirical study on the effects of code visibility on program testability",
abstract = "Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an abstract property of programs, testability can be measured by various metrics, which are affected by different factors of design and implementation. In object-oriented software development, code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has some effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed a comprehensive empirical study to shed light on this problem. We first use code coverage as a concrete proxy for testability. We select 27 real-world software programs as subjects and ran two state-of-the-art automated testing tools, Randoop and EvoSuite, on these programs to analyze their code coverage, in comparison with that of developer-written tests. The results show that code visibility does not necessarily have effects on code coverage, but can significantly affect automated tools. Developer-written tests achieve similar coverage on code areas with different visibility, while low code visibility often leads to low code coverage for automated tools. In addition, we have developed two enhanced variants of Randoop that implement multiple strategies to handle code visibility. The results on Randoop variants show that different treatments on code visibility can result in significant differences in code coverage for automated tools. In the second part, our study uses fault detection rate as another concrete measurement of testability. We apply the automated testing tools on 357 real faults. The result of our in-depth analysis is consistent with that of the first part, demonstrating the significant effects of code visibility on program testability.",
author = "Lei Ma and Cheng Zhang and Bing Yu and Hiroyuki Sato",
year = "2017",
month = "9",
day = "1",
doi = "10.1007/s11219-016-9340-8",
language = "English",
volume = "25",
pages = "951--978",
journal = "Software Quality Journal",
issn = "0963-9314",
publisher = "Springer New York",
number = "3",

}

TY - JOUR

T1 - An empirical study on the effects of code visibility on program testability

AU - Ma, Lei

AU - Zhang, Cheng

AU - Yu, Bing

AU - Sato, Hiroyuki

PY - 2017/9/1

Y1 - 2017/9/1

N2 - Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an abstract property of programs, testability can be measured by various metrics, which are affected by different factors of design and implementation. In object-oriented software development, code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has some effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed a comprehensive empirical study to shed light on this problem. We first use code coverage as a concrete proxy for testability. We select 27 real-world software programs as subjects and ran two state-of-the-art automated testing tools, Randoop and EvoSuite, on these programs to analyze their code coverage, in comparison with that of developer-written tests. The results show that code visibility does not necessarily have effects on code coverage, but can significantly affect automated tools. Developer-written tests achieve similar coverage on code areas with different visibility, while low code visibility often leads to low code coverage for automated tools. In addition, we have developed two enhanced variants of Randoop that implement multiple strategies to handle code visibility. The results on Randoop variants show that different treatments on code visibility can result in significant differences in code coverage for automated tools. In the second part, our study uses fault detection rate as another concrete measurement of testability. We apply the automated testing tools on 357 real faults. The result of our in-depth analysis is consistent with that of the first part, demonstrating the significant effects of code visibility on program testability.

AB - Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an abstract property of programs, testability can be measured by various metrics, which are affected by different factors of design and implementation. In object-oriented software development, code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has some effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed a comprehensive empirical study to shed light on this problem. We first use code coverage as a concrete proxy for testability. We select 27 real-world software programs as subjects and ran two state-of-the-art automated testing tools, Randoop and EvoSuite, on these programs to analyze their code coverage, in comparison with that of developer-written tests. The results show that code visibility does not necessarily have effects on code coverage, but can significantly affect automated tools. Developer-written tests achieve similar coverage on code areas with different visibility, while low code visibility often leads to low code coverage for automated tools. In addition, we have developed two enhanced variants of Randoop that implement multiple strategies to handle code visibility. The results on Randoop variants show that different treatments on code visibility can result in significant differences in code coverage for automated tools. In the second part, our study uses fault detection rate as another concrete measurement of testability. We apply the automated testing tools on 357 real faults. The result of our in-depth analysis is consistent with that of the first part, demonstrating the significant effects of code visibility on program testability.

UR - http://www.scopus.com/inward/record.url?scp=84991396299&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84991396299&partnerID=8YFLogxK

U2 - 10.1007/s11219-016-9340-8

DO - 10.1007/s11219-016-9340-8

M3 - Article

AN - SCOPUS:84991396299

VL - 25

SP - 951

EP - 978

JO - Software Quality Journal

JF - Software Quality Journal

SN - 0963-9314

IS - 3

ER -