Automatic topic classification of test cases using text mining at an Android smartphone vendor

Junji Shimagaki, Yasutaka Kamei, Naoyasu Ubayashi, Abram Hindle

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Background: An Android smartphone is an ecosystem of applications, drivers, operating system components, and assets. The volume of the software is large and the number of test cases needed to cover the functionality of an Android system is substantial. Enormous effort has been already taken to properly quantify "what features and apps were tested and verified?". This insight is provided by dashboards that summarize test coverage and results per feature. One method to achieve this is to manually tag or label test cases with the topic or function they cover, much like function points. At the studied Android smartphone vendor, tests are labelled with manually defined tags, so-called "feature labels (FLs)", and the FLs serve to categorize 100s to 1000s test cases into 10 to 50 groups. Aim: Unfortunately for developers, manual assignment of FLs to 1000s of test cases is a time consuming task, leading to inaccurately labeled test cases, which will render the dashboard useless. We created an automated system that suggests tags/labels to the developers for their test cases rather than manual labeling. Method: We use machine learning models to predict and label the functionality tested by 10,000 test cases developed at the company. Results: Through the quantitative experiments, our models achieved acceptable F-1 performance of 0.3 to 0.88. Also through the qualitative studies with expert teams, we showed that the hierarchy and path of tests was a good predictor of a feature's label. Conclusions: We find that this method can reduce tedious manual effort that software developers spent classifying test cases, while providing more accurate classification results.

Original languageEnglish
Title of host publicationProceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018
PublisherIEEE Computer Society
ISBN (Electronic)9781450358231
DOIs
Publication statusPublished - Oct 11 2018
Event12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018 - Oulu, Finland
Duration: Oct 11 2018Oct 12 2018

Publication series

NameInternational Symposium on Empirical Software Engineering and Measurement
ISSN (Print)1949-3770
ISSN (Electronic)1949-3789

Conference

Conference12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018
CountryFinland
CityOulu
Period10/11/1810/12/18

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'Automatic topic classification of test cases using text mining at an Android smartphone vendor'. Together they form a unique fingerprint.

  • Cite this

    Shimagaki, J., Kamei, Y., Ubayashi, N., & Hindle, A. (2018). Automatic topic classification of test cases using text mining at an Android smartphone vendor. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018 [a32] (International Symposium on Empirical Software Engineering and Measurement). IEEE Computer Society. https://doi.org/10.1145/3239235.3268927