Investigating the Effects of Balanced Training and Testing Datasets on Effort-Aware Fault Prediction Models

Kwabena Ebo Bennin, Jacky Keung, Akito Monden, Yasutaka Kamei, Naoyasu Ubayashi

研究成果: 著書/レポートタイプへの貢献会議での発言

14 引用 (Scopus)

抜粋

To prioritize software quality assurance efforts, faultprediction models have been proposed to distinguish faulty modules from clean modules. The performances of such models are often biased due to the skewness or class imbalance of the datasets considered. To improve the prediction performance of these models, sampling techniques have been employed to rebalance the distribution of fault-prone and non-fault-prone modules. The effect of these techniques have been evaluated in terms of accuracy/geometric mean/F1-measure in previous studies, however, these measures do not consider the effort needed to fixfaults. To empirically investigate the effect of sampling techniqueson the performance of software fault prediction models in a morerealistic setting, this study employs Norm(Popt), an effort-awaremeasure that considers the testing effort. We performed two setsof experiments aimed at (1) assessing the effects of samplingtechniques on effort-aware models and finding the appropriateclass distribution for training datasets (2) investigating the roleof balanced training and testing datasets on performance ofpredictive models. Of the four sampling techniques applied, the over-sampling techniques outperformed the under-samplingtechniques with Random Over-sampling performing best withrespect to the Norm (Popt) evaluation measure. Also, performanceof all the prediction models improved when sampling techniqueswere applied between the rates of (20-30)% on the trainingdatasets implying that a strictly balanced dataset (50% faultymodules and 50% clean modules) does not result in the bestperformance for effort-aware models. Our results also indicatethat performances of effort-aware models are significantly dependenton the proportions of the two types of the classes in thetesting dataset. Models trained on moderately balanced datasetsare more likely to withstand fluctuations in performance as theclass distribution in the testing data varies.

元の言語英語
ホスト出版物のタイトルProceedings - 2016 IEEE 40th Annual Computer Software and Applications Conference, COMPSAC 2016
編集者William Claycomb, Dejan Milojicic, Ling Liu, Mihhail Matskin, Zhiyong Zhang, Sorel Reisman, Hiroyuki Sato, Zhiyong Zhang, Sheikh Iqbal Ahamed
出版者IEEE Computer Society
ページ154-163
ページ数10
ISBN(電子版)9781467388450
DOI
出版物ステータス出版済み - 8 24 2016
イベント2016 IEEE 40th Annual Computer Software and Applications Conference, COMPSAC 2016 - Atlanta, 米国
継続期間: 6 10 20166 14 2016

出版物シリーズ

名前Proceedings - International Computer Software and Applications Conference
1
ISSN(印刷物)0730-3157

その他

その他2016 IEEE 40th Annual Computer Software and Applications Conference, COMPSAC 2016
米国
Atlanta
期間6/10/166/14/16

    フィンガープリント

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications

これを引用

Bennin, K. E., Keung, J., Monden, A., Kamei, Y., & Ubayashi, N. (2016). Investigating the Effects of Balanced Training and Testing Datasets on Effort-Aware Fault Prediction Models. : W. Claycomb, D. Milojicic, L. Liu, M. Matskin, Z. Zhang, S. Reisman, H. Sato, Z. Zhang, ... S. I. Ahamed (版), Proceedings - 2016 IEEE 40th Annual Computer Software and Applications Conference, COMPSAC 2016 (pp. 154-163). [7552003] (Proceedings - International Computer Software and Applications Conference; 巻数 1). IEEE Computer Society. https://doi.org/10.1109/COMPSAC.2016.144