DeepMutation: Mutation Testing of Deep Learning Systems

Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, Yadong Wang

研究成果: 著書/レポートタイプへの貢献会議での発言

26 引用 (Scopus)

抜粋

Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.

元の言語英語
ホスト出版物のタイトルProceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018
編集者Sudipto Ghosh, Bojan Cukic, Robin Poston, Roberto Natella, Nuno Laranjeiro
出版者IEEE Computer Society
ページ100-111
ページ数12
ISBN(電子版)9781538683217
DOI
出版物ステータス出版済み - 11 16 2018
イベント29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018 - Memphis, 米国
継続期間: 10 15 201810 18 2018

出版物シリーズ

名前Proceedings - International Symposium on Software Reliability Engineering, ISSRE
2018-October
ISSN(印刷物)1071-9458

その他

その他29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018
米国
Memphis
期間10/15/1810/18/18

    フィンガープリント

All Science Journal Classification (ASJC) codes

  • Software
  • Safety, Risk, Reliability and Quality

これを引用

Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., ... Wang, Y. (2018). DeepMutation: Mutation Testing of Deep Learning Systems. : S. Ghosh, B. Cukic, R. Poston, R. Natella, & N. Laranjeiro (版), Proceedings - 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018 (pp. 100-111). [8539073] (Proceedings - International Symposium on Software Reliability Engineering, ISSRE; 巻数 2018-October). IEEE Computer Society. https://doi.org/10.1109/ISSRE.2018.00021