DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems

Lei Ma, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao

研究成果: 著書/レポートタイプへの貢献会議での発言

7 引用 (Scopus)

抄録

Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

元の言語英語
ホスト出版物のタイトルSANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering
編集者Emad Shihab, David Lo, Xinyu Wang
出版者Institute of Electrical and Electronics Engineers Inc.
ページ614-618
ページ数5
ISBN(電子版)9781728105918
DOI
出版物ステータス出版済み - 3 15 2019
イベント26th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2019 - Hangzhou, 中国
継続期間: 2 24 20192 27 2019

出版物シリーズ

名前SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering

会議

会議26th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2019
中国
Hangzhou
期間2/24/192/27/19

Fingerprint

Learning systems
Testing
Deep learning

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Software
  • Computational Theory and Mathematics

これを引用

Ma, L., Juefei-Xu, F., Xue, M., Li, B., Li, L., Liu, Y., & Zhao, J. (2019). DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. : E. Shihab, D. Lo, & X. Wang (版), SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering (pp. 614-618). [8668044] (SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SANER.2019.8668044

DeepCT : Tomographic Combinatorial Testing for Deep Learning Systems. / Ma, Lei; Juefei-Xu, Felix; Xue, Minhui; Li, Bo; Li, Li; Liu, Yang; Zhao, Jianjun.

SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering. 版 / Emad Shihab; David Lo; Xinyu Wang. Institute of Electrical and Electronics Engineers Inc., 2019. p. 614-618 8668044 (SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering).

研究成果: 著書/レポートタイプへの貢献会議での発言

Ma, L, Juefei-Xu, F, Xue, M, Li, B, Li, L, Liu, Y & Zhao, J 2019, DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. : E Shihab, D Lo & X Wang (版), SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering., 8668044, SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering, Institute of Electrical and Electronics Engineers Inc., pp. 614-618, 26th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2019, Hangzhou, 中国, 2/24/19. https://doi.org/10.1109/SANER.2019.8668044
Ma L, Juefei-Xu F, Xue M, Li B, Li L, Liu Y その他. DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. : Shihab E, Lo D, Wang X, 編集者, SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering. Institute of Electrical and Electronics Engineers Inc. 2019. p. 614-618. 8668044. (SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering). https://doi.org/10.1109/SANER.2019.8668044
Ma, Lei ; Juefei-Xu, Felix ; Xue, Minhui ; Li, Bo ; Li, Li ; Liu, Yang ; Zhao, Jianjun. / DeepCT : Tomographic Combinatorial Testing for Deep Learning Systems. SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering. 編集者 / Emad Shihab ; David Lo ; Xinyu Wang. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 614-618 (SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering).
@inproceedings{3a091f4dfcbb490c95fecc30ee4cce99,
title = "DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems",
abstract = "Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.",
author = "Lei Ma and Felix Juefei-Xu and Minhui Xue and Bo Li and Li Li and Yang Liu and Jianjun Zhao",
year = "2019",
month = "3",
day = "15",
doi = "10.1109/SANER.2019.8668044",
language = "English",
series = "SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "614--618",
editor = "Emad Shihab and David Lo and Xinyu Wang",
booktitle = "SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering",
address = "United States",

}

TY - GEN

T1 - DeepCT

T2 - Tomographic Combinatorial Testing for Deep Learning Systems

AU - Ma, Lei

AU - Juefei-Xu, Felix

AU - Xue, Minhui

AU - Li, Bo

AU - Li, Li

AU - Liu, Yang

AU - Zhao, Jianjun

PY - 2019/3/15

Y1 - 2019/3/15

N2 - Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

AB - Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

UR - http://www.scopus.com/inward/record.url?scp=85064176909&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064176909&partnerID=8YFLogxK

U2 - 10.1109/SANER.2019.8668044

DO - 10.1109/SANER.2019.8668044

M3 - Conference contribution

AN - SCOPUS:85064176909

T3 - SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering

SP - 614

EP - 618

BT - SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering

A2 - Shihab, Emad

A2 - Lo, David

A2 - Wang, Xinyu

PB - Institute of Electrical and Electronics Engineers Inc.

ER -