Deephunter: A coverage-guided fuzz testing framework for deep neural networks

Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, Simon See

研究成果: 著書/レポートタイプへの貢献会議での発言

7 引用 (Scopus)

抄録

The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.

元の言語英語
ホスト出版物のタイトルISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis
編集者Dongmei Zhang, Anders Moller
出版者Association for Computing Machinery, Inc
ページ158-168
ページ数11
ISBN(電子版)9781450362245
DOI
出版物ステータス出版済み - 7 10 2019
イベント28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019 - Beijing, 中国
継続期間: 7 15 20197 19 2019

出版物シリーズ

名前ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

会議

会議28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019
中国
Beijing
期間7/15/197/19/19

Fingerprint

Seed
Testing
Defects
Deep neural networks
Accidents
Semantics
Feedback
Experiments

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

これを引用

Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., ... See, S. (2019). Deephunter: A coverage-guided fuzz testing framework for deep neural networks. : D. Zhang, & A. Moller (版), ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 158-168). (ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/3293882.3330579

Deephunter : A coverage-guided fuzz testing framework for deep neural networks. / Xie, Xiaofei; Ma, Lei; Juefei-Xu, Felix; Xue, Minhui; Chen, Hongxu; Liu, Yang; Zhao, Jianjun; Li, Bo; Yin, Jianxiong; See, Simon.

ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 版 / Dongmei Zhang; Anders Moller. Association for Computing Machinery, Inc, 2019. p. 158-168 (ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis).

研究成果: 著書/レポートタイプへの貢献会議での発言

Xie, X, Ma, L, Juefei-Xu, F, Xue, M, Chen, H, Liu, Y, Zhao, J, Li, B, Yin, J & See, S 2019, Deephunter: A coverage-guided fuzz testing framework for deep neural networks. : D Zhang & A Moller (版), ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, Association for Computing Machinery, Inc, pp. 158-168, 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, 中国, 7/15/19. https://doi.org/10.1145/3293882.3330579
Xie X, Ma L, Juefei-Xu F, Xue M, Chen H, Liu Y その他. Deephunter: A coverage-guided fuzz testing framework for deep neural networks. : Zhang D, Moller A, 編集者, ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. Association for Computing Machinery, Inc. 2019. p. 158-168. (ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis). https://doi.org/10.1145/3293882.3330579
Xie, Xiaofei ; Ma, Lei ; Juefei-Xu, Felix ; Xue, Minhui ; Chen, Hongxu ; Liu, Yang ; Zhao, Jianjun ; Li, Bo ; Yin, Jianxiong ; See, Simon. / Deephunter : A coverage-guided fuzz testing framework for deep neural networks. ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 編集者 / Dongmei Zhang ; Anders Moller. Association for Computing Machinery, Inc, 2019. pp. 158-168 (ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis).
@inproceedings{0923dd9d4e504073ac168affe18c4d17,
title = "Deephunter: A coverage-guided fuzz testing framework for deep neural networks",
abstract = "The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98{\%} validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.",
author = "Xiaofei Xie and Lei Ma and Felix Juefei-Xu and Minhui Xue and Hongxu Chen and Yang Liu and Jianjun Zhao and Bo Li and Jianxiong Yin and Simon See",
year = "2019",
month = "7",
day = "10",
doi = "10.1145/3293882.3330579",
language = "English",
series = "ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis",
publisher = "Association for Computing Machinery, Inc",
pages = "158--168",
editor = "Dongmei Zhang and Anders Moller",
booktitle = "ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis",

}

TY - GEN

T1 - Deephunter

T2 - A coverage-guided fuzz testing framework for deep neural networks

AU - Xie, Xiaofei

AU - Ma, Lei

AU - Juefei-Xu, Felix

AU - Xue, Minhui

AU - Chen, Hongxu

AU - Liu, Yang

AU - Zhao, Jianjun

AU - Li, Bo

AU - Yin, Jianxiong

AU - See, Simon

PY - 2019/7/10

Y1 - 2019/7/10

N2 - The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.

AB - The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.

UR - http://www.scopus.com/inward/record.url?scp=85070586358&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070586358&partnerID=8YFLogxK

U2 - 10.1145/3293882.3330579

DO - 10.1145/3293882.3330579

M3 - Conference contribution

AN - SCOPUS:85070586358

T3 - ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

SP - 158

EP - 168

BT - ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

A2 - Zhang, Dongmei

A2 - Moller, Anders

PB - Association for Computing Machinery, Inc

ER -