Audee: Automated Testing for Deep Learning Frameworks

Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, Chao Shen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Deep learning (DL) has been applied widely, and the quality of DL system becomes crucial, especially for safety-critical applications. Existing work mainly focuses on the quality analysis of DL models, but lacks attention to the underlying frameworks on which all DL models depend. In this work, we propose Audee, a novel approach for testing DL frameworks and localizing bugs. Audee adopts a search-based approach and implements three different mutation strategies to generate diverse test cases by exploring combinations of model structures, parameters, weights and inputs. Audee is able to detect three types of bugs: logical bugs, crashes and Not-a-Number (NaN) errors. In particular, for logical bugs, Audee adopts a cross-reference check to detect behavioural inconsistencies across multiple frameworks (e.g., TensorFlow and PyTorch), which may indicate potential bugs in their implementations. For NaN errors, Audee adopts a heuristic-based approach to generate DNNs that tend to output outliers (i.e., too large or small values), and these values are likely to produce NaN. Furthermore, Audee leverages a causal-testing based technique to localize layers as well as parameters that cause inconsistencies or bugs. To evaluate the effectiveness of our approach, we applied Audee on testing four DL frameworks, i.e., TensorFlow, PyTorch, CNTK, and Theano. We generate a large number of DNNs which cover 25 widely-used APIs in the four frameworks. The results demonstrate that Audee is effective in detecting inconsistencies, crashes and NaN errors. Intotal, 26 unique unknown bugs were discovered, and 7 of them have already been confirmed or fixed by the developers.

Original languageEnglish
Title of host publicationProceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages486-498
Number of pages13
ISBN (Electronic)9781450367684
DOIs
Publication statusPublished - Sep 2020
Externally publishedYes
Event35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020 - Virtual, Melbourne, Australia
Duration: Sep 22 2020Sep 25 2020

Publication series

NameProceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020

Conference

Conference35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020
Country/TerritoryAustralia
CityVirtual, Melbourne
Period9/22/209/25/20

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Audee: Automated Testing for Deep Learning Frameworks'. Together they form a unique fingerprint.

Cite this