Diffchaser: Detecting disagreements for deep neural networks

Xiaofei Xie, Lei Ma, Haijun Wang, Yuekang Li, Yang Liu, Xiaohong Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

The platform migration and customization have become an indispensable process of deep neural network (DNN) development lifecycle. A high-precision but complex DNN trained in the cloud on massive data and powerful GPUS often goes through an optimization phase (e.g., quantization, compression) before deployment to a target device (e.g., mobile device). A test set that effectively uncovers the disagreements of a DNN and its optimized variant provides certain feedback to debug and further enhance the optimization procedure. However, the minor inconsistency between a DNN and its optimized version is often hard to detect and easily bypasses the original test set. This paper proposes DiffChaser, an automated black-box testing framework to detect untargeted/targeted disagreements between version variants of a DNN. We demonstrate 1) its effectiveness by comparing with the state-of-the-art techniques, and 2) its usefulness in real-world DNN product deployment involved with quantization and optimization.

Original languageEnglish
Title of host publicationProceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019
EditorsSarit Kraus
PublisherInternational Joint Conferences on Artificial Intelligence
Pages5772-5778
Number of pages7
ISBN (Electronic)9780999241141
Publication statusPublished - Jan 1 2019
Event28th International Joint Conference on Artificial Intelligence, IJCAI 2019 - Macao, China
Duration: Aug 10 2019Aug 16 2019

Publication series

NameIJCAI International Joint Conference on Artificial Intelligence
Volume2019-August
ISSN (Print)1045-0823

Conference

Conference28th International Joint Conference on Artificial Intelligence, IJCAI 2019
CountryChina
CityMacao
Period8/10/198/16/19

Fingerprint

Black-box testing
Mobile devices
Deep neural networks
Feedback

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Cite this

Xie, X., Ma, L., Wang, H., Li, Y., Liu, Y., & Li, X. (2019). Diffchaser: Detecting disagreements for deep neural networks. In S. Kraus (Ed.), Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019 (pp. 5772-5778). (IJCAI International Joint Conference on Artificial Intelligence; Vol. 2019-August). International Joint Conferences on Artificial Intelligence.

Diffchaser : Detecting disagreements for deep neural networks. / Xie, Xiaofei; Ma, Lei; Wang, Haijun; Li, Yuekang; Liu, Yang; Li, Xiaohong.

Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019. ed. / Sarit Kraus. International Joint Conferences on Artificial Intelligence, 2019. p. 5772-5778 (IJCAI International Joint Conference on Artificial Intelligence; Vol. 2019-August).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Xie, X, Ma, L, Wang, H, Li, Y, Liu, Y & Li, X 2019, Diffchaser: Detecting disagreements for deep neural networks. in S Kraus (ed.), Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019. IJCAI International Joint Conference on Artificial Intelligence, vol. 2019-August, International Joint Conferences on Artificial Intelligence, pp. 5772-5778, 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, 8/10/19.
Xie X, Ma L, Wang H, Li Y, Liu Y, Li X. Diffchaser: Detecting disagreements for deep neural networks. In Kraus S, editor, Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019. International Joint Conferences on Artificial Intelligence. 2019. p. 5772-5778. (IJCAI International Joint Conference on Artificial Intelligence).
Xie, Xiaofei ; Ma, Lei ; Wang, Haijun ; Li, Yuekang ; Liu, Yang ; Li, Xiaohong. / Diffchaser : Detecting disagreements for deep neural networks. Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019. editor / Sarit Kraus. International Joint Conferences on Artificial Intelligence, 2019. pp. 5772-5778 (IJCAI International Joint Conference on Artificial Intelligence).
@inproceedings{dadc6f9e232f4d498cd0eb7184b489b1,
title = "Diffchaser: Detecting disagreements for deep neural networks",
abstract = "The platform migration and customization have become an indispensable process of deep neural network (DNN) development lifecycle. A high-precision but complex DNN trained in the cloud on massive data and powerful GPUS often goes through an optimization phase (e.g., quantization, compression) before deployment to a target device (e.g., mobile device). A test set that effectively uncovers the disagreements of a DNN and its optimized variant provides certain feedback to debug and further enhance the optimization procedure. However, the minor inconsistency between a DNN and its optimized version is often hard to detect and easily bypasses the original test set. This paper proposes DiffChaser, an automated black-box testing framework to detect untargeted/targeted disagreements between version variants of a DNN. We demonstrate 1) its effectiveness by comparing with the state-of-the-art techniques, and 2) its usefulness in real-world DNN product deployment involved with quantization and optimization.",
author = "Xiaofei Xie and Lei Ma and Haijun Wang and Yuekang Li and Yang Liu and Xiaohong Li",
year = "2019",
month = "1",
day = "1",
language = "English",
series = "IJCAI International Joint Conference on Artificial Intelligence",
publisher = "International Joint Conferences on Artificial Intelligence",
pages = "5772--5778",
editor = "Sarit Kraus",
booktitle = "Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019",

}

TY - GEN

T1 - Diffchaser

T2 - Detecting disagreements for deep neural networks

AU - Xie, Xiaofei

AU - Ma, Lei

AU - Wang, Haijun

AU - Li, Yuekang

AU - Liu, Yang

AU - Li, Xiaohong

PY - 2019/1/1

Y1 - 2019/1/1

N2 - The platform migration and customization have become an indispensable process of deep neural network (DNN) development lifecycle. A high-precision but complex DNN trained in the cloud on massive data and powerful GPUS often goes through an optimization phase (e.g., quantization, compression) before deployment to a target device (e.g., mobile device). A test set that effectively uncovers the disagreements of a DNN and its optimized variant provides certain feedback to debug and further enhance the optimization procedure. However, the minor inconsistency between a DNN and its optimized version is often hard to detect and easily bypasses the original test set. This paper proposes DiffChaser, an automated black-box testing framework to detect untargeted/targeted disagreements between version variants of a DNN. We demonstrate 1) its effectiveness by comparing with the state-of-the-art techniques, and 2) its usefulness in real-world DNN product deployment involved with quantization and optimization.

AB - The platform migration and customization have become an indispensable process of deep neural network (DNN) development lifecycle. A high-precision but complex DNN trained in the cloud on massive data and powerful GPUS often goes through an optimization phase (e.g., quantization, compression) before deployment to a target device (e.g., mobile device). A test set that effectively uncovers the disagreements of a DNN and its optimized variant provides certain feedback to debug and further enhance the optimization procedure. However, the minor inconsistency between a DNN and its optimized version is often hard to detect and easily bypasses the original test set. This paper proposes DiffChaser, an automated black-box testing framework to detect untargeted/targeted disagreements between version variants of a DNN. We demonstrate 1) its effectiveness by comparing with the state-of-the-art techniques, and 2) its usefulness in real-world DNN product deployment involved with quantization and optimization.

UR - http://www.scopus.com/inward/record.url?scp=85074665023&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074665023&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85074665023

T3 - IJCAI International Joint Conference on Artificial Intelligence

SP - 5772

EP - 5778

BT - Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019

A2 - Kraus, Sarit

PB - International Joint Conferences on Artificial Intelligence

ER -