Coverage-Directed differential testing of JVM implementations

Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, Jianjun Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

30 Citations (Scopus)

Abstract

Java virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (∗.class) along with their test oracles. An alternative is to employ binary fuzzing to differentially test JVMs by blindly mutating seeding classfiles and then executing the resulting mutants on different JVM binaries for revealing inconsistent behaviors. However, this blind approach is not cost effective in practice because most of the mutants are invalid and redundant. This paper tackles this challenge by introducing classfuzz, a coverage-directed fuzzing approach that focuses on representative classfiles for differential testing of JVMs' startup processes. Our core insight is to (1) mutate seeding classfiles using a set of predefined mutation operators (mutators) and employ Markov Chain Monte Carlo (MCMC) sampling to guide mutator selection, and (2) execute the mutants on a reference JVM implementation and use coverage uniqueness as a discipline for accepting representative ones. The accepted classfiles are used as inputs to differentially test different JVM implementations and find defects. We have implemented classfuzz and conducted an extensive evaluation of it against existing fuzz testing algorithms. Our evaluation results show that classfuzz can enhance the ratio of discrepancy-triggering classfiles from 1:7% to 11:9%. We have also reported 62 JVM discrepancies, along with the test classfiles, to JVM developers. Many of our reported issues have already been confirmed as JVM defects, and some even match recent clarifications and changes to the Java SE 8 edition of the JVM specification.

Original languageEnglish
Title of host publicationPLDI 2016 - Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation
EditorsChandra Krintz, Emery Berger
PublisherAssociation for Computing Machinery
Pages85-99
Number of pages15
ISBN (Electronic)9781450342612
DOIs
Publication statusPublished - Jun 2 2016
Event37th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2016 - Santa Barbara, United States
Duration: Jun 13 2016Jun 17 2016

Publication series

NameProceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)
Volume13-17-June-2016

Other

Other37th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2016
CountryUnited States
CitySanta Barbara
Period6/13/166/17/16

All Science Journal Classification (ASJC) codes

  • Software

Fingerprint Dive into the research topics of 'Coverage-Directed differential testing of JVM implementations'. Together they form a unique fingerprint.

  • Cite this

    Chen, Y., Su, T., Sun, C., Su, Z., & Zhao, J. (2016). Coverage-Directed differential testing of JVM implementations. In C. Krintz, & E. Berger (Eds.), PLDI 2016 - Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (pp. 85-99). (Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI); Vol. 13-17-June-2016). Association for Computing Machinery. https://doi.org/10.1145/2908080.2908095