TY - JOUR

T1 - A Robustness-Based Confidence Measure for Hybrid System Falsification

AU - Takisaka, Toru

AU - Zhang, Zhenya

AU - Arcaini, Paolo

AU - Hasuo, Ichiro

N1 - Publisher Copyright:
Author

PY - 2022

Y1 - 2022

N2 - Verification of hybrid systems is very challenging, if not impossible, due to their continuous dynamics that leads to infinite state space. As a countermeasure, falsification is usually applied to show that a specification does not hold, by searching for a falsifying input as a counterexample that refutes the specification. A falsification algorithm exploits the quantitative robust semantics of temporal specifications, which provides a numerical robustness that tells how robustly a specification holds or not, and uses it as a guide to explore the input space towards the direction of robustness descent—once negative robustness is observed, it indicates that a falsifying input is found. However, if a falsification algorithm does not return any falsifying input, a user is not sure whether the specification does indeed hold, or there exist counterexamples that the algorithm did not manage to reach. In this case, a measurement on how likely there indeed exists no counterexample in the input space is necessary for better understanding the safety of the system and deciding whether more budget should be allocated for the falsification. To this end, we propose a confidence measure that assesses the likelihood that the system is not falsifiable, i.e., how confident a user should be that a specification holds, given the fact that an algorithm has sampled a set of inputs but did not find any falsifying one. The confidence measure is defined in terms of a coverage criterion of the input space that assesses to which extent the whole input space is explored and a local area is exploited where low robustness is observed. Experiments on commonly-used falsification benchmarks show that our proposed confidence measure is reasonable and can distinguish different specifications.

AB - Verification of hybrid systems is very challenging, if not impossible, due to their continuous dynamics that leads to infinite state space. As a countermeasure, falsification is usually applied to show that a specification does not hold, by searching for a falsifying input as a counterexample that refutes the specification. A falsification algorithm exploits the quantitative robust semantics of temporal specifications, which provides a numerical robustness that tells how robustly a specification holds or not, and uses it as a guide to explore the input space towards the direction of robustness descent—once negative robustness is observed, it indicates that a falsifying input is found. However, if a falsification algorithm does not return any falsifying input, a user is not sure whether the specification does indeed hold, or there exist counterexamples that the algorithm did not manage to reach. In this case, a measurement on how likely there indeed exists no counterexample in the input space is necessary for better understanding the safety of the system and deciding whether more budget should be allocated for the falsification. To this end, we propose a confidence measure that assesses the likelihood that the system is not falsifiable, i.e., how confident a user should be that a specification holds, given the fact that an algorithm has sampled a set of inputs but did not find any falsifying one. The confidence measure is defined in terms of a coverage criterion of the input space that assesses to which extent the whole input space is explored and a local area is exploited where low robustness is observed. Experiments on commonly-used falsification benchmarks show that our proposed confidence measure is reasonable and can distinguish different specifications.

UR - http://www.scopus.com/inward/record.url?scp=85137574935&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85137574935&partnerID=8YFLogxK

U2 - 10.1109/TCAD.2022.3201157

DO - 10.1109/TCAD.2022.3201157

M3 - Article

AN - SCOPUS:85137574935

SP - 1

JO - IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

JF - IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

SN - 0278-0070

ER -