Automatic solar panel inspection systems are essential to maintain power generation efficiency and reduce the cost. Thermal images generated by thermographic cameras can be used for solar panel fault diagnosis because defective panels show abnormal temperature. However, it is difficult to identify an anomaly from a single panel image when similar temperature features appear in normal panels and abnormal panels. In this paper, we propose a different feature based method to identify defective solar panels in thermal images. To determine abnormal panel from input panel images, we apply a voting strategy by using the prediction results of subtraction network. In our experiments, we construct two datasets to evaluate our method: the clean panels dataset which is constructed by manually extracted panel images and the noise containing dataset which is consisting of panel images extracted by the automatic panel extraction method. Our method achieves more than 90% classification accuracy on both clean panels dataset and noise containing dataset.