Image annotation with incomplete labelling by modelling image specific structured loss

In this paper, we address the problem of image annotation with incomplete labeling, where multiple objects in each training image are not fully labeled. The conventional one-versus-all support vector machine (OVA-SVM), which performs fairly well on full labeling, decays drastically under the setting of incompleteness. Recently, a structured output learning method termed OVA-SSVM was proposed to boost the performance of OVA-SVM by modeling the structured associations of labels and show efficiency under the setting of incompleteness. OVA-SSVM assumes that each training sample includes a single label and adopts an loss measure of classification style where, as long as one of the predicted label is correct, the overall prediction should be considered correct. However, this may not be appropriate for the multilabel annotation task. Therefore, we extend the OVA-SSVM method to the multilabel situation and design a novel image-specific structured loss to account for the dependences between predicted labels relying on image label associations. The superiority of the proposed image-specific structured loss is that it can directly learn the semantic relationships of labels from training data without predefined semantic hierarchy. Extensive empirical results on a variety of benchmark datasets show that the proposed method performs significantly better than OVA-SSVM on image annotation tasks with incomplete labeling and achieves competitive performance compared to other state-of-the-art methods.