TU-Net and TDeepLab: Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery

Yumi Iwashita, Kazuto Nakashima, Adrian Stoica, Ryo Kurazume

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper we propose two novel deep learning-based terrain classification methods robust to illumination changes. The use of cameras is challenged by a variety of factors, of most importance being the changes in illumination. On the other hand, since the temperature of various types of terrains depends on the thermal characteristics of the terrain, the terrain classification can be aided by utilizing the thermal information in addition to visible information. Thus we propose 'TU-Net (Two U-Net)' based on the U-Net and 'TDeepLab (Two DeepLab)' based on DeepLab, which combine visible and thermal images and train the network robust to illumination changes implicitly. To improve the network's learning capability, we expand the proposed methods to the Siamese-based method, which explicitly trains the network to be robust to illumination changes. We also investigate multiple options to fuse the visible and thermal images at at the bottom layer, middle layer, or the top layer of the network. We evaluate the proposed methods with a challenging new dataset consisting of visible and thermal images, which were collected from 10 am till 5 pm (after sunset), and we show the effectiveness of the proposed methods.

Original languageEnglish
Title of host publicationProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages280-285
Number of pages6
ISBN (Electronic)9781728111988
DOIs
Publication statusPublished - Apr 22 2019
Event2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 - San Jose, United States
Duration: Mar 28 2019Mar 30 2019

Publication series

NameProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

Conference

Conference2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
CountryUnited States
CitySan Jose
Period3/28/193/30/19

Fingerprint

Lighting
Electric fuses
Cameras
Hot Temperature
Deep learning
Temperature

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Signal Processing
  • Media Technology

Cite this

Iwashita, Y., Nakashima, K., Stoica, A., & Kurazume, R. (2019). TU-Net and TDeepLab: Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery. In Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 (pp. 280-285). [8695421] (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MIPR.2019.00057

TU-Net and TDeepLab : Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery. / Iwashita, Yumi; Nakashima, Kazuto; Stoica, Adrian; Kurazume, Ryo.

Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 280-285 8695421 (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Iwashita, Y, Nakashima, K, Stoica, A & Kurazume, R 2019, TU-Net and TDeepLab: Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery. in Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019., 8695421, Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019, Institute of Electrical and Electronics Engineers Inc., pp. 280-285, 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019, San Jose, United States, 3/28/19. https://doi.org/10.1109/MIPR.2019.00057
Iwashita Y, Nakashima K, Stoica A, Kurazume R. TU-Net and TDeepLab: Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery. In Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 280-285. 8695421. (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019). https://doi.org/10.1109/MIPR.2019.00057
Iwashita, Yumi ; Nakashima, Kazuto ; Stoica, Adrian ; Kurazume, Ryo. / TU-Net and TDeepLab : Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery. Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 280-285 (Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019).
@inproceedings{f380b71a55b746a39ee22caf9fd483e2,
title = "TU-Net and TDeepLab: Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery",
abstract = "In this paper we propose two novel deep learning-based terrain classification methods robust to illumination changes. The use of cameras is challenged by a variety of factors, of most importance being the changes in illumination. On the other hand, since the temperature of various types of terrains depends on the thermal characteristics of the terrain, the terrain classification can be aided by utilizing the thermal information in addition to visible information. Thus we propose 'TU-Net (Two U-Net)' based on the U-Net and 'TDeepLab (Two DeepLab)' based on DeepLab, which combine visible and thermal images and train the network robust to illumination changes implicitly. To improve the network's learning capability, we expand the proposed methods to the Siamese-based method, which explicitly trains the network to be robust to illumination changes. We also investigate multiple options to fuse the visible and thermal images at at the bottom layer, middle layer, or the top layer of the network. We evaluate the proposed methods with a challenging new dataset consisting of visible and thermal images, which were collected from 10 am till 5 pm (after sunset), and we show the effectiveness of the proposed methods.",
author = "Yumi Iwashita and Kazuto Nakashima and Adrian Stoica and Ryo Kurazume",
year = "2019",
month = "4",
day = "22",
doi = "10.1109/MIPR.2019.00057",
language = "English",
series = "Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "280--285",
booktitle = "Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019",
address = "United States",

}

TY - GEN

T1 - TU-Net and TDeepLab

T2 - Deep Learning-Based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery

AU - Iwashita, Yumi

AU - Nakashima, Kazuto

AU - Stoica, Adrian

AU - Kurazume, Ryo

PY - 2019/4/22

Y1 - 2019/4/22

N2 - In this paper we propose two novel deep learning-based terrain classification methods robust to illumination changes. The use of cameras is challenged by a variety of factors, of most importance being the changes in illumination. On the other hand, since the temperature of various types of terrains depends on the thermal characteristics of the terrain, the terrain classification can be aided by utilizing the thermal information in addition to visible information. Thus we propose 'TU-Net (Two U-Net)' based on the U-Net and 'TDeepLab (Two DeepLab)' based on DeepLab, which combine visible and thermal images and train the network robust to illumination changes implicitly. To improve the network's learning capability, we expand the proposed methods to the Siamese-based method, which explicitly trains the network to be robust to illumination changes. We also investigate multiple options to fuse the visible and thermal images at at the bottom layer, middle layer, or the top layer of the network. We evaluate the proposed methods with a challenging new dataset consisting of visible and thermal images, which were collected from 10 am till 5 pm (after sunset), and we show the effectiveness of the proposed methods.

AB - In this paper we propose two novel deep learning-based terrain classification methods robust to illumination changes. The use of cameras is challenged by a variety of factors, of most importance being the changes in illumination. On the other hand, since the temperature of various types of terrains depends on the thermal characteristics of the terrain, the terrain classification can be aided by utilizing the thermal information in addition to visible information. Thus we propose 'TU-Net (Two U-Net)' based on the U-Net and 'TDeepLab (Two DeepLab)' based on DeepLab, which combine visible and thermal images and train the network robust to illumination changes implicitly. To improve the network's learning capability, we expand the proposed methods to the Siamese-based method, which explicitly trains the network to be robust to illumination changes. We also investigate multiple options to fuse the visible and thermal images at at the bottom layer, middle layer, or the top layer of the network. We evaluate the proposed methods with a challenging new dataset consisting of visible and thermal images, which were collected from 10 am till 5 pm (after sunset), and we show the effectiveness of the proposed methods.

UR - http://www.scopus.com/inward/record.url?scp=85065603700&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065603700&partnerID=8YFLogxK

U2 - 10.1109/MIPR.2019.00057

DO - 10.1109/MIPR.2019.00057

M3 - Conference contribution

AN - SCOPUS:85065603700

T3 - Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

SP - 280

EP - 285

BT - Proceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -