Lyric video analysis using text detection and tracking

Shota Sakaguchi; Jun Kato; Masataka Goto; Seiichi Uchida

doi:10.1007/978-3-030-57058-3_30

Lyric video analysis using text detection and tracking

Shota Sakaguchi, Jun Kato, Masataka Goto, Seiichi Uchida

実世界ロボティクス

研究成果: Chapter in Book/Report/Conference proceeding › Conference contribution

抄録

We attempt to recognize and track lyric words in lyric videos. Lyric video is a music video showing the lyric words of a song. The main characteristic of lyric videos is that the lyric words are shown at frames synchronously with the music. The difficulty of recognizing and tracking the lyric words is that (1) the words are often decorated and geometrically distorted and (2) the words move arbitrarily and drastically in the video frame. The purpose of this paper is to analyze the motion of the lyric words in lyric videos, as the first step of automatic lyric video generation. In order to analyze the motion of lyric words, we first apply a state-of-the-art scene text detector and recognizer to each video frame. Then, lyric-frame matching is performed to establish the optimal correspondence between lyric words and the frames. After fixing the motion trajectories of individual lyric words from correspondence, we analyze the trajectories of the lyric words by k-medoids clustering and dynamic time warping (DTW).

本文言語	英語
ホスト出版物のタイトル	Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings
編集者	Xiang Bai, Dimosthenis Karatzas, Daniel Lopresti
出版社	Springer
ページ	426-440
ページ数	15
ISBN（印刷版）	9783030570576
DOI	https://doi.org/10.1007/978-3-030-57058-3_30
出版ステータス	出版済み - 2020
イベント	14th IAPR International Workshop on Document Analysis Systems, DAS 2020 - Wuhan, 中国継続期間: 7 26 2020 → 7 29 2020

出版物シリーズ

名前	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
巻	12116 LNCS
ISSN（印刷版）	0302-9743
ISSN（電子版）	1611-3349

会議

会議	14th IAPR International Workshop on Document Analysis Systems, DAS 2020
Country	中国
City	Wuhan
Period	7/26/20 → 7/29/20

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

文献へのアクセス

10.1007/978-3-030-57058-3_30

フィンガープリント「Lyric video analysis using text detection and tracking」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル

Sakaguchi, S., Kato, J., Goto, M., & Uchida, S. (2020). Lyric video analysis using text detection and tracking. In X. Bai, D. Karatzas, & D. Lopresti (Eds.), Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings (pp. 426-440). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12116 LNCS). Springer. https://doi.org/10.1007/978-3-030-57058-3_30

Lyric video analysis using text detection and tracking. / Sakaguchi, Shota; Kato, Jun; Goto, Masataka; Uchida, Seiichi.

Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings. ed. / Xiang Bai; Dimosthenis Karatzas; Daniel Lopresti. Springer, 2020. p. 426-440 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12116 LNCS).

研究成果: Chapter in Book/Report/Conference proceeding › Conference contribution

Sakaguchi, S, Kato, J, Goto, M & Uchida, S 2020, Lyric video analysis using text detection and tracking. in X Bai, D Karatzas & D Lopresti (eds), Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12116 LNCS, Springer, pp. 426-440, 14th IAPR International Workshop on Document Analysis Systems, DAS 2020, Wuhan, 中国, 7/26/20. https://doi.org/10.1007/978-3-030-57058-3_30

Sakaguchi S, Kato J, Goto M, Uchida S. Lyric video analysis using text detection and tracking. In Bai X, Karatzas D, Lopresti D, editors, Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings. Springer. 2020. p. 426-440. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-57058-3_30

Sakaguchi, Shota ; Kato, Jun ; Goto, Masataka ; Uchida, Seiichi. / Lyric video analysis using text detection and tracking. Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings. editor / Xiang Bai ; Dimosthenis Karatzas ; Daniel Lopresti. Springer, 2020. pp. 426-440 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{ec83433ba1624f42bd6edca915727141,

title = "Lyric video analysis using text detection and tracking",

abstract = "We attempt to recognize and track lyric words in lyric videos. Lyric video is a music video showing the lyric words of a song. The main characteristic of lyric videos is that the lyric words are shown at frames synchronously with the music. The difficulty of recognizing and tracking the lyric words is that (1) the words are often decorated and geometrically distorted and (2) the words move arbitrarily and drastically in the video frame. The purpose of this paper is to analyze the motion of the lyric words in lyric videos, as the first step of automatic lyric video generation. In order to analyze the motion of lyric words, we first apply a state-of-the-art scene text detector and recognizer to each video frame. Then, lyric-frame matching is performed to establish the optimal correspondence between lyric words and the frames. After fixing the motion trajectories of individual lyric words from correspondence, we analyze the trajectories of the lyric words by k-medoids clustering and dynamic time warping (DTW).",

author = "Shota Sakaguchi and Jun Kato and Masataka Goto and Seiichi Uchida",

note = "Funding Information: This work was supported by JSPS KAKENHI Grant Number JP17H06100.; 14th IAPR International Workshop on Document Analysis Systems, DAS 2020 ; Conference date: 26-07-2020 Through 29-07-2020",

year = "2020",

doi = "10.1007/978-3-030-57058-3_30",

language = "English",

isbn = "9783030570576",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "426--440",

editor = "Xiang Bai and Dimosthenis Karatzas and Daniel Lopresti",

booktitle = "Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings",

}

TY - GEN

T1 - Lyric video analysis using text detection and tracking

AU - Sakaguchi, Shota

AU - Kato, Jun

AU - Goto, Masataka

AU - Uchida, Seiichi

N1 - Funding Information: This work was supported by JSPS KAKENHI Grant Number JP17H06100.

PY - 2020

Y1 - 2020

N2 - We attempt to recognize and track lyric words in lyric videos. Lyric video is a music video showing the lyric words of a song. The main characteristic of lyric videos is that the lyric words are shown at frames synchronously with the music. The difficulty of recognizing and tracking the lyric words is that (1) the words are often decorated and geometrically distorted and (2) the words move arbitrarily and drastically in the video frame. The purpose of this paper is to analyze the motion of the lyric words in lyric videos, as the first step of automatic lyric video generation. In order to analyze the motion of lyric words, we first apply a state-of-the-art scene text detector and recognizer to each video frame. Then, lyric-frame matching is performed to establish the optimal correspondence between lyric words and the frames. After fixing the motion trajectories of individual lyric words from correspondence, we analyze the trajectories of the lyric words by k-medoids clustering and dynamic time warping (DTW).

AB - We attempt to recognize and track lyric words in lyric videos. Lyric video is a music video showing the lyric words of a song. The main characteristic of lyric videos is that the lyric words are shown at frames synchronously with the music. The difficulty of recognizing and tracking the lyric words is that (1) the words are often decorated and geometrically distorted and (2) the words move arbitrarily and drastically in the video frame. The purpose of this paper is to analyze the motion of the lyric words in lyric videos, as the first step of automatic lyric video generation. In order to analyze the motion of lyric words, we first apply a state-of-the-art scene text detector and recognizer to each video frame. Then, lyric-frame matching is performed to establish the optimal correspondence between lyric words and the frames. After fixing the motion trajectories of individual lyric words from correspondence, we analyze the trajectories of the lyric words by k-medoids clustering and dynamic time warping (DTW).

UR - http://www.scopus.com/inward/record.url?scp=85090098010&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85090098010&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-57058-3_30

DO - 10.1007/978-3-030-57058-3_30

M3 - Conference contribution

AN - SCOPUS:85090098010

SN - 9783030570576

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 426

EP - 440

BT - Document Analysis Systems - 14th IAPR International Workshop, DAS 2020, Proceedings

A2 - Bai, Xiang

A2 - Karatzas, Dimosthenis

A2 - Lopresti, Daniel

PB - Springer

T2 - 14th IAPR International Workshop on Document Analysis Systems, DAS 2020

Y2 - 26 July 2020 through 29 July 2020

ER -