Person re-identification in TV series using robust face recognition and user feedback

Mika Fischer*, Hazim Kemal Ekenel, Rainer Stiefelhagen

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

13 Citations (Scopus)

Abstract

In this paper, we present a system for person re-identification in TV series. In the context of video retrieval, person re-identification refers to the task where a user clicks on a person in a video frame and the system then finds other occurrences of the same person in the same or different videos. The main characteristic of this scenario is that no previously collected training data is available, so no person-specific models can be trained in advance. Additionally, the query data is limited to the image that the user clicks on. These conditions pose a great challenge to the re-identification system, which has to find the same person in other shots despite large variations in the person's appearance. In the study, facial appearance is used as the re-identification cue, since, in contrast to surveillance-oriented re-identification studies, the person can have different clothing in different shots. In order to increase the amount of available face data, the proposed system employs a face tracker that can track faces up to full profile views. This makes it possible to use a profile face image as query image and also to retrieve images with non-frontal poses. It also provides temporal association of the face images in the video, so that instead of using single images for query or target, whole tracks can be used. A fast and robust face recognition algorithm is used to find matching faces. If the match result is highly confident, our system adds the matching face track to the query set. Finally, if the user is not satisfied with the number of returned results, the system can present a small number of candidate face images and lets the user confirm the ones that belong to the queried person. These features help to increase the variation in the query set, making it possible to retrieve results with different poses, illumination conditions, etc. The system is extensively evaluated on two episodes of the TV series Coupling, showing very promising results.

Original languageEnglish
Pages (from-to)83-104
Number of pages22
JournalMultimedia Tools and Applications
Volume55
Issue number1
DOIs
Publication statusPublished - Oct 2011
Externally publishedYes

Funding

Acknowledgements This study is partially funded by OSEO, French State agency for innovation, as part of the Quaero Programme, and by the “Concept for the Future” of the Karlsruhe Institute of Technology within the framework of the German Excellence Initiative.

FundersFunder number
French State agency for innovation
OSEO
Karlsruhe Institute of Technology

    Keywords

    • Face recognition
    • Face tracking
    • Person re-identification
    • Person retrieval
    • Video retrieval

    Fingerprint

    Dive into the research topics of 'Person re-identification in TV series using robust face recognition and user feedback'. Together they form a unique fingerprint.

    Cite this