- & , University of Brighton, UK
- Paul Lewis & Jonathon Hare, University of Southampton, UK
The Reality of the Semantic Gap in Image Retrieval
The semantic gap is referred to frequently in papers on image retrieval or multimedia information handling. However, whilst many authors have been happy to make reference to it, few have attempted to characterize the gap in any detail. This tutorial attempts to rectify this situation by characterizing the semantic gap in image retrieval rather more specifically than hitherto. It summarises current attempts to begin to bridge the gap both through developments in content-based techniques, the application of semantic web and knowledge technologies and recent progress in auto image annotation. The tutorial consists of presentations/demonstrations partly based on research in recent European and UK projects, and particularly on a project to investigate the semantic gap funded by the Arts and Humanities Research Council in the UK involving the four presenters.
The tutorial aims to provide valuable insights for those involved in research and development on image or multimedia retrieval and who wish to understand and address the concerns of real end-users and exploit recent research results in the field. In particular, the tutorial provides practical insights in the problems associated with bridging the communication gap between the computer science/vision research community and the image management/practitioner community.
Tutorial PresentationProgram
| Room: Themistocles A | |
| 09:30 - 10:15 |
Image Retrieval from a Practitioner's Perspective
|
|---|---|
| 10:15 - 11:00 |
Characterising the Gap
|
| 11:00 - 11:30 | Coffee Break |
| 11:30 - 12:15 |
Ontologies, Thesauri, Taxonomies
|
| 12:15 - 13:00 |
Semantic spaces and multimodal searching
|
Summary
Image retrieval from a practitioner's perspective
The first part of the tutorial summarises research into the way picture searchers articulate real queries, how they are typically resolved through a combination of traditional metadata and the knowledge of the searcher. This section includes our own investigations into query categorization and image categorization and the identification of recurring semantic issues in image search such as significance of events, abstract and emotive concepts and unwanted features.
Characterising the gap
The second part reviews the progress in content-based image retrieval, automatic annotation and extraction of semantics in recent years and explore the types of query that they are able to address. The semantic gap between features that can be extracted directly from images and the semantics that the human searcher attaches to the visual information is revisited and various staging posts across the gap are identified such as raw data to features, features to objects, objects to labels and labels to semantics. The ways in which these sub gaps can be bridged in some instances are discussed together with the substantial current interest in machine learning and auto annotation.
Ontologies, thesauri, taxonomies
In the third part, recent interest in and applications of semantic web technologies for image retrieval support are discussed. The presentation will explore both the growth in interest in formal knowledge representation schemes, best exemplified by ontologies, and the informal approaches, exemplified by folksonomies and community annotation schemes. These approaches can be regarded as addressing issues at the semantic end of the semantic gap. The growing activity in multimedia ontology development and attempts to integrate ontological representation schemes such as OWL and multimedia representation schemes like MPEG7 is included.
Semantic spaces and multimodal searching
The final part shows how the application of ideas from cross language latent semantic indexing can be extended to build multimedia semantic spaces in which visual and conceptual "terms" are mapped to similar locations in the space. This means that images can be retrieved using either textual or visual descriptors whether or not they have collateral textual annotations.