The motivation of this work is to tackle the problem of high-level concept detection within image and video documents using a globally annotated training set. The goal is to determine whether a concept exists within an image along with a degree of confidence and not its actual position. Since this approach begins with a coarse image segmentation, the high-level concepts that is able to tackle can be described as "materials" or "scenes". MPEG-7 color and texture features are locally extracted from coarsely segmented regions using an RSST variation. Using a significantly large set of images and after the application of a hierarchical clustering algorithm on all regions, a relatively small number of them, is selected. These regions are called "region types". This set of region types composes a visual dictionary which facilitates the mapping of low- to high-level features.

The process of calculating a model vector using a visual dictionary and some examples of model vectors.

A flowchart depicting the overall classification process. By red, the offline parts are depicted, while the dotted box depicts the LSA which is an optional part of the process.

A "model vector" is finally extracted to represent the visual properties in terms of the region types it contains. This model vector representation is then used to train SVM-based concept detectors. To further improve the performance of this algorithm, the Latent Semantic Analysis technique is applied to exploit the hidden relations among region types.

Average Precision of the proposed approach vs the ratio λ of negative to positive examples of the test set, with and without LSA.

For certain concepts such as Outdoor, Office and Road, LSA improves the results, while λ increases. This means that positive examples are detected in a lower and more correct rank. The common thing among these concepts is that they cannot be described in a straightforward way, such as Vegetation and Sky.

Extensive results for the cases of λ=4 and λ=max.

This approach has also been applied as part of the joint COST292 and K-Space submissions to the high-level feature extraction task of TRECVID 2006 and 2007.