In this chapter, we present our approach to semantic image analysis. Ontologies are used to capture a domain’s general, spatial and contextual knowledge and a genetic algorithm is applied to fulfil the final annotation. The employed domain knowledge considers high-level information in terms of the concepts of interest of the examined domain, contextual information in the form of fuzzy ontological relations, as well as low-level information in terms of prototypical low-level visual descriptors. To account for the inherent ambiguities in visual information, uncertainty has been introduced and utilized within the spatial relations definition. To illustrate the proposed process, a hypotheses set of graded annotations is produced initially for each image region, and then context is exploited to update appropriately the estimated degrees of confidence. A genetic algorithm is applied as the last step, in order to select the most plausible annotation by utilizing the visual and spatial concept definitions that are included in the domain ontology. Experiments with a collection of photographs derived from two distinct domains demonstrate the performance of the proposed approach.