In this paper we present a framework for simultaneous image segmentation and region labeling leading to automatic image annotation. The proposed framework operates at semantic level using possible semantic labels to make decisions on handling image regions instead of visual features used tradi-tionally. In order to stress its independence of a specific image segmentation approach we applied our idea on two region growing algorithms, i.e. watershed and recursive shortest spanning tree. Additionally we exploit the notion of vis-ual context by employing fuzzy algebra and ontological taxonomic knowledge representation, incorporating in this way global information and improving re-gion interpretation. In this process, semantic region growing labeling results are being re-adjusted appropriately, utilizing contextual knowledge in the form of domain-specific semantic concepts and relations. The performance of the over-all methodology is demonstrated on a real-life still image dataset from the popular domains of beach holidays and motorsports.