This paper proposes a hybrid query refinement model for distance-based index structures supporting content-based image retrievals. The framework refines a query by considering both the low-level feature space as well as the high-level semantic interpretations separately. Thus, it successfully handles queries where the gap between the feature components and the semantics is large. It refines the low-level feature space, indexed by the distance based index structure, in multiple iterations by introducing the concept of multipoint query in a metric space. It refines the high-level semantic space by dynamically adjusting the constructs of a framework, called the Markov Model Mediator (MMM), utilized to introduce the semantic relationships in the index structure. A k-nearest neighbor (k-NN) algorithm is designed to handle similarity searches that refine a query in multiple iterations utilizing the proposed hybrid query refinement model. Extensive experiments are performed demonstrating an increased relevance of query results in subsequent iterations while incurring a low computational overhead. Further, an evaluation metric, called the Model_Score, is proposed to compare the performance of different retrieval frameworks in terms of both computation overhead and query result relevance. This metric enables the users to choose the retrieval framework appropriate for their requirements.