An Improved Content Based Image Retrieval Technique by Exploiting Bi-layer Concept

1. INTRODUCTION

Digital image processing plays a significant role in various areas such as medical image processing [1], image inpainting [2], pattern recognition, biometrics, content-based image retrieval (CBIR), image compression, information hiding [3], and multimedia security [4]. The retrieval of similar images from a large range of images is becoming a serious challenge with the advent of digital communication technology and the growing use of the Internet. Several penetrating and retrieval utilities are essential for end users to retrieve the images efficiently from different domains of the image databases such as medical, education, weather forecasting, criminal investigation, advertising, social media, web, art design, and entertainment. The query information is either text format or image format.

Different techniques for image retrieval have been developed and they are classified into two approaches: Text-based image retrieval (TBIR) and CBIR [5]. TBIR was first introduced in 1970 for searching and retrieving images from image databases [6]. In TBIR, the images are denoted by text, and then the text is used to retrieve or search the images. Such a system is text-based search and is generally referred to as TBIR. The TBIR method relies on the manual text search or keyword matching of the existing image keywords and the result has been dependent on the human labeling of the images. TBIR approach requires information such as image keyword, image location, image tags, image name, and other information related to the image. It needs human intervention to enter the data of images in the database and that is the difficulty of the process. TBIR has the following limitations: (1) It leads to inaccurate results when human has been doing datasets annotation process wrongly, (2) single keyword of image information is not efficient to transfer the overall image description, (3) it is based on manual annotation of the images, which is time consuming [5], [7].

To overcome those mentioned limitations of TBIR, a new approach for image retrieval has been invented by researcher which is known as CBIR. CBIR can be considered as a common tool for retrieving, searching, and browsing images of a query information from a large database of digital images. In CBIR, the image information, visual features such as low level features (color, texture, and/or shape), or bag of features (BoF) have been extracted from the images to find similar images in the database [8]. Fig. 1 shows the general block diagram of CBIR approach [7].

Fig. 1. General block diagram of content-based image retrieval approach.

In general, CBIR entails two main steps: The feature extraction and feature matching. In the first step, features are extracted from a dataset of images and stored in a feature vector. In the second step, the extracted features from the query image are compared with the extracted features of images in the dataset using certain distance measurement. If the distance between feature vector of the query image and the image in the database is small enough, the corresponding image in the database is considered as a match/similar image to the query image. Consequently, the matched images are then ranked accordingly to a similarity index from the smallest distance value to the largest one. Finally, the retrieved images are specified according to the highest similarity, that is, lowest distance value [9].

The main objective of CBIR techniques is to improve the efficiency of the system by increasing the performance using the combination of features [6]. Image features can be classified into two types: Local features and global features. Local features work locally which are focused on the key point in images whereas global features extract information from the entire image [10]. When the image dataset is quite large, image relevant to the query image are very few. Therefore, it is important to eliminate those irrelevant images. The main contribution of our proposed approach is filtering the images in the dataset to eliminate/minimize the most irrelevant images, then from the remaining images find the most similar/match images. In this paper, a new CBIR approach based on two layers is developed. The first layer aims in filtering the images using (BoF) strategy on the basis of extracting local features, while the second layer aims to retrieve similar images, from the remaining images, to the query image based on extracting global features such as color, shape, and texture.

The rest of the paper is organized as follows: Section 2 presents the literature review, Section 3 gives the background, Section 4 addresses the proposed approach in detail, Section 5 illustrates the experimental results, and finally, Section 6 presents the conclusion.

2. LITERATURE REVIEW

There are several CBIR techniques proposed for image retrieval applications using various feature extraction methods. Each of these techniques competes to improve the precision rate of finding the best similar images to the query image. In general, all the CBIR techniques have two main steps; the first step is feature extraction and the second step is feature matching. This section concerns the review of the most related and important existing CBIR techniques.

The concept of CBIR was first introduced by Kato in 1992 by developing a technique for sketch retrieval, similarity retrieval, and sense retrieval to support visual interaction [11]. Sketch retrieval accepts the image data of sketches, similarity retrieval evaluates the similarity based on the personal view of each user, and sense retrieval evaluates based on the text data and the image data at content level based on the personal view. In 2009, Lin et al. proposed a CBIR technique depending on extracting three types of image features [12]. The first feature, color co-occurrence matrix was extracted as a color feature, while for the second feature, difference between pixels of scan pattern was used for extracting texture feature, and the third feature, color histogram for K-mean was extracted which is based on color distribution. Consequently, feature selection techniques were implemented to select the optimal features not only to maximize the detection rate but also to simplify the computation of image retrieval. In addition, this proposed technique further uses sequential for-ward selection to select features with better discriminability for image retrieval and to overcome the problem of excessive features. Finally, Euclidean distance was used to find the similarity in the feature matching step. The results reported in this work claimed that the proposed technique reached a precision rate of 72.70% for the top-20.

Huang et al. proposed a new CBIR technique, in 2010, in which combined/fused the Gabor texture feature and Hue Saturation Value (HSV) color moment feature [13]. Furthermore, the normalized Euclidean distance was used to calculate the similarity between the feature vector of the query image and the feature vector of the images in the dataset. This proposed technique achieved the precision rate of 63.6% for the top-15. In 2012, Singha et al. proposed an algorithm for CBIR by extracting features called wavelet based color histogram image retrieval as a color and texture features [14]. The color and texture features are extracted through color histogram as well as wavelet transformation, for the combination of these features is robust to object translation and scaling in an image. This technique was used the histogram intersection distance for feature matching purposes. The results reported in this work claimed that this technique achieved a precision rate of 76.2% for the top-10. Another CBIR technique was proposed by Yu et al., in 2013, that aims to investigate various combinations of mid-level features to build an effective image retrieval system based on the BoF model [15]. Specifically, this work studies two ways of integrating: 1- scale-invariant feature transform (SIFT) with local binary pattern (LBP) descriptors and, 2- histogram of oriented gradients with LBP descriptors. Based on the qualitative and quantitative evaluations on two benchmark datasets, the integrations of these features yield complementary and substantial improvement on image retrieval even with noisy background and ambiguous objects. Consequently, two integration models are proposed, the patch-based integration and the image-based integration. Using a weighted K-means clustering algorithm, the image-based SIFT-LBP integration achieved a precision rate of 65% for the top-20. A new CBIR technique was proposed by Somnugpong et al., in 2016, by combining color correlograms and edge direction histogram (EDH) features to give precedence for spatial information in an image [16]. Color correlogram treats information about spatial color correlation, while EDH provides the geometry information in the case of the same image but different color. Evaluation is performed by simple calculation like Euclidean distance between the query image and the images in the database. Researchers claimed that their proposed technique achieved 65% of precision rate for the top-15. In 2018, Al-Jubouri et al. proposed a new CBIR technique that addresses the semantic gap issue by exploiting cluster shapes [17]. The technique first extracts local color using YCbCr color space and texture feature using Discrete Cosine Transform coefficients. The Expectation-Maximization Gaussian Mixture Model clustering algorithm is then applied to the local feature vectors to obtain clusters of different shapes. To compare dissimilarity between two images, the technique uses a dissimilarity measure based on the principle of Kullback-Leibler divergence to compare pair-wise dissimilarity of cluster shapes. This work further investigates two respective scenarios when the number of clusters is fixed and adaptively determined according to cluster quality. The results reported in this work illustrate that the proposed retrieval mechanism based on cluster shapes increases the image discrimination, and when the number of cluster is fixed to a large number, the precision of image retrieval is better than that when the relatively small number of clusters is adaptively determined. Authors claimed that their technique achieved a precision rate of 75% for the top-10.

In 2018, Nazir et al. proposed a new CBIR technique in which used color and texture features [18]. The edge histogram descriptor is extracted as a local feature and discrete wavelet transform as well as color histogram features are extracted as global features. Consequently, Manhattan distance measurement was used to measure the similarity between the feature vector of the query image and the feature vector of the images in the dataset. The reported results of the work revealed that this proposed technique achieved a precision rate of 73.5% for the top-20. Pradhan et al., in 2019, developed a new CBIR scheme based on multi-level colored directional motif histogram [7]. The proposed scheme extracts local structural features at three different levels. The performance of this proposed scheme has been evaluated using different Corel/natural, object, texture, and heterogeneous image datasets. Regarding to the Corel-1k, the precision rate of 64% and 59.6% was obtained for top-10 and top-20, respectively. Qazanfari et al., in 2019, proposed a CBIR technique based on HSV color space [19]. The human visual system is very sensitive to the color and edge orientation, also color histogram and color difference histogram are two kinds of low-level feature extraction which are meaningful representatives of the image color and edge orientation information. This proposed technique was used Canberra distance measurement and this work has been evaluated using three standard databases Corel 5k, Corel 10 and UKBench and achieved 61.82%, 50,67%, and 74.77% of precision rate for the top-12, respectively. In 2019, Rashno et al., proposed a new technique for CBIR in which color and texture features were used. HSV, Red, green, and blue (RGB) and norm of low frequency components were used as color features, while wavelet transformation was used to extract texture features [20]. Consequently, ant colony optimization-based feature selection was used to select the most relevant features, to minimize the number of features, and to maximize F-measure in the proposed CBIR system. Furthermore, Euclidean distance measurement was used to find the similarity between query and database images. The results reported in this work demonstrate that this approach reached the precision rate of 60.79% for the top-20. In 2019, Rana et al. proposed a CBIR technique by fusing parametric color and shape features with nonparametric texture feature [21]. The color moments and moment invariants which are parametric feature are extracted to describe color distribution and shapes of an image. The non-parametric ranklet transformation is performed to narrate the texture features. These parametric and non-parametric features were integrated to propose a robust and effective CBIR algorithm. In this proposed work, four similarity measurements are investigated during the experiment, namely, Chi-squared, Manhattan distance or City block distance, Euclidean distance, and Canberra distance. The experimental results demonstrate that Euclidean distance metric yields better precision and recall than other distance measuring criteria. Authors claimed that their technique achieved a precision rate of 67.6% for the top-15 using Euclidean distance. Finally, Sadique et al., in 2019, proposed a CBIR technique in which investigates various global and local feature extraction methods for image retrieval [22]. The proposed work uses a combination of speeded up robust features (SURF) detector and descriptor with color moments as local features, and modified grey level cooccurrence matrices as global features. Both global and local features are used as the only local features are not suitable when the variety of images is large. Finally, fast approximate nearest neighbor search was used for matching the extracted features. Authors claimed that their proposed technique achieved a precision rate of 70.48% for the top-20.

3. BACKGROUND

This section aims to present a reasonable amount of background information about useful techniques such as (SURF) feature descriptor, color-based features, texture-based features, shape-based features, and feature matching techniques.

3.1. SURF Feature Descriptor

The most popular feature descriptor is SURF, which is also the most important one. However, there are other available feature descriptors. SURF can be considered as a local feature. Local features can provide more detailed characters in an image in comparison with global features such as color, texture, and shape. It is a rotation and scale invariant descriptor that performs better with respect to distinctiveness, repeatability, and robustness. It is also photometric deformations, detection errors, geometric, and robust to noise [23]. SURF is used in many applications such as BoF which is used and successful in image analysis and classification [24]. In BoF technique, SURF descriptor is often used to extract local feature first. In the next stage, a quantization algorithm such as K-means is separately applied to the extracted SURF features to reduce high dimensional feature vectors to clusters, which are also known as visual words. Then, K-means clustering is used to initialize the M center point to build M visual words. The K-means clustering algorithm takes feature space as input and reduces it into M cluster as output. Then, the image is represented as a histogram of code word occurrences by mapping the local features to a vocabulary [24]. The methodology of the image representation based on the BoF model is illustrated in Fig. 2.

Fig. 2. Methodology of the bag of features based image representation for content-based image retrieval.

3.2. Color-based Features Extraction

The color-based features have commonly been used in CBIR systems because of its easy and fast computation [14]. Color-based features can be extracted using a histogram of quantized values of color in Hue (H), Saturation (S), and Value (V) of the HSV color space. HSV color space is more robust to human perception as compared to the RGB color space. Due to the robustness of the HSV color space, first RGB images are converted to HSV color space and then uniform quantization is applied. Feature vectors are generated by considering the values of H=9, S=3, and V=3 to form the feature vector of size 81 (9 × 3 ×3) bins. Representation of HSV color feature vector of an image is presented in Fig. 3.

Fig. 3. Hue saturation value feature vector.

3.5. Feature Matching

There are certain similarity measurements that used to compute the similarity between query image and images in the database, in our proposed approach, Manhattan distance is used for the BoF, see equation (1) [27], and Euclidean distance is used for color, texture and, shape features, equation (2) [13].

Where Qf = (Qf₁ ,Qf₂,..., Qf_L-1) is the feature vector of query image, Df = (Df₁ ,Df₂,..., Df_L-1) is the feature vector of the database of images, and L is the dimension of image feature.

Next section will present the proposed approach in detail.

4. PROPOSED APPROACH

This section presents the detailed steps of the proposed bi-layer approach as follows:

The block diagram of the proposed bi-layer approach is illustrated in Fig. 4.

Fig. 4. Block diagram of the proposed bi-layer content-based image retrieval system for top-5.

5. EXPERIMENTAL RESULTS

In this section, experiments are performed comprehensively to assess the performance of the proposed approach in terms of precision rate, the most common confusion matrix measurement used in the CBIR research area. Furthermore, the proposed approach is compared to the most recent existing works.

5.2. Evaluation Measurements

Precision confusion matrix measurement is used to assess the performance of the proposed approach. The precision determines the number of correctly retrieved images over the total number of the retrieved images from the tested database of images and it measures the specificity of image retrieval system, as presents in the following equation [21]:

where R_c represents the total number of correctly retrieved images and R_t represents the total number of retrieved images. Precision can also be expressed in the following equation.

Where TP represents true positive and FP represents false positive. In this work, top-10 and top-20 have been tested. Top-10 means the total number of retrieved images is 10 images, and top-20 means the total number of retrieved images is 20 images. Figs. 5 and 6 present examples for the query image based on top-10 and top-20.

Fig. 5. Query result for top-10.

Fig. 6. Query result for top-20.

5.3. Results

The experiments conducted in this work involve two phases: (a) Single layer CBIR model and (b) Bi-layer CBIR model. In the first phase, the single layer model (i.e., BoF technique) is evaluated alone, and on the other hand, CBIR technique based on extracting shape, texture, and color features is evaluated. In the second phase, the proposed bi-layer model is evaluated. The experiments are detailed in the following steps:

1. BoF-based CBIR technique is tested with different number of clusters, as BoF technique depends on the K-means clustering algorithm to create clusters, which is commonly called visual words. The number of clusters cannot be selected automatically; it needs manual selection. To select the proper number of clusters, (i.e., value of k-means), the different number of clusters have been tested to obtain the best precision result of BoF technique. The precision results of different numbers of clusters are illustrated in the following tables.

From Tables 1 and 2, one can observe that the best result is obtained when k = 500 for both top-10 and top-20.

TABLE 1: Precision rate of BoF technique for different number of clusters for top-20

TABLE 2: Precision rate of bof technique for different number of clusters for top-10

2. The proposed CBIR technique that relies on extracting shape, color, and texture features has been tested, and the results are presented in Table 3.

TABLE 3: Precision rate for the tested feature extractors for top-20 and top-10

3. The proposed bi-layer approach has been tested. It includes two layers: First layer implements BoF technique (for K=500) and M most similar images are retrieved, M is user defined. In the second layer, shape, color, and texture features are extracted from the query image and the M images, as a result, N most similar images are retrieved. The following tables investigate the best value of M. In other words, Tables 4 and 5 show testing different number of M for top-20 and top-10, respectively.

TABLE 4: Precision rate for different number of M for top-20

TABLE 5: Precision results for different number of M for top-10

Results in Tables 4 and 5 demonstrate that the best precision results are obtained when M = 200. For this reason, different small numbers of M, in the range of M = 100 to M = 300, are investigated to gain better precision results, Tables 6 and 7.

TABLE 6: Precision results for different number of M in the range 100–300 for top-20

TABLE 7: Precision results for different number of M in the range 100–300 for top-10

From Tables 6 and 7, it is quite clear that the best result is obtained when M =225 for both top-20 and top-10.

The ratio of correctly retrieved images over the total number of images of the semantic class in the image database is known as recall and it measures the sensitivity of the image retrieval system, equation (5):

Where Rc is the total number of retrieved images and Ts is the total number of images in the semantic class in the database. More experiments have been done to compare the proposed approach with the state-of-the-art techniques, Table 8. In all experiments, each image in the Corel-1K database is used as a query image. The retrieval performance of tested techniques is measured in terms of average retrieval precision (ARP) and average retrieval recall (ARR). The higher ARP and ARR values mean the better performance.

TABLE 8: ARP results of tested CBIR techniques

According to the results in Table 8, the best result is achieved by the proposed approach for both top-10 and top-20. All the tested state-of-the-art techniques, except technique in Al-Jubouri and Du [7], they tested their approach either for top-10 or for top-20, and this is why in Table 8 some cells do not contain the ARP and ARR results.

6. CONCLUSION

This paper has developed an effective CBIR technique for retrieving images from a wide range of images in the dataset. The proposed approach involves two layers; in the first layer, all images in the database are compared to the query image based on the BoF technique, and as a result, 225 most similar images to the query image are selected. Color, texture, and shape features are used in the second layer to extract significant features from the selected 225 images to retrieve the most similar images to the query image. The obtained results depicted that the proposed approach has reached an optimal average precision of 82.27% and 76.13% for top 10 and top 20, respectively.