DANNET: deep attention neural network for efficient ear identification in biometrics

Ensemble model overview

The proposed ensemble model integrates two YSegNets, each with a unique configuration to enhance segmentation accuracy. The first YSegNet utilizes a pretrained VGG-16 as the encoder and incorporates dense blocks to improve feature extraction and reuse throughout the network. Dense blocks allow for better gradient flow and prevent vanishing gradients, which is crucial for learning deep features effectively.

The second YSegNet deviates from the traditional structure by taking input directly and combining with output of the first YSegNet, instead of using a separate pretrained VGG-16 encoder. This setup enables the second YSegNet to process enhanced feature representations from the first YSegNet. Additionally, the second YSegNet employs attention blocks in the skip connections, allowing it to focus on the most salient features while filtering out irrelevant information. This attention mechanism enhances the feature transfer between the encoder and decoder, further refining the segmentation accuracy.

The rationale for this ensemble design is to capitalize on the complementary strengths of the two YSegNets. The first YSegNet, with its dense blocks, is highly effective at capturing detailed hierarchical features, while the second YSegNet, utilizing attention blocks, fine-tunes these features by concentrating on critical regions. By combining the outputs of both networks, the model delivers more accurate and robust segmentation results. The integration of these two architectures creates a synergy that boosts performance in handling complex segmentation tasks. The flow diagram and the schematic representation of the proposed model is given in Figs. 1 and 2, respectively.

Base model

YSegNet 1: The YSegNet model integrates VGG-16 as its encoder and employs dense blocks as skip connections to enhance its segmentation capabilities. By leveraging the VGG-16 network, YSegNet benefits from a robust and deep feature extraction backbone. VGG-16, pre-trained on extensive datasets, provides a strong foundation for capturing hierarchical and detailed features from input images. Its series of convolutional and pooling layers progressively downsample the input, effectively extracting rich, multi-scale features necessary for accurate segmentation.

In addition to the VGG-16 encoder, YSegNet incorporates dense blocks specifically as skip connections between the encoder and decoder. These dense blocks serve to facilitate the fusion of features from various layers of the network. Unlike traditional architectures where dense blocks are part of the encoder, their role here is to enhance the combination of high-resolution details with the features processed by VGG-16. Dense blocks concatenate outputs from multiple preceding layers, improving the integration and utilization of spatial information. This approach enhances the flow of information and ensures that detailed features are effectively preserved and utilized during the segmentation process.

The decoder in YSegNet utilizes transposed convolutions to upsample the feature maps, reconstructing the segmented output. The dense blocks used in the skip connections play a crucial role in this process by merging high-resolution features from the VGG-16 encoder with the upsampled features in the decoder. This combination of features ensures that the segmentation output is both precise and detailed, effectively reconstructing the input image’s important attributes.

YSegNet 2: The YSegNet 2 model builds upon the architecture of YSegNet 1 by incorporating advanced features and modifications to enhance its performance. One of the key innovations in YSegNet 2 is its utilization of the output from YSegNet 1 as input to its encoder. This approach ensures that YSegNet 2 benefits from the refined and high-quality features extracted by YSegNet 1, leading to a more cohesive and continuous flow of information between the two models. By leveraging the features from YSegNet 1, YSegNet 2 can enhance its feature representation and achieve improved segmentation accuracy.

In addition to integrating features from YSegNet 1, YSegNet 2 incorporates attention gates within its skip connections. These attention gates are designed to focus on the most relevant spatial regions in the feature maps, filtering out less important information. The attention mechanism is applied to the skip connections between the encoder and decoder, allowing the model to selectively emphasize critical features while suppressing irrelevant background information. This targeted approach improves the precision of feature selection and enhances the overall quality of the segmentation output.

The integration of attention gates in YSegNet 2 works in conjunction with the features received from YSegNet 1, ensuring that the model effectively utilizes both the detailed information from the initial model and the refined features enhanced by the attention mechanism. This combination of inputs and advanced skip connections contributes to more accurate and reliable segmentation results, making YSegNet 2 a powerful tool for complex and detailed image analysis tasks.

The implementation of attention gates as skip connections in YSegNet 2 involves a systematic approach to refining feature processing and enhancing segmentation accuracy. Initially, the attention gate takes inputs from both the encoder and decoder. The decoder’s features serve as the gating signal, guiding the attention mechanism in determining which regions of the encoder’s features should be emphasized or suppressed. This is achieved by computing an attention map through convolutional layers followed by a sigmoid activation function, which produces values between 0 and 1. This map effectively highlights important features while attenuating irrelevant ones.

The refined encoder features are obtained by applying this attention map through element-wise multiplication, thereby weighting the features according to their relevance. These processed features are then passed to the decoder, where they are upsampled and combined with the decoder’s own features. This integration ensures that the decoder receives only the most pertinent information, enhancing its ability to reconstruct accurate and detailed segmentation outputs.

By focusing attention on relevant spatial regions and filtering out less important information, attention gates significantly improve the precision of the segmentation results. This mechanism allows YSegNet 2 to better utilize the features from both the encoder and decoder, resulting in a more refined and effective segmentation model.

Performance metrics

Quantitative analysis is achieved using metrics derived from the confusion matrix (Shultz et al., 2011). The parameters considered are true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN) as defined by Kumar et al. (2018). The evaluated metrics range from 0% to 1% or 0% to 100%, with the model’s efficacy increasing as the metric values rise.

(a) Accuracy: Segmentation accuracy refers to the alignment between the segmented maps of ground truth and the prediction (Csurka et al., 2013) and is expressed by Eq. (1).

(1) $$Accuracy={\displaystyle \frac{T_P+T_N}{T_P+T_N+F_P+F_N}.}$$

(b) Recall: The accurate detection of the foreground in relation to the overall ground truth foreground is termed recall (Sokolova, Japkowicz & Szpakowicz, 2006) or sensitivity, and its computation is defined by Eq. (2).

(2) $$\mathrm{Re}call={\displaystyle \frac{T_P}{T_P+F_N}.}$$

(c) Precision: Precision, alternatively recognized as positive predictive value, gauges the precision of accurately predicted foreground pixels in relation to the total number of segmented foreground pixels, and this is articulated through Eq. (3).

(3) $$\mathrm{P}\mathrm{r}ecision={\displaystyle \frac{T_P}{T_P+F_P}.}$$

(d) Specificity: Specificity quantifies the accuracy of background prediction (Thanh, Prasath & Hien, 2019) and is determined using Eq. (4).

(4) $$Specificity={\displaystyle \frac{T_N}{T_N+F_P}.}$$

(e) DICE score: DICE score also known as F₁ score is calculated as the reciprocal of the average of the reciprocals of recall and precision (Kumar et al., 2018; Thanh et al., 2019b) and is given by Eq. (5).

(5) $$F_{1}score=2\times {\displaystyle \frac{\mathrm{Re}call\times \mathrm{P}\mathrm{r}ecision}{\mathrm{Re}call+\mathrm{P}\mathrm{r}ecision}.}$$

(f) IoU: The Intersection over Union (IoU) or Jaccard index metrics, as outlined in Thanh et al. (2019a), Taha & Hanbury (2015), represent the proportion of the intersection between elements in the true and predicted sets to the union of elements in these sets. This relationship is articulated in Eq. (6).

(6) $$IoU={\displaystyle \frac{DICE}{2-DICE}.}$$

Performance evaluation

Experimentation of the proposed framework is carried out on Google Colab, an open-source platform that offers a T4 GPU backend for Python programming. To enhance computational efficiency, the proposed ensemble model incorporates a pretrained VGG-16 as the initial encoder. The ensemble model is experimented with various competing encoder—decoder frameworks such as UNet, Double UNet (Jha et al., 2020), UNet++ (Zhou et al., 2018) and LS_YSegNet. Also, YSegNet with attention block as the skip connection represented as AG_YSegNet is considered for comparison. The proposed model is called as E_YSegNet (DANNET), where ‘E’ stands for ensemble. In the proposed work, YSegNet ensemble is used to enhance accuracy of segmentation in both constrained and unconstrained environment. This approach can help reduce errors and improve overall efficiency by considering multiple perspectives. The model undergoes a comprehensive evaluation encompassing both qualitative and quantitative analyses, with a qualitative comparison to existing methods presented in Fig. 3. The visual interpretation of segmented maps from the figure shows that in almost all cases the proposed model predicted the segmented maps effectively. All the edges of the segmented maps have a great similarity to the ground truth. Competing models were able to segment ear better in cases where the foreground ear occupied the major area of the image and performance of the models degraded in cases where the foreground was every small compared to the background. Due to the incorporation of dense bock, attention block and boundary extraction networks E_YSegNet outperformed other models in identifying the foreground and background correctly. The segmented maps of UNet and UNet++ was not effective even in cases where the foreground occupied the major area. This is because of the ambiguity or uncertainty in distinguishing between the foreground and background regions, especially if they share similar colours, textures, or structures. It is also noted that in the fifth case given in Fig. 3 the ear extension for the costume is also predicted by the competing models as ear. Whereas in the second case the boundary of the ear is not predicted desirably since some part of the ear is covered by hair. In such cases, E_YSegNet is able to predict efficiently as the model is trained based on the outputs from both the models. The validation accuracy and validation loss of LS_YSegNet, AG_YSegNet and E_YSegNet is plotted in Fig. 4 for 50 epochs. Validation accuracy and validation loss for E_YSegNet is highest and lowest, respectively, making it the most effective model for ear segmentation. Quantitative assessment of the proposed framework’s performance with other commonly used framewoks is tabulated in Table 1. The results indicate that the proposed ensemble model attains an overall accuracy of 98.93%. It can be seen that of the existing models, LS_YSegNet was only able to desirably predict foreground with a precision of 0.907 but AG_YSegNet outperformed with an increase of 2.4% in precision. The proposed ensemble performed remarkably with an increase of 5.4% in precision. Further, the proposed E_YSegNet improves the recall rate by 6.4%, while the DICE score and IoU parameters are increased by 5.9% and 10.8%, respectively. Finally, the execution time for each experimented model is analysed. The training and testing durations are outlined in Table 2. When training and testing models in Python using TensorFlow, execution time can be measured to assess performance. The suggested model exhibits the longest training duration due to its depth and complexity, while UNet demands the least time owing to its relative simplicity compared to other models. Additionally, it is noted that the testing time for the proposed model is significant. Consequently, the performance analysis indicates that the ensemble model incorporating a boundary extraction network outperforms others in obtaining a segmented map. This positions the proposed model as a valuable tool for ear detection across diverse applications.