Next Article in Journal
Research on Infrared Dim Target Detection Based on Improved YOLOv8
Previous Article in Journal
Significant Increase in African Water Vapor over 2001–2020
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Remote Sensing
Volume 16
Issue 16
10.3390/rs16162877
remotesensing-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Implicit Sharpness-Aware Minimization for Domain Generalization

by
Mingrong Dong
1,2,
Yixuan Yang
1,2,
Kai Zeng
1,2,
Qingwang Wang
1,2 and
Tao Shen
1,2,*
1
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Wujiaying Street, Kunming 650500, China
2
Yunnan Key Laboratory of Computer Technologies Application, Kunming University of Science and Technology, Wujiaying Street, Kunming 650500, China
*
Author to whom correspondence should be addressed.
Remote Sens. 2024, 16(16), 2877; https://doi.org/10.3390/rs16162877
Submission received: 14 May 2024 / Revised: 15 July 2024 / Accepted: 19 July 2024 / Published: 6 August 2024
(This article belongs to the Topic Computer Vision and Image Processing, 2nd Edition)
Browse Figures
Review Reports Versions Notes

Abstract

:
Domain generalization (DG) aims to learn knowledge from multiple related domains to achieve a robust generalization performance in unseen target domains, which is an effective approach to mitigate domain shift in remote sensing image classification. Although the sharpness-aware minimization (SAM) method enhances DG capability and improves remote sensing image classification performance by promoting the convergence of the loss minimum to a flatter loss surface, the perturbation loss (maximum loss within the neighborhood of a local minimum) of SAM fails to accurately measure the true sharpness of the loss landscape. Furthermore, its variants often overlook gradient conflicts, thereby limiting further improvement in DG performance. In this paper, we introduce implicit sharpness-aware minimization (ISAM), a novel method that addresses the deficiencies of SAM and mitigates gradient conflicts. Specifically, we demonstrate that the discrepancy in training loss during gradient ascent or descent serves as an equivalent measure of the dominant eigenvalue of the Hessian matrix. This discrepancy provides a reliable measure for sharpness. ISAM effectively reduces sharpness and mitigates potential conflicts between gradients by implicitly minimizing the discrepancy between training losses while ensuring a sufficiently low minimum through minimizing perturbation loss. Extensive experiments and analyses demonstrate that ISAM significantly enhances the model’s generalization ability on remote sensing and DG datasets, outperforming existing state-of-the-art methods.

1. Introduction

With the advancement of deep neural networks, deep learning techniques have achieved remarkable success in the field of the computer vision [1,2,3,4]. However, this success usually relies on the assumption that the training and test data have the same distribution. This enables an exceptional performance on training data (source domain), but the performance often degrades significantly when evaluated on test data (target domain), with distributions differing from the training data [5]. In remote sensing image acquisition, variations in geographic location, climate conditions, and sensors may exist. These factors give rise to domain shift phenomena for remote sensing images obtained at different times or locations, leading to changes in their distribution. Under such conditions, it is frequently necessary to retrain the model on data with different distributions to maintain its performance [6].
In order to ensure the model exhibits a robust performance across diverse data distributions, domain adaptation (DA) has been introduced as a solution for addressing out-of-distribution generalization [7,8]. DA necessitates access to well-labeled source domain data and unlabeled target domain data during training, leveraging these unlabeled target domain data to reduce the gap between the source and target domains [9,10]. However, acquiring target domain data can be both costly and challenging in real-word scenarios [11]. For instance, remote sensing data collection not only demands expensive specialized equipment, but also requires skilled personnel for operation. Moreover, due to the wide range of environmental factors involved, it becomes arduous to encompass all potential situations. These challenges hinder the effective generalization of the model through DA in the absence of target domain data [12].
Domain generalization (DG) effectively addresses the limitations of relying on target domain data in DA [13]. DG does not necessitate any target domain data during training, providing a robust solution for scenarios where target domain data are either inaccessible or completely absent. DG enables the model to generalize to unseen domains by training the model across multiple accessible source domains [14]. Existing DG techniques comprise data manipulation [15,16], domain invariant representation learning [17,18], meta-learning [19,20,21], and gradient operations [22,23]. For example, Zhou et al. [24] probabilistically mix feature statistics to achieve data manipulation, and Hu et al. [5] extract domain invariant representations through an inter-domain alignment and inter-domain expansion. However, a study known as Domainbed [25] has revealed that most existing DG methods have an inferior performance compared to the empirical risk minimization (ERM) method under identical evaluation conditions. These failures have prompted the exploration of new methods.
A recent study, named SAM [26], proposed a method to ensure the loss minimum is located on a flat loss surface, which significantly enhances the model’s DG performance. SAM introduces adversarial perturbations, ε , to the model parameters, θ , defining the loss at the point θ + ε as the maximum loss within the neighborhood of θ . By optimizing the perturbation loss, L θ + ε , instead of the original loss, L θ , this method aims to find a flat loss surface. However, it is important to note that both sharp and flat loss surfaces can achieve lower L θ + ε . The optimization objective employed by SAM does not accurately reflect the sharpness of the loss landscape, leading to minima that do not always converge to flat loss surfaces. Furthermore, most existing methods [27,28] primarily focus on optimizing the efficiency of SAM, neglecting this issue.
To address the issue present in SAM, Zhuang et al. [29] employed h θ = L θ + ε − L θ as an equivalent measure of sharpness and improved SAM by simultaneously optimizing L θ + ε and h θ . SAGM [30] further introduces an approach to simultaneously optimize L θ , L θ + ε , and h θ in order to find a flatter loss surface. However, these methods, which explicitly optimize combinations of L θ , L θ + ε , and h θ , result in the parameter update direction becoming a linear combination of the gradients ∇ L θ of the original loss and ∇ L θ + ε of the perturbation loss. It is important to note that conflicts may arise between different gradients, and the gradients obtained at different parameter points may exacerbate conflicts. Consequently, the linear combination of ∇ L θ and ∇ L θ + ε acquired at different locations may leads to a deviation between the actual parameter update direction and the expected update direction, resulting in a suboptimal optimization outcome.
In this paper, we aim to address the issue of SAM and mitigate the adverse effects caused by gradient conflicts, thereby enhancing DG capability. We have demonstrated that the discrepancy in training loss values before and after executing gradient ascent or descent can serve as an equivalent measure of the dominant eigenvalue of the Hessian matrix. This discrepancy can be substituted for h θ as an equivalent measure of the sharpness of the loss landscape. As the logits vector output by the model leads to changes in the loss, the discrepancy between the logits vectors obtained in two trainings implies a discrepancy in the losses before and after training, thereby presenting information about the sharpness. Based on these findings, this paper introduces a new implicit measure of sharpness and a novel method named implicit sharpness-aware minimization (ISAM). ISAM avoids obtaining gradients at different locations, mitigating the adverse effects of gradient conflicts. Additionally, the implicit measure of sharpness prevents the issue in SAM, where L θ + ε fails to accurately reflect the sharpness, thus avoiding convergence to sharp regions.
In summary, our contributions are as follows:
Firstly, we examined the limitations of SAM and its variants in finding flat loss surfaces. Additionally, this paper explores how the discrepancy in training loss before and after executing a gradient ascent or descent can serve as an equivalent measure of the dominant eigenvalue of the Hessian matrix.
Secondly, we introduced a novel implicit measure of sharpness and an ISAM method, which mitigates the adverse effects caused by gradient conflicts between ∇ L θ and ∇ L θ + ε while improving on the inadequacy of SAM. This improvement significantly enhances the model’s DG capabilities in unseen domains.
Finally, we conducted a series of experiments on several remote sensing datasets and DG datasets. The experimental results demonstrate that ISAM can effectively improve the generalization performance of the remote sensing image classification model, and the DG performance significantly outperforms current state-of-the-art DG methods.

2. Related Word

2.1. Domain Adaptation

DA enhances the generalization performance of the model across domains by narrowing the domain gap between the source and target domains [31]. Unsupervised DA, an effective approach within DA, improves the model’s generalization capabilities during training by utilizing labeled source domain data and unlabeled target domain data [32,33]. Another pioneering approach, source-free DA, allows model tuning using solely unlabeled target domain data [34,35,36]. These techniques enhance the generalization performance by enabling the model to adapt to diverse scenarios across various environments. However, the acquisition of target domain data is not always feasible in real-life situations, thus limiting the scope of DA applications [37]. In contrast to DA, the goal of DG is to train a model to generalize to unseen domains in the complete absence of target domain data, relying only on source domain data. This positions DG closer to real-world conditions and makes it a more effective solution for practical scenarios.

2.2. Domain Generalization

Existing DG methods can be broadly categorized into data manipulation, representation learning, meta-learning, and gradient manipulation. Data manipulation enables the training space of the model to broaden to cover more unknown domains by increasing the number and diversity of source domain samples. Blanchard et al. [16] adjusted the distribution of training data to better approximate the distribution of data in unseen domains. Zhou et al. [38] created new domains by mixing training sample styles. Representation learning seeks to identify stable and invariant features across different domains and transfer these learned features to other domains to enhance DG performance. Kim et al. [39] employed contrastive regularization to encourage the model to learn representations. Nam et al. [40] removed style coding from a category prediction task to mitigate the effects of style differences. Meta-learning develops a generalized model by learning from previous experiences or tasks. Li et al. [21] used a meta-learning approach to simulate virtual target domains during training. Li et al. [20] improved the quality of feature representations through meta-learning strategies and guided the optimization of the feature extractor based on the learning output of the feature critic. Gradient manipulation utilizes gradient information to compel the neural network to learn a generalized representation. Shi et al. [22] optimized the training process by aligning gradients across different domains. However, the failure of most existing DG methods in Domainbed has motivated further explorations for new methods.

2.3. Sharpness-Aware Minimization

Recent research analyzing 40 different complexity measures has revealed a high correlation between sharpness-based metrics and model generalization capabilities [41]. This discovery promotes an in-depth study of sharpness in model loss landscape. Foret et al. [26] introduced an efficient and generalized training method named SAM, which encourages the convergence of the minimum to a flat loss surface during training. Zhang et al. [42] introduced the concept of first-order flatness and identified a flat surface by examining the magnitude of gradients. Zhuang et al. [29] proposed GSAM, aimed at addressing the issue where SAM does not consistently converge the minimum to a flat loss surface. Wang et al. [30] introduced SAGM, utilizing a gradient matching technique, and successfully achieved a leading performance in DG tasks. Unfortunately, methods optimized for sharpness tend to improve SAM by explicitly weighing the relationship between original loss and perturbation loss. This overlooks potential conflicts between the gradients of the original loss and the perturbation loss, resulting in parameter updates in directions that diverge from the expected trajectory. In this paper, we propose a novel ISAM method to facilitate the convergence of the minimum to a flat loss surface while mitigating potential conflicts between gradients.

3. Methodology

3.1. Preliminaries

In this paper, we define the model with weight parameters, θ , denoted as f · ; θ . Assume there are K different source domains, D S = D S i i = 1 , 2 , ⋯ , K , where each source domain, D S i , contains N i sample-label pairs, ( x j i , y j i ) j = 1 N i . The source domains exhibit different joint distributions: P x y i ≠ P x y j for 1 ≤ i ≠ j ≤ K . The goal of DG is to train the model solely with source domains, enabling it to demonstrate a good generalization performance on any unseen target domain, D T = x j i j = 1 N t , where N t represents the number of samples in the target domain. The training loss, using empirical risk minimization (ERM) across all source domains, D S , is defined as follows:
L θ ; D = 1 K ∑ i = 1 K 1 N i ∑ j = 1 N i l f x j i ; θ , y j i
where l · ; · represents the loss function employed to measure the gap between the model’s predictions and the actual labels, and we assume that L θ ; D is twice differentiable.
The conventional ERM method often results in minimum converging to a sharp region of the loss landscape when optimizing weight parameters. As shown in Figure 1, although these sharp regions exhibit significantly low loss on source domain data, they are highly sensitive to slight parameter variations. This sensitivity can hinder the model’s generalization ability on unseen domain data. In contrast, SAM encourages the minimum to be located on a flat surface of the loss landscape through a two-step iterative process.
The first iteration finds the point θ + ε where the loss is maximized within the Euclidean space centered at θ with a radius ρ , where ρ ≥ 0 , by using an adversarial approach:
max ‖ ε ‖ ≤ ρ L θ + ε ; D
where ε represents the perturbation to θ , and ‖ · ‖ represents the L 2 norm. In the second iteration, the gradient ∇ L θ + ε ; D is computed through backpropagation at θ + ε , which is then utilized to update θ . Thus, the optimization objective of SAM can be formulated as follows:
min θ max ‖ ε ‖ ≤ ρ L θ + ε ; D
The goal is to seek a flat loss surface by maximizing L θ + ε ; D followed by minimizing it. The approximate value of perturbation ε , denoted as ε ^ , can be approximated through a first-order Taylor expansion:
ε ^ = a r g m a x ‖ ε ‖ ≤ ρ L θ + ε ; D ≈ a r g m a x ‖ ε ‖ ≤ ρ L θ ; D + ε T ∇ L θ ; D = ρ ∇ L θ ; D / ‖ ∇ L θ ; D ‖
Ultimately, the perturbation loss of SAM can be formulated as:
L p θ ; D = L θ + ε ^ ; D

3.2. SAM’s Optimization and Gradient Conflict

Although SAM has demonstrated effectiveness in experiments, both sharp and flat minima regions can yield low perturbation loss values, indicating that perturbation loss is not always consistent with sharpness. As illustrated in Figure 1, despite the flatter loss surface at θ 2 compared to θ 1 , L p θ 2 ; D = L p θ 1 ; D . This inconsistency may lead SAM to erroneously select θ 1 during the parameter selection process. Furthermore, although the perturbation loss at θ 3 is lower than that at θ 2 , the gap between L θ 3 ; D and L p θ 3 ; D is significantly greater than that between L θ 2 ; D and L p θ 2 ; D , prompting SAM to favor the sharper θ 3 over θ 2 . It can be seen that SAM focuses too much on the perturbation loss in the training procedure, which results in a minimum that does not necessarily converge to a flat surface.
Fortunately, in the definition of SAM, the perturbation loss is defined as the maximum loss within a specified range, while original loss is considered a local minimum within that area. Therefore, the gap between the perturbation loss and the original loss can effectively serve as a measure of sharpness, as explained below:
h θ ; D = L p θ ; D − L θ ; D
The function h θ ; D more accurately describes the gap between θ and θ + ε ^ , where a smaller gap indicates a flatter loss surface. By minimizing h θ ; D , it is possible to prevent the minimum from settling into a sharp loss region. Intuitively, optimizing both L p θ ; D and h θ ; D , or L θ ; D and h θ ; D simultaneously, becomes feasible. Minimizing L p θ ; D ensures that the minimum point is sufficiently low, while minimizing h θ ; D ensures that sharpness is also sufficiently small. As a result, it is possible to ensure convergence to a flat surface. The optimization objective is as follows:
min θ L p θ ; D , h θ ; D
The loss function can be formulated as α L p θ ; D + β h θ ; D . Its gradient is as follows:
∇ α L p θ ; D + β h θ ; D = α ∇ L p θ ; D + β ∇ L p θ ; D − ∇ L θ ; D = α + β ∇ L p θ ; D − β ∇ L θ ; D
where α and β are positive scalars utilized to adjust the weight of the two objectives within the overall optimization.
Similarly, minimizing L θ ; D ensures the attainment of a low loss, and minimizing h θ ; D guarantees that the minimum converges to a flat surface. The optimization objective is as follows:
min θ L θ ; D , h θ ; D
Its gradient can be represented as follows:
∇ α L θ ; D + β h θ ; D = α ∇ L θ ; D + β ∇ L p θ ; D − ∇ L θ ; D = α − β ∇ L θ ; D + β ∇ L p θ ; D
Since h θ ; D is closely related to L θ ; D and L p θ ; D , the method of explicitly combining L θ ; D , L p θ ; D , and h θ ; D eventually results in the gradient updating direction to be a linear combination of ∇ L θ ; D and ∇ L p θ ; D . Figure 2 illustrates the gradient update directions for L θ ; D , L p θ ; D , and h θ ; D . As depicted in Figure 3, updating the gradient in a manner such as in Equations (8) and (10) would result in the direction of the update being significantly different from the directions of − ∇ L θ ; D and − ∇ L p θ ; D . This results in updating parameters that focus on decreasing the value of one optimization goal in one direction while increasing the value of the other optimization goal. Ultimately, the conflict between ∇ L θ ; D and ∇ L p θ ; D manifests as follows: (1) the process of minimizing L θ ; D and h θ ; D results in reducing h θ ; D by increasing L θ ; D ; (2) the process of minimizing L p θ ; D and h θ ; D results in increasing L θ ; D to reduce the overall loss.

3.3. Implicit Sharpness-Aware Minimization

As previously discussed, by explicitly optimizing the original loss L θ ; D , the perturbation loss L p θ ; D , and h θ ; D , these actions ultimately translate into operations on ∇ L θ ; D and ∇ L p θ ; D . This optimization method had to face the adverse effects of conflicts between gradients. We must explore new methods to mitigate this drawback.
Assuming the loss function, L , can be approximated by a second-order Taylor expansion near a local minimum, θ , we proceed with the following analysis. In the gradient descent process, if the same mini batch of samples, B , is used in iterations, we observe that the change in loss can be expressed as:
L θ − δ ; D = L θ ; D − δ T ∇ L θ ; D + δ T ∇ 2 L θ ; D δ / 2 + O η 3
R θ ; D = L θ ; D − L θ − δ ; D = δ T ∇ L θ ; D − δ T ∇ 2 L θ ; D δ / 2 − O η 3
where δ = η ∇ L θ ; D represents the step size, η is the learning rate, and · denotes taking the absolute value. Around a local minimum, θ , ∇ 2 L θ ; D denotes the Hessian matrix at θ . σ m a x is defined as the dominant eigenvalue of the Hessian matrix. Consequently, we can derive:
R θ ; D = L θ ; D − L θ − δ ; D ≈ δ T ∇ 2 L θ ; D δ / 2
σ m a x ∇ 2 L θ ; D ≈ 2 R θ ; D / ‖ δ ‖ 2
Kaur et al. [43] have demonstrated that σ m a x can effectively measure the curvature at a minimum, allowing us to use R θ ; D to replace h θ ; D as an equivalent measure of sharpness.
Because changes in the logits vector can significantly influence the outcomes of the loss function calculations, we utilized the discrepancy between logits vectors to implicitly measure R θ ; D , thereby achieving an implicit measure of sharpness. For a mini batch B = x 1 , x 2 , ⋯ , x M containing M samples, the logits vector Z B = z 1 , z 2 , ⋯ , z M generated by the model f θ ; D and subsequently processed by the softmax function can be obtained as follows:
p x i = e x p z i ∑ j = 1 M e x p z j
Predicted probabilities p θ − δ x i and p θ x i are obtained from forward propagation at parameters θ − δ and θ , respectively. By implicitly optimizing R θ ; D , we aim to optimize sharpness. Consequently, the following method can serve as an implicit measure of sharpness:
1 M ∑ i = 1 M K L p θ − δ x i ‖ p θ x i
Simultaneously obtaining both p θ − δ x i and p θ x i necessitates an additional forward propagation step beyond SAM, leading to increased computation or memory consumption. We have discovered that, when performing a gradient ascent around a local minimum, θ , the discrepancy between losses can also serve as an equivalent measure of sharpness:
L θ ; D − L θ − δ ; D = L θ + δ ; D − L θ ; D ≈ δ T ∇ 2 L θ ; D δ / 2
Therefore, the discrepancy between p θ − δ x i and p θ x i , or p θ + δ x i and p θ x i , can serve as an implicit measure of sharpness. Before calculating ∇ L p θ ; D , SAM has already obtained the logits vector at the parameters θ + ε ^ . Moreover, the direction of δ = η ∇ L θ ; D aligns with that of ε ^ = ρ ∇ L θ ; D / ‖ ∇ L θ ; D ‖ , and both can be considered as increments for performing a gradient ascent at θ . To avoid additional computational overhead or memory consumption, we can equivalently substitute p θ + ε ^ x i obtained at the point θ + ε ^ for p θ + δ x i . Consequently, the implicit measure of sharpness term is defined as follows:
1 M ∑ i = 1 M K L p θ + ε ^ x i ‖ p θ x i
In Figure 4, we demonstrate the impact of the implicit measure of sharpness term on sharpness. It can be observed that lower sharpness can be obtained with the use of the implicit measure of sharpness. Our final optimization objective is:
L I S A M θ ; D = L p θ ; D + λ 1 M ∑ i = 1 M K L p θ + ε ^ x i ‖ p θ x i
where λ is a hyperparameter used to balance the perturbation loss and the implicit measure of sharpness term.
In order to address the shortcomings of SAM, previous work employed h θ ; D as a measure of sharpness. A flat loss surface was sought by minimizing h θ ; D , and the convergence of the minimum to a lower loss surface was ensured by minimizing either L θ ; D or L p θ ; D . Ultimately, the parameters were updated using ∇ L θ ; D and ∇ L p θ ; D obtained at θ and θ + ε ^ . Since the two gradients are obtained at different parameter locations, these methods for finding a flat loss surface ignore and exacerbate the likelihood of conflicts between gradients. This leads to a suboptimal outcome by increasing L θ ; D or L p θ ; D to minimize the overall loss.
In contrast to previous work, ISAM employs 1 M ∑ i = 1 M K L p θ + ε ^ x i ‖ p θ x i as an implicit measure of sharpness to address the issue that L p θ ; D cannot accurately reflect sharpness in SAM. This implicit measure enhances the model’s predictive consistency within the parameter space before and after perturbation, ensuring that the model’s output remains relatively stable, even with slight changes in parameters. By minimizing this implicit sharpness measure, the minimum converges to a flat surface. Concurrently, ISAM ensures that the minimum converges to a surface with lower loss by minimizing L p θ ; D . Since both optimization objectives of ISAM compute the gradient at θ + ε ^ , this greatly avoids exacerbating gradient conflicts. ISAM also mitigates the problem of previous work, where gradient conflicts between ∇ L θ ; D and ∇ L p θ ; D resulted in increasing L θ ; D or L p θ ; D to reduce the overall loss. Algorithm 1 describes the complete ISAM algorithm.
Algorithm 1: The Algorithm of ISAM
Input: Model f , source domains D s , initial weight θ 0 , learning rate η , training steps T, sample mini-batch B ∈ D s , hyperparameter λ .
Output: Model trained with ISAM:
1:
for  i ← 1  to T do
2:
 Compute the logits vector for the current parameters: p θ t = f B ; θ t ;
3:
 Calculate the original loss gradient: ∇ L θ t ; D ;
4:
 Calculate ε ^ according to Equation (4);
5:
 Compute the logits vector for perturbed parameters: P θ t + ε ^ = f B ; θ t + ε ^ ;
6:
  L I S A M = L p θ t ; D + λ 1 M ∑ i = 1 M K L p θ t + ε ^ x i ‖ p θ t x i ;
7:
 Compute the gradient approximation of the ISAM objective: g = ∇ L I S A M θ t ; D θ t + ε ^
8:
 Update weights: θ t + 1 = θ t − η g ;
9:
end for

4. Experiments

4.1. Experiment Setups and Implementation Details

4.1.1. Dataset

We conducted image classification tasks on four remote sensing datasets to evaluate the influence of our proposed method on the generalization performance of the models. The UC Merced Land-Use dataset [46] contains a total of 2100 images with dimensions of (3, 256, 256), spanning 21 categories. The SIRI-WHU dataset [45] contains a total of 2400 images with dimensions of (3, 200, 200), spanning 12 categories. The RSSCN7 dataset [47] contains a total of 2800 images with dimensions of (3, 400, 400), spanning 7 categories. The RSC11 dataset [48] contains a total of 1232 images with dimensions of (3, 400, 400), spanning 11 categories. Concurrently, due to the lack of a dedicated DG dataset within the remote sensing field, we conducted image classification tasks on 3 public datasets of DG to evaluate our proposed method and competing methods. The PACS dataset [49] contains a total of 9991 images with dimensions of (3, 224, 224), spanning 7 categories and 4 domains: Art, Cartoon, Photo, and Sketch. The VLCS dataset [50] includes a total of 10,729 images with dimensions of (3, 224, 224), covering 5 categories and 4 domains: Caltech101, LabelMe, SUN09, and VOC2007. The Office-Home dataset [51] consists of 15,588 images with dimensions of (3, 224, 224), across 65 categories and 4 domains: Art, Clipart, Product, and Real.

4.1.2. Evaluation Protocol

For remote sensing datasets, we used 80% of the dataset as the training set and 20% as the validation set. Four metrics were used to evaluate the performance: accuracy, precision, recall, and F1-score. For DG datasets, we followed the model selection strategy of training domain validation sets in DG, as per previous work [52]. The strategy splits the source domain into two parts, 80% and 20%, where 80% is used as the training set for training the model and 20% is used as the validation set for model selection. Ultimately, this strategy selects the model that performs best on the validation set of the source domain.

4.1.3. Implementation Details

For the remote sensing datasets, the pretrained ResNet18 and ResNet50 [44] models on the ImageNet [53] dataset were used as the backbones. All methods use the SGD optimizer as the base optimizer, with the learning rate set to 0.01, the batch size set to 32, and each training after 200 epochs. For the DG datasets, the pretrained ResNet18 model on the ImageNet dataset was used as the backbone. Hyperparameters were randomly searched from predefined distributions in the Domainbed benchmark: learning rates were chosen from 10 U n i f o r m − 5 , − 2 , batch sizes from 2 U n i f o r m 3.5 , 5 , weight decay from 10 U n i f o r m − 6 , − 2 , and dropout rate from 0 ,   0.1 ,   0.5 . Each training of the model went through 5000 iterations, with validation every 300 iterations.

4.1.4. Baseline

On the remote sensing datasets, we compared the proposed ISAM with empirical risk minimization (ERM) [54], sharpness-aware minimization (SAM) [26], and sharpness-aware gradient matching (SAGM) [30]. On the DG datasets, we compared the proposed method, ISAM, with conventional DG methods. The compared DG methods include: Mixup [55], MTL [16], Maximum Mean Discrepancy (MMD) [56], Class-conditional DANN (CDANN) [57], Adaptive Risk Minimization (ARM) [19], empirical risk minimization (ERM) [54], Group Distributionally Robust Optimization (GroupDRO) [58], Conditional Contrastive Adversarial Domain Bottleneck (CondCAD) [15], Deep CORrelation ALignment (CORAL) [59], Fish [22], Meta-Learning Domain Generalization (MLDG) [21], Fishr [23], Style-Agnostic Network (SagNet) [40], Self-Supervised Contrastive Regularization (SelfReg) [39], Variance Risk Extrapolation (VREx) [18], Spectral Decoupling (SD) [60], and Contrastive Adversarial Domain Bottleneck (CAD) [15].

4.2. Comparison Results

4.2.1. Comparison on Remote Sensing Datasets

We compared SAM, SAGM, and ISAM for their improvement of model generalization performance, and the results are shown in Table 1. In the SIRI-WHU and UC Merced Land-Use datasets, our approach achieved the best results on all four metrics. ISAM also demonstrated excellent performance on the RSC11 dataset. On ResNet18, ISAM’s accuracy improved by 2.3%, 0.7%, 2%, and 1.2% compared to ERM across the four datasets. Similarly, on ResNet50, ISAM’s accuracy improved by 1.7%, 0.9%, 2%, and 0.7% compared to ERM. ISAM exhibited a superior generalization performance improvement compared to SAM and SAGM. In Figure 5, we demonstrate the average accuracy, precision, recall, and F1-score of different methods on the four remote sensing datasets using ResNet18. Experimental results across these datasets indicate that ISAM can effectively enhance the generalization performance of the model in the remote sensing field.

4.2.2. Comparison on DG Datasets

Table 2 reports the results on the PACS dataset. Our method achieved the best results in the Photo domain and in terms of average accuracy, also outperforming the ERM baseline across all four domains. Additionally, we conducted a visual comparison between ISAM and ERM using Grad-CAM [61] in Figure 6. The Grad-CAM heatmaps show that ISAM is able to better focus on important feature regions of the images while covering a broader area. As detailed in Table 3, our method excelled in the Caltech, VOC, and Sun domains of the VLCS dataset and achieved superior average accuracy. Its performance was only marginally lower than the ERM method by 0.2% in the LabelMe domain. Moreover, in Table 3, we note that the performance of the ERM method exceeds more than half of the compared methods. This phenomenon further demonstrates that methods that converge the minimum to a flat surface can significantly enhance DG performance compared to other approaches. The results for the Office-Home dataset are shown in Table 4. Our method still outperforms the other compared DG methods and achieved the best results in the Clipart and Real domains. Collectively, the results across all three datasets demonstrate the effectiveness of our proposed method in enhancing DG performance.

4.3. Ablation Study and Parameter Analysis

4.3.1. Ablation Study on DG Datasets

To clearly demonstrate the capability of the proposed ISAM method in finding flat loss surfaces, we conducted ablation experiments with ERM, SAM, SAGM, and ISAM. The experimental results are detailed in Table 5, Table 6 and Table 7. Experiments across three datasets showed that the average accuracy of the SAM, SAGM, and ISAM methods all exceeded that of the ERM method. This indicates that methods enabling the minimum to converge to flat loss surfaces can effectively enhance DG performance. Compared to SAM, the improvements made by ISAM are significant. Apart from the Sun domain in the VLCS dataset, ISAM outperformed SAM across all domains and in average accuracy. Figure 7 demonstrates the feature clustering effects of SAM and ISAM, and it can be observed that ISAM has a better clustering effect.
Table 8 shows the optimization objectives of different methods. Compared to SAGM, which has the optimization objective of L θ ; D + L p θ − λ ∇ θ L θ ; D ; D , ISAM leads by 0.6%, 0.5%, and 0.9% in accuracy on the PACS, VLCS, and Office-Home datasets, respectively. This fully demonstrates that the implicit measure of sharpness can effectively mitigate the adverse effects caused by conflicts between ∇ L θ ; D and ∇ L p θ ; D .

4.3.2. Ablation Study on CIFAR-10

We conducted ablation experiments on the CIFAR-10 [63] dataset using ResNet18 to compare ERM, SAM, SAGM, and ISAM. As depicted in Figure 8, ISAM demonstrated superior performance among these four methods. Although SAGM, an improved version of SAM, outperformed ERM, its accuracy was slightly lower than that of SAM. These results further validate that ISAM effectively mitigates the adverse effects of conflicts between ∇ L θ ; D and ∇ L p θ ; D while improving SAM.

4.3.3. Parameter Analysis

We conducted a study on the hyperparameter, λ , for the implicit measure of sharpness term. λ is set to (0, 0.1, 0.2, 0.3, 0.4, 0.5). The experiments were carried out with all other parameters remaining constant, and the sequence of data used in each trial was fixed. Figure 9 illustrates the impact of different λ values on the generalization performance within the PACS, VLCS, and Office-Home datasets. It can be observed that the addition of the implicit measure of sharpness term effectively enhances DG performance.

5. Discussion

In this work, we introduce ISAM by proposing a novel measure of sharpness. ISAM addresses the limitations of SAM and mitigates the adverse effects caused by gradient conflicts. We validated the effectiveness of ISAM through extensive experiments on various remote sensing and DG datasets. Although ISAM has made progress in terms of DG performance, it has not reduced computational complexity during optimization. Consequently, the computational time and memory consumption increase when handling large-scale datasets. Our future research will focus on optimizing performance while incorporating more efficient computational techniques to reduce computational costs.

6. Conclusions

This paper provides a detailed analysis of the issues encountered by SAM and its variants in optimizing the sharpness of the loss landscape. In particular, we analyze the adverse effects resulting from gradient conflicts between the original loss and the perturbation loss. To mitigate these issues, we introduce an implicit measure of sharpness. Subsequently, we propose an algorithm named ISAM, which promotes the convergence of the loss minimum to a flat loss surface by minimizing the perturbation loss and the implicit measure of sharpness. ISAM effectively mitigates the adverse effects of conflicts between gradients while improving SAM. The extensive experiments on several remote sensing and DG datasets demonstrate that ISAM effectively enhances the model’s DG performance.

Author Contributions

Conceptualization, M.D., Y.Y. and T.S.; methodology, M.D. and Y.Y.; software, Y.Y.; validation, Y.Y.; formal analysis, Y.Y.; resources, M.D., K.Z., Q.W. and T.S.; data curation, Y.Y.; writing—original draft preparation, Y.Y.; writing—review and editing, M.D.; visualization, Y.Y.; supervision, T.S.; project administration, Y.Y.; funding acquisition, M.D., K.Z. and T.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work was funded in part by the Yunnan Fundamental Research Projects under Grants 202301AV070003 and 202101BE070001-008, and in part by the Major Science and Technology Projects in Yunnan Province under Grants 202302AG050009 and 202202AD080013.

Data Availability Statement

Data for this article can be obtained by contacting the author.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Aggarwal, K.; Singh, S.K.; Chopra, M.; Kumar, S.; Colace, F. Deep learning in robotics for strengthening industry 4.0.: Opportunities, challenges and future directions. In Robotics and AI for Cybersecurity and Critical Infrastructure in Smart Cities; Springer: Cham, Switzerland, 2022; pp. 1–19. [Google Scholar] [CrossRef]
  2. Jiang, W.; Yang, H.; Zhang, Y.; Kwok, J. An adaptive policy to employ sharpness-aware minimization. arXiv 2023. [Google Scholar] [CrossRef]
  3. Tsuneki, M. Deep learning models in medical image analysis. J. Oral Biosci. 2022, 64, 312–320. [Google Scholar] [CrossRef] [PubMed]
  4. Yang, B.; Wang, C.; Ma, X.; Song, B.; Liu, Z.; Sun, F. Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization. Remote Sens. 2024, 16, 1653. [Google Scholar] [CrossRef]
  5. Hu, J.; Qi, L.; Zhang, J.; Shi, Y. Domain generalization via Inter-domain Alignment and Intra-domain Expansion. Pattern Recognit. 2024, 146, 110029. [Google Scholar] [CrossRef]
  6. Zhang, X.; Chen, Y.C. Adaptive Domain Generalization Via Online Disagreement Minimization. IEEE Trans. Image Process. 2023, 32, 4247–4258. [Google Scholar] [CrossRef] [PubMed]
  7. Xu, Q.; Zhang, R.; Fan, Z.; Wang, Y.; Wu, Y.-Y.; Zhang, Y. Fourier-based augmentation with applications to domain generalization. Pattern Recognit. 2023, 139, 109474. [Google Scholar] [CrossRef]
  8. Guan, H.; Liu, M. Domain Adaptation for Medical Image Analysis: A Survey. IEEE Trans. Biomed. Eng. 2022, 69, 1173–1185. [Google Scholar] [CrossRef] [PubMed]
  9. Shao, Y.; Li, L.; Ren, W.; Gao, C.; Sang, N. Domain adaptation for image dehazing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 2808–2817. [Google Scholar]
  10. Hsu, H.-K.; Yao, C.-H.; Tsai, Y.-H.; Hung, W.-C.; Tseng, H.-Y.; Singh, M.; Yang, M.-H. Progressive domain adaptation for object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA, 2–5 March 2020; pp. 749–757. [Google Scholar]
  11. Niu, Z.; Yuan, J.; Ma, X.; Xu, Y.; Liu, J.; Chen, Y.W.; Tong, R.; Lin, L. Knowledge Distillation-based Domain-invariant Representation Learning for Domain Generalization. IEEE Trans. Multimed. 2023, 26, 245–255. [Google Scholar] [CrossRef]
  12. Zhou, K.; Liu, Z.; Qiao, Y.; Xiang, T.; Loy, C.C. Domain Generalization: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 4396–4415. [Google Scholar] [CrossRef] [PubMed]
  13. Wang, J.; Lan, C.; Liu, C.; Ouyang, Y.; Qin, T.; Lu, W.; Chen, Y.; Zeng, W.; Yu, P.S. Generalizing to Unseen Domains: A Survey on Domain Generalization. IEEE Trans. Knowl. Data Eng. 2023, 35, 8052–8072. [Google Scholar] [CrossRef]
  14. Eastwood, C.; Robey, A.; Singh, S.; Von Kügelgen, J.; Hassani, H.; Pappas, G.J.; Schölkopf, B. Probable domain generalization via quantile risk minimization. In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, LO, USA & Online, 28 November–9 December; Volume 35, pp. 17340–17358.
  15. Dubois, Y.; Ruan, Y.; Maddison, C.J. Optimal representations for covariate shifts. In Proceedings of the Thirty-Fifth Annual Conference on Neural Information Processing Systems (NeurIPS 2021), Online, 6–14 December 2021. [Google Scholar]
  16. Blanchard, G.; Deshmukh, A.A.; Dogan, U.; Lee, G.; Scott, C. Domain generalization by marginal transfer learning. J. Mach. Learn. Res. 2021, 22, 1–55. [Google Scholar]
  17. Dayal, A.; KB, V.; Cenkeramaddi, L.R.; Mohan, C.; Kumar, A.; Balasubramanian, N.V. MADG: Margin-based Adversarial Learning for Domain Generalization. In Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA, 10–16 December 2023; Volume 36, pp. 58938–58952. [Google Scholar]
  18. Krueger, D.; Caballero, E.; Jacobsen, J.-H.; Zhang, A.; Binas, J.; Zhang, D.; Le Priol, R.; Courville, A. Out-of-distribution generalization via risk extrapolation (rex). In Proceedings of the International Conference on Machine Learning, Online, 18–24 July 2021; pp. 5815–5826. [Google Scholar]
  19. Zhang, M.M.; Marklund, H.; Dhawan, N.; Gupta, A.; Levine, S.; Finn, C. Adaptive risk minimization: A meta-learning approach for tackling group shift. arXiv 2020. [Google Scholar] [CrossRef]
  20. Li, Y.; Yang, Y.; Zhou, W.; Hospedales, T. Feature-critic networks for heterogeneous domain generalization. In Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA, 10–15 June 2019; pp. 3915–3924. [Google Scholar]
  21. Li, D.; Yang, Y.; Song, Y.-Z.; Hospedales, T. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 32. [Google Scholar] [CrossRef]
  22. Shi, Y.; Seely, J.; Torr, P.H.; Siddharth, N.; Hannun, A.; Usunier, N.; Synnaeve, G. Gradient matching for domain generalization. arXiv 2021. [Google Scholar] [CrossRef]
  23. Rame, A.; Dancette, C.; Cord, M. Fishr: Invariant gradient variances for out-of-distribution generalization. In Proceedings of the International Conference on Machine Learning, Baltimore, MA, USA, 17–23 July 2022; pp. 18347–18377. [Google Scholar]
  24. Zhou, K.; Yang, Y.; Qiao, Y.; Xiang, T. MixStyle Neural Networks for Domain Generalization and Adaptation. Int. J. Comput. Vis. 2024, 132, 822–836. [Google Scholar] [CrossRef]
  25. Gulrajani, I.; Lopez-Paz, D. In search of lost domain generalization. arXiv 2020. [Google Scholar] [CrossRef]
  26. Foret, P.; Kleiner, A.; Mobahi, H.; Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv 2020. [Google Scholar] [CrossRef]
  27. Du, J.; Yan, H.; Feng, J.; Zhou, J.T.; Zhen, L.; Goh, R.S.M.; Tan, V.Y. Efficient sharpness-aware minimization for improved training of neural networks. arXiv 2021. [Google Scholar] [CrossRef]
  28. Liu, Y.; Mai, S.; Chen, X.; Hsieh, C.J.; You, Y. Towards Efficient and Scalable Sharpness-Aware Minimization. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 12350–12360. [Google Scholar] [CrossRef]
  29. Zhuang, J.; Gong, B.; Yuan, L.; Cui, Y.; Adam, H.; Dvornek, N.; Tatikonda, S.; Duncan, J.; Liu, T. Surrogate gap minimization improves sharpness-aware training. arXiv 2022. [Google Scholar] [CrossRef]
  30. Wang, P.; Zhang, Z.; Lei, Z.; Zhang, L. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 3769–3778. [Google Scholar]
  31. Wilson, G.; Cook, D.J. A survey of unsupervised deep domain adaptation. ACM Trans. Intell. Syst. Technol. 2020, 11, 51. [Google Scholar] [CrossRef] [PubMed]
  32. Zhao, S.; Yue, X.; Zhang, S.; Li, B.; Zhao, H.; Wu, B.; Krishna, R.; Gonzalez, J.E.; Sangiovanni-Vincentelli, A.L.; Seshia, S.A.; et al. A Review of Single-Source Deep Unsupervised Visual Domain Adaptation. IEEE Trans. Neural Netw. Learn. Syst. 2022, 33, 473–493. [Google Scholar] [CrossRef] [PubMed]
  33. Zhang, Y.; Wei, Y.; Wu, Q.; Zhao, P.; Niu, S.; Huang, J.; Tan, M. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Trans. Image Process. 2020, 29, 7834–7844. [Google Scholar] [CrossRef]
  34. Li, R.; Jiao, Q.; Cao, W.; Wong, H.-S.; Wu, S. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 9641–9650. [Google Scholar]
  35. Liang, J.; Hu, D.; Feng, J. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the International Conference on Machine Learning, Online, 12–18 July 2020; pp. 6028–6039. [Google Scholar]
  36. Yang, S.; Wang, Y.; Van De Weijer, J.; Herranz, L.; Jui, S. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 8978–8987. [Google Scholar]
  37. Li, D.; Wu, A.; Wang, Y.; Han, Y. Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization. arXiv 2024. [Google Scholar] [CrossRef]
  38. Zhou, K.; Yang, Y.; Qiao, Y.; Xiang, T. Domain generalization with mixstyle. arXiv 2021. [Google Scholar] [CrossRef]
  39. Kim, D.; Yoo, Y.; Park, S.; Kim, J.; Lee, J. SelfReg: Self-supervised Contrastive Regularization for Domain Generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 9599–9608. [Google Scholar] [CrossRef]
  40. Nam, H.; Lee, H.; Park, J.; Yoon, W.; Yoo, D. Reducing Domain Gap by Reducing Style Bias. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 8686–8695. [Google Scholar] [CrossRef]
  41. Jiang, Y.; Neyshabur, B.; Mobahi, H.; Krishnan, D.; Bengio, S. Fantastic generalization measures and where to find them. arXiv 2019. [Google Scholar] [CrossRef]
  42. Zhang, X.; Xu, R.; Yu, H.; Zou, H.; Cui, P. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 20247–20257. [Google Scholar]
  43. Kaur, S.; Cohen, J.; Lipton, Z.C. On the maximum hessian eigenvalue and generalization. In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, LO, USA & Online, 28 November–9 December 2022; pp. 51–65. [Google Scholar]
  44. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
  45. Zhu, Q.; Zhong, Y.; Zhao, B.; Xia, G.S.; Zhang, L. Bag-of-Visual-Words Scene Classifier With Local and Global Features for High Spatial Resolution Remote Sensing Imagery. IEEE Geosci. Remote Sens. Lett. 2016, 13, 747–751. [Google Scholar] [CrossRef]
  46. Yang, Y.; Newsam, S. Bag-of-visual-words and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA, 2–5 November 2010; pp. 270–279. [Google Scholar]
  47. Zou, Q.; Ni, L.; Zhang, T.; Wang, Q. Deep learning based feature selection for remote sensing scene classification. IEEE Geosci. Remote Sens. Lett. 2015, 12, 2321–2325. [Google Scholar] [CrossRef]
  48. Zhao, L.; Tang, P.; Huo, L. Feature significance-based multibag-of-visual-words model for remote sensing image scene classification. J. Appl. Remote Sens. 2016, 10, 035004. [Google Scholar] [CrossRef]
  49. Li, D.; Yang, Y.; Song, Y.Z.; Hospedales, T.M. Deeper, Broader and Artier Domain Generalization. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 5543–5551. [Google Scholar] [CrossRef]
  50. Fang, C.; Xu, Y.; Rockmore, D.N. Unbiased Metric Learning: On the Utilization of Multiple Datasets and Web Images for Softening Bias. In Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 1657–1664. [Google Scholar] [CrossRef]
  51. Venkateswara, H.; Eusebio, J.; Chakraborty, S.; Panchanathan, S. Deep Hashing Network for Unsupervised Domain Adaptation. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 5385–5394. [Google Scholar] [CrossRef]
  52. Chen, L.; Zhang, Y.; Song, Y.; Shan, Y.; Liu, L. Improved test-time adaptation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 24172–24182. [Google Scholar]
  53. Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Kai, L.; Li, F.-F. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar] [CrossRef]
  54. Vapnik, V.N. An overview of statistical learning theory. IEEE Trans. Neural Netw. 1999, 10, 988–999. [Google Scholar] [CrossRef] [PubMed]
  55. Yan, S.; Song, H.; Li, N.; Zou, L.; Ren, L. Improve unsupervised domain adaptation with mixup training. arXiv 2020. [Google Scholar] [CrossRef]
  56. Li, H.; Pan, S.J.; Wang, S.; Kot, A.C. Domain Generalization with Adversarial Feature Learning. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 5400–5409. [Google Scholar] [CrossRef]
  57. Li, Y.; Tian, X.; Gong, M.; Liu, Y.; Liu, T.; Zhang, K.; Tao, D. Deep Domain Generalization via Conditional Invariant Adversarial Networks. In Proceedings of the Computer Vision—ECCV 2018, Munich, Germany, 8–14 September 2018; pp. 647–663. [Google Scholar] [CrossRef]
  58. Sagawa, S.; Koh, P.W.; Hashimoto, T.B.; Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv 2019. [Google Scholar] [CrossRef]
  59. Sun, B.; Saenko, K. Deep CORAL: Correlation Alignment for Deep Domain Adaptation. In Proceedings of the Computer Vision—ECCV 2016 Workshops, Amsterdam, The Netherlands, 11–14 October 2016; pp. 443–450. [Google Scholar] [CrossRef]
  60. Pezeshki, M.; Kaba, O.; Bengio, Y.; Courville, A.C.; Precup, D.; Lajoie, G. Gradient starvation: A learning proclivity in neural networks. In Proceedings of the Thirty-Fifth Annual Conference on Neural Information Processing Systems (NeurIPS 2021), Online, 6–14 December 2021; Volume 34, pp. 1256–1272. [Google Scholar]
  61. Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 618–626. [Google Scholar]
  62. Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605. [Google Scholar]
  63. Krizhevsky, A.; Hinton, G. Learning Multiple Layers of Features From Tiny Images; University of Toronto: Toronto, ON, Canada, 2009. [Google Scholar]
Remotesensing 16 02877 g001
Figure 1. The case of different minima points in the loss landscape. Specifically, θ 3 situated within the sharp region exhibits a large generalization error, whereas θ 2 situated within the flatter region demonstrates a smaller generalization error.
Figure 1. The case of different minima points in the loss landscape. Specifically, θ 3 situated within the sharp region exhibits a large generalization error, whereas θ 2 situated within the flatter region demonstrates a smaller generalization error.
Remotesensing 16 02877 g001
Remotesensing 16 02877 g002
Figure 2. Illustration of the gradient update directions using different losses at parameter θ . The direction indicated by − ∇ L θ ; D is the gradient update direction of the original loss L θ ; D , the direction indicated by − ∇ L p θ ; D is the gradient update direction of the perturbation loss L p θ ; D , and the direction indicated by ∇ L θ ; D − ∇ L p θ ; D is the gradient update direction of h θ ; D .
Figure 2. Illustration of the gradient update directions using different losses at parameter θ . The direction indicated by − ∇ L θ ; D is the gradient update direction of the original loss L θ ; D , the direction indicated by − ∇ L p θ ; D is the gradient update direction of the perturbation loss L p θ ; D , and the direction indicated by ∇ L θ ; D − ∇ L p θ ; D is the gradient update direction of h θ ; D .
Remotesensing 16 02877 g002
Remotesensing 16 02877 g003
Figure 3. Potential directions for gradient updates: The pink region represents the possible directions of gradient updates when the optimization target is m i n θ L p θ ; D , h θ ; D . The green region indicates the possible directions of gradient updates when the optimization target is m i n θ L θ ; D , h θ ; D .
Figure 3. Potential directions for gradient updates: The pink region represents the possible directions of gradient updates when the optimization target is m i n θ L p θ ; D , h θ ; D . The green region indicates the possible directions of gradient updates when the optimization target is m i n θ L θ ; D , h θ ; D .
Remotesensing 16 02877 g003
Remotesensing 16 02877 g004
Figure 4. The impact of the implicit measure of sharpness term on sharpness. Sharpness is denoted by h θ ; D = L p θ ; D − L θ ; D . The results were obtained using ResNet18 [44] on the SIRI-WHU dataset [45].
Figure 4. The impact of the implicit measure of sharpness term on sharpness. Sharpness is denoted by h θ ; D = L p θ ; D − L θ ; D . The results were obtained using ResNet18 [44] on the SIRI-WHU dataset [45].
Remotesensing 16 02877 g004
Remotesensing 16 02877 g005
Figure 5. Average accuracy, precision, recall, and F1-score comparison of ISAM with ERM, SAM, and SAGM on four remote sensing datasets.
Figure 5. Average accuracy, precision, recall, and F1-score comparison of ISAM with ERM, SAM, and SAGM on four remote sensing datasets.
Remotesensing 16 02877 g005
Remotesensing 16 02877 g006
Figure 6. The Grad-CAM heatmap visualization. The results were obtained using ResNet18 on the PACS dataset.
Figure 6. The Grad-CAM heatmap visualization. The results were obtained using ResNet18 on the PACS dataset.
Remotesensing 16 02877 g006
Remotesensing 16 02877 g007
Figure 7. The t-SNE [62] visualization shows the clustering effects of SAM and ISAM. The results were obtained using ResNet18 on the PACS dataset. (a) Target domain: Art. (b) Target domain: Photo.
Figure 7. The t-SNE [62] visualization shows the clustering effects of SAM and ISAM. The results were obtained using ResNet18 on the PACS dataset. (a) Target domain: Art. (b) Target domain: Photo.
Remotesensing 16 02877 g007
Remotesensing 16 02877 g008
Figure 8. Classification accuracy of ERM, SAM, SAGM, and ISAM evaluated by ResNet18 on CIFAR10.
Figure 8. Classification accuracy of ERM, SAM, SAGM, and ISAM evaluated by ResNet18 on CIFAR10.
Remotesensing 16 02877 g008
Remotesensing 16 02877 g009aRemotesensing 16 02877 g009b
Figure 9. The impact of the hyperparameter λ of the implicit measure of sharpness term on DG performance in ISAM. Results on different datasets with varying λ values using ResNet18: (a) PACS; (b) VLCS; (c) Office-Home.
Figure 9. The impact of the hyperparameter λ of the implicit measure of sharpness term on DG performance in ISAM. Results on different datasets with varying λ values using ResNet18: (a) PACS; (b) VLCS; (c) Office-Home.
Remotesensing 16 02877 g009aRemotesensing 16 02877 g009b
Table 1. Comparison of ERM, SAM, SAGM, and ISAM on remote sensing datasets. The best and second-best results are indicated in bold and underlined, respectively.
Table 1. Comparison of ERM, SAM, SAGM, and ISAM on remote sensing datasets. The best and second-best results are indicated in bold and underlined, respectively.
ResNet18ResNet50
DatasetMethodAccuracyPrecisionRecallF1-scoreAccuracyPrecisionRecallF1-Score
SIRI-WHUERM95.495.495.395.396.696.796.596.6
SAM96.796.896.796.797.297.397.497.3
SAGM96.996.996.996.997.998.097.897.9
ISAM (ours)97.797.697.797.798.398.298.298.2
RSSCN7ERM96.496.496.496.496.896.996.996.8
SAM96.896.796.896.797.197.297.297.2
SAGM97.096.897.397.097.397.497.397.3
ISAM (ours)97.197.197.297.297.797.797.897.7
RSC11ERM96.096.496.296.197.297.597.497.4
SAM97.297.397.097.198.898.898.698.7
SAGM98.098.497.698.098.898.798.898.7
ISAM (ours)98.097.897.897.899.299.099.399.1
UC Merced Land-UseERM97.497.697.697.598.698.598.698.5
SAM97.497.497.797.598.898.798.798.6
SAGM97.998.097.897.899.099.099.099.0
ISAM (ours)98.698.498.498.499.399.299.299.2
Table 2. Comparison with state-of-the-art domain generalization methods on PACS dataset. The best and second-best results are indicated in bold and underlined, respectively. The results marked with † are from [52].
Table 2. Comparison with state-of-the-art domain generalization methods on PACS dataset. The best and second-best results are indicated in bold and underlined, respectively. The results marked with † are from [52].
MethodArtCartoonPhotoSketchAverage
Mixup80.771.794.671.379.6
MTL †78.773.494.174.480.2
MMD77.976.794.271.980.2
CDANN †80.473.793.174.280.4
ARM †79.475.094.373.880.6
ERM78.774.495.174.780.7
GroupDRO †77.776.494.074.880.7
CondCAD †79.774.294.674.880.8
CORAL79.777.493.873.781.2
Fish77.777.194.575.581.2
MLDG †78.475.194.876.781.3
Fishr †81.275.894.373.881.3
SagNet †82.973.294.676.181.7
SelfReg82.574.495.474.981.8
VREx78.875.494.079.281.9
SD †83.274.694.675.181.9
CAD †83.974.294.675.081.9
ISAM (ours)82.674.895.875.582.2
Table 3. Comparison with state-of-the-art domain generalization methods on VLCS dataset. The best and second-best results are indicated in bold and underlined, respectively. The results marked with † are from [52].
Table 3. Comparison with state-of-the-art domain generalization methods on VLCS dataset. The best and second-best results are indicated in bold and underlined, respectively. The results marked with † are from [52].
MethodCaltechLabelMeSunVOCAverage
MMD96.064.368.570.874.9
MTL †94.465.069.671.775.2
MLDG †95.863.368.573.175.2
CAD †94.563.570.472.475.2
VREx96.262.569.373.175.3
GroupDRO †96.761.770.272.975.4
SagNet94.961.969.675.275.4
SD †96.562.269.773.675.5
CORAL96.562.869.173.875.6
ERM97.762.170.373.275.8
ARM †96.961.971.673.375.9
CDANN †95.462.669.976.276.0
CondCAD †96.562.669.176.076.1
Fishr †97.263.370.474.076.2
Mixup95.662.771.375.476.3
SelfReg95.863.471.175.376.4
Fish97.463.471.575.276.9
ISAM (ours)98.761.971.877.977.6
Table 4. Comparison with state-of-the-art domain generalization methods on Office-Home dataset. The best and second-best results are indicated in bold and underlined, respectively. The results marked with † are from [52].
Table 4. Comparison with state-of-the-art domain generalization methods on Office-Home dataset. The best and second-best results are indicated in bold and underlined, respectively. The results marked with † are from [52].
MethodArtClipartProductRealAverage
VREx49.246.268.068.057.9
MMD49.246.769.470.358.9
CDANN †51.446.968.470.459.3
ARM †51.348.568.070.659.6
MTL †51.647.769.171.059.9
ERM52.147.170.070.559.9
CAD †52.148.369.771.960.5
GroupDRO †52.648.269.971.560.6
Fishr †52.648.669.972.460.9
MLDG †53.148.470.571.760.9
CondCAD †53.348.469.872.661.0
Fish55.649.171.471.762.0
SagNet56.549.670.672.262.2
SelfReg55.149.272.273.062.4
Mixup55.949.871.672.462.4
SD †55.051.372.572.762.9
CORAL55.451.571.873.263.0
ISAM (ours)55.852.272.273.463.4
Table 5. The ablation study of ISAM on the PACS dataset. The best and second-best results are indicated in bold and underlined, respectively. All results were obtained by a hyperparameter search of Domainbed.
Table 5. The ablation study of ISAM on the PACS dataset. The best and second-best results are indicated in bold and underlined, respectively. All results were obtained by a hyperparameter search of Domainbed.
MethodArtCartoonPhotoSketchAverage
ERM78.774.495.174.780.7
SAM81.074.295.174.881.3
SAGM81.674.395.375.181.6
ISAM82.674.895.875.582.2
Table 6. The ablation study of ISAM on the VLCS dataset. The best and second-best results are indicated in bold and underlined, respectively. All results were obtained by a hyperparameter search of Domainbed.
Table 6. The ablation study of ISAM on the VLCS dataset. The best and second-best results are indicated in bold and underlined, respectively. All results were obtained by a hyperparameter search of Domainbed.
MethodCaltechLabelMeSunVOCAverage
ERM97.762.170.373.275.8
SAM98.660.772.076.877.0
SAGM97.862.671.077.077.1
ISAM98.761.971.877.977.6
Table 7. The ablation study of ISAM on the Office-Home dataset. The best and second-best results are indicated in bold and underlined, respectively. All results were obtained by a hyperparameter search of Domainbed.
Table 7. The ablation study of ISAM on the Office-Home dataset. The best and second-best results are indicated in bold and underlined, respectively. All results were obtained by a hyperparameter search of Domainbed.
MethodArtClipartProductRealAverage
ERM52.147.170.070.559.9
SAM54.149.572.273.662.4
SAGM54.349.872.173.662.5
ISAM55.852.272.273.463.4
Table 8. Ablation study of ISAM. Optimization objectives of different methods and their DG accuracy (%).
Table 8. Ablation study of ISAM. Optimization objectives of different methods and their DG accuracy (%).
MethodOptimization ObjectivePACSVLCSOffice-Home
ERM L θ ; D 80.775.859.9
SAM L p θ ; D 81.377.062.4
SAGM L θ ; D + L p θ − λ ∇ θ L θ ; D ; D 81.677.162.5
ISAM L p θ ; D + λ 1 M ∑ i = 1 M K L p θ + ε ^ x i ‖ p θ x i 82.277.663.4
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Dong, M.; Yang, Y.; Zeng, K.; Wang, Q.; Shen, T. Implicit Sharpness-Aware Minimization for Domain Generalization. Remote Sens. 2024, 16, 2877. https://doi.org/10.3390/rs16162877

AMA Style

Dong M, Yang Y, Zeng K, Wang Q, Shen T. Implicit Sharpness-Aware Minimization for Domain Generalization. Remote Sensing. 2024; 16(16):2877. https://doi.org/10.3390/rs16162877

Chicago/Turabian Style

Dong, Mingrong, Yixuan Yang, Kai Zeng, Qingwang Wang, and Tao Shen. 2024. "Implicit Sharpness-Aware Minimization for Domain Generalization" Remote Sensing 16, no. 16: 2877. https://doi.org/10.3390/rs16162877

APA Style

Dong, M., Yang, Y., Zeng, K., Wang, Q., & Shen, T. (2024). Implicit Sharpness-Aware Minimization for Domain Generalization. Remote Sensing, 16(16), 2877. https://doi.org/10.3390/rs16162877

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop