Skip to main content

Advertisement

Springer Nature Link
Log in

Self-training transformer for source-free domain adaptation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In this paper, we study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. Previous works on SFDA mainly focus on aligning the cross-domain distributions. However, they ignore the generalization ability of the pretrained source model, which largely influences the initial target outputs that are vital to the target adaptation stage. To address this, we make the interesting observation that the model accuracy is highly correlated with whether attention is focused on the objects in an image. To this end, we propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA. First, we apply the Transformer blocks as the attention module and inject it into a convolutional network. By doing so, the model is encouraged to turn attention towards the object regions, which can effectively improve the model’s generalization ability on unseen target domains. Second, a novel self-supervised knowledge distillation approach is proposed to adapt the Transformer with target pseudo-labels, further encouraging the network to focus on the object regions. Extensive experiments conducted on three domain adaptation tasks, including closed-set, partial-set, and open-set adaption, demonstrate that TransDA can significantly improve the accuracy over the source model and can produce state-of-the-art results on all settings. The source code and pretrained models are publicly available at: https://github.com/ygjwd12345/TransDA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets generated and analysed during this study are available in the Github repository: https://github.com/ygjwd12345/TransDA

References

  1. Long M, Cao Y, Wang J, Jordan M (2015) Learning transferable features with deep adaptation networks. In: International conference on machine learning, PMLR, pp 97–105

  2. Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: roceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7167–7176

  3. Zhang Y, David P, Gong B (2017) Curriculum domain adaptation for semantic segmentation of urban scenes. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2020–2030

  4. Long M, Zhu H, Wang J, Jordan MI (2016) Unsupervised domain adaptation with residual transfer networks. In: Advances in neural information processing systems, vol 29

  5. Kang G, Jiang L, Yang Y, Hauptmann AG (2019) Contrastive adaptation network for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4893–4902

  6. Lee C-Y, Batra T, Baig MH, Ulbricht D (2019) Sliced wasserstein discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10285–10295

  7. Li M, Zhai Y-M, Luo Y-W, Ge P-F, Ren C-X (2020) Enhanced transport distance for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13936–13944

  8. Yang G, Xia H, Ding M, Ding Z (2020) Bi-directional generation for unsupervised domain adaptation. In: Proceedings of the Proceedings of the AAAI conference on artificial intelligence conference on artificial intelligence. vol 34, no 04, pp 6615–6622

  9. Li R, Jiao Q, Cao W, Wong H-S, Wu S (2020) Model adaptation: unsupervised domain adaptation without source data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9641–9650

  10. Sankaranarayanan S, Balaji Y, Castillo CD, Chellappa R (2018) Generate to adapt: aligning domains using generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8503–8512

  11. Saito K, Watanabe K, Ushiku Y, Harada T (2018) Maximum classifier discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3723–3732

  12. Kurmi VK, Kumar S, Namboodiri VP (2019) Attending to discriminative certainty for domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 491–500

  13. Liang J, Hu D, Feng J (2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In: International conference on machine learning, PMLR, pp 6028–6039

  14. Ahmed SM, Raychaudhuri DS, Paul S, Oymak S, Roy-Chowdhury AK (2021) Unsupervised multi-source domain adaptation without access to source data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10103–10112

  15. Qiao F, Zhao L, Peng X (2020) Learning to learn single domain generalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12556–12565

  16. Qiao F, Peng X (2021) Uncertainty-guided model generalization to unseen domains. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6790–6800

  17. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: International conference on learning representations

  18. Cao Z, Long M, Wang J, Jordan MI (2018) Partial transfer learning with selective adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2724–2732

  19. Panareda Busto P, Gall J (2017) Open set domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 754–763

  20. Hu C, He S, Wang Y (2021) A classification method to detect faults in a rotating machinery based on kernelled support tensor machine and multilinear principal component analysis. Appl Intell 51 (4):2609–2621

    Article  Google Scholar 

  21. Hu C, Wang Y, Gu J (2020) Cross-domain intelligent fault classification of bearings based on tensor-aligned invariant subspace learning and two-dimensional convolutional neural networks, vol 209

  22. Carlucci FM, Porzi L, Caputo B, Ricci E, Bulo SR (2020) Multidial: domain alignment layers for (multisource) unsupervised domain adaptation. IEEE Trans Pattern Anal Mach Intell 43(12):4441–4452

    Article  Google Scholar 

  23. Chen X, Wang S, Long M, Wang J (2019) Transferability vs. discriminability: batch spectral penalization for adversarial domain adaptation. In: International conference on machine learning, PMLR, pp 1081–1090

  24. Cui S, Wang S, Zhuo J, Li L, Huang Q, Tian Q (2020) Towards discriminability and diversity: batch nuclear-norm maximization under label insufficient situations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3941–3950

  25. Wang X, Jin Y, Long M, Wang J, Jordan MI (2019) Transferable normalization: towards improving transferability of deep neural networks. Advances in Neural Information Processing Systems, vol. 32

  26. Cui S, Wang S, Zhuo J, Su C, Huang Q, Tian Q (2020) Gradually vanishing bridge for adversarial domain adaptation. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 12455–12464

  27. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need, Advances in Neural Information Processing Systems, vol. 30

  28. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European Conference on Computer Vision, 213–229

  29. Zhu X, Su W, Lu L, Li B, Wang X, Dai J (2021) Deformable detr: deformable transformers for end-to-end object detection. In: International Conference on Learning Representations

  30. Zheng S, Lu J, Zhao H, Zhu X, Luo Z, Wang Y, Fu Y, Feng J, Xiang T, Torr PH et al (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6881–6890

  31. Zeng Y, Fu J, Chao H (2020) Learning joint spatial-temporal transformations for video inpainting. In: European Conference on Computer Vision, pp 528–543

  32. Neimark D, Bar O, Zohar M, Asselmann D (2021) Video transformer network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3163–3172

  33. Huang L, Tan J, Liu J, Yuan J (2020) Hand-transformer: non-autoregressive structured modeling for 3d hand pose estimation. In: European Conference on Computer Vision, pp 17–33

  34. He S, Luo H, Wang P, Wang F, Li H, Jiang W (2021) Transreid: transformer-based object re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 15013–15022

  35. Yang G, Tang H, Ding M, Sebe N, Ricci E (2021) Transformer-based attention networks for continuous pixel-wise prediction. In: Proceedings of the IEEE/CVF International Conference on Computer vision, pp 16269–16279

  36. Müller R, Kornblith S, Hinton G (2019) When does label smoothing help? in Advances in neural information processing systems, vol 32

  37. Hu W, Miyato T, Tokui S, Matsumoto E, Sugiyama M (2017) Learning discrete representations via information maximizing self-augmented training. In: International Conference on Machine Learning, pp 1558–1567

  38. Caron M, Touvron H, Misra I, Jégou H., Mairal J, Bojanowski P, Joulin A (2021) Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision

  39. Peng X, Usman B, Kaushik N, Hoffman J, Wang D, Saenko K (2017) Visda: the visual domain adaptation challenge, arXiv

  40. Hoffman J, Tzeng E, Park T, Zhu J. -Y., Isola P, Saenko K, Efros A, Darrell T (2018) Cycada: Cycle-consistent adversarial domain adaptation. In: International conference on machine learning, Pmlr, pp 1989–1998

  41. Cao Z, You K, Long M, Wang J, Yang Q (2019) Learning to transfer examples for partial domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2985–2994

  42. Liu H, Cao Z, Long M, Wang J, Yang Q (2019) Separate to adapt: open set domain adaptation via progressive separation

  43. Hou Y, Zheng L (2021) Visualizing adapted knowledge in domain transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13824–13833

  44. Yang G, Ding M, Zhang Y (2022) Bi-directional class-wise adversaries for unsupervised domain adaptation. Appl Intell 52(4):3623–3639

    Article  Google Scholar 

  45. Acuna D, Zhang G, Law MT, Fidler S (2021) F-domain adversarial learning: theory and algorithms. In: International Conference on Machine Learning, PMLR, pp 66–75

  46. Sharma A, Kalluri T, Chandraker M (2021) Instance level affinity-based transfer for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5361–5371

  47. Yang S, van de Weijer J, Herranz L, Jui S, et al. (2021) Exploiting the intrinsic neighborhood structure for source-free domain adaptation. Adv Neural Inf Process Syst 34:29393–29405

    Google Scholar 

  48. Hu L, Kan M, Shan S, Chen X (2020) Unsupervised domain adaptation with hierarchical gradient synchronization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4043–4052

  49. Huang J, Guan D, Xiao A, Lu S (2021) Model adaptation: historical contrastive learning for unsupervised domain adaptation without source data. Adv Neural Inf Process Syst 34:3635–3649

    Google Scholar 

  50. Chu T, Liu Y, Deng J, Li W, Duan L (2022) Denoised maximum classifier discrepancy for sourcefree unsupervised domain adaptation. In: Proceedings of the AAAI conference on artificial intelligence, vol 2

  51. Li S, Xie M, Lv F, Liu CH, Liang J, Qin C, Li W (2021) Semantic concentration for domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9102–9111

  52. Xia H, Zhao H, Ding Z (2021) Adaptive adversarial network for source-free domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9010–9019

  53. Gu X, Sun J, Xu Z (2020) Spherical space domain adaptation with robust pseudo-label loss. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9101–9110

  54. Li S, Xie M, Gong K, Liu CH, Wang Y, Li W (2021) Transferable semantic augmentation for domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11516–11525

  55. Luo Y. -W., Ren C. -X. (2021) Conditional bures metric for domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13989–13998

  56. Yue Z, Sun Q, Hua X-S, Zhang H (2021) Transporting causal mechanisms for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8599–8608

  57. Gao Z, Zhang S, Huang K, Wang Q, Zhong C (2021) Gradient distribution alignment certificates better adversarial domain adaptation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 8937–8946

  58. Na J, Jung H, Chang HJ, Hwang W (2021) Fixbi: bridging domain spaces for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1094–1103

  59. Long M, Cao Z, Wang J, Jordan MI (2018) Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems, 31

  60. Nguyen AT, Tran T, Gal Y, Torr PH, Baydin AG (2022) Kl guided domain adaptation. In: International conference on learning representations

  61. Zhang Y, Liu T, Long M, Jordan M (2019) Bridging theory and algorithm for domain adaptation. In: International Conference on Machine Learning, PMLR, pp 7404–7413

  62. Xu R, Li G, Yang J, Lin L (2019) Larger norm more transferable: an adaptive feature norm approach for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1426–1435

  63. Kundu JN, Venkat N, Revanur A, Babu RV et al (2020) Towards inheritable models for open-set domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12376–12385

  64. Li G, Kang G, Zhu Y, Wei Y, Yang Y (2021) Domain consensus clustering for universal domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9757–9766

  65. Gu X, Yu X, Sun J, Xu Z et al (2021) Advances in neural information processing systems. Adversarial Reweighting for Partial Domain Adaptation 34:14860–14872

    Google Scholar 

  66. Liang J, Wang Y, Hu D, He R, Feng J (2020) A balanced and uncertainty-aware approach for partial domain adaptation. In: European Conference on Computer Vision, pp 123–140

  67. Du Z, Li J, Su H, Zhu L, Lu K (2021) Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3937–3946

  68. Lu Z, Yang Y, Zhu X, Liu C, Song Y-Z, Xiang T (2020) Stochastic classifiers for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9111–9120

  69. Zhang J, Ding Z, Li W, Ogunbona P (2018) Importance weighted adversarial nets for partial domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8156–8164

  70. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7132–7141

  71. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7794–7803

Download references

Author information

Authors and Affiliations

  1. School of Instrument Science and Engineering, Harbin Institute of Technology (HIT), Harbin, China

    Guanglei Yang & Mingli Ding

  2. Department of Information Engineering and Computer Science, University of Trento, Trento, Italy

    Zhun Zhong, Nicu Sebe & Elisa Ricci

Authors
  1. Guanglei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhun Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingli Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicu Sebe
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Elisa Ricci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhun Zhong.

Ethics declarations

Conflict of Interests

We would like to note that in the manuscript entitled “Self-Training Transformer for Source-Free Domain Adaptation”, no conflict of interest exits in the submission of this manuscript, and manuscript is approved by all authors for publication.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, G., Zhong, Z., Ding, M. et al. Self-training transformer for source-free domain adaptation. Appl Intell 53, 16560–16574 (2023). https://doi.org/10.1007/s10489-022-04364-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04364-9

Keywords

Search

Navigation