Skip to main content
Springer Nature Link
Log in

Finding Visual Task Vectors

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15101))

Included in the following conference series:

  • 303 Accesses

Abstract

Visual Prompting is a technique for teaching models to perform a visual task via in-context examples, without any additional training. In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model [4], and find Task Vectors, activations that encode task-specific information. Equipped with this insight, we demonstrate that it is possible to identify the Task Vectors and use them to guide the network towards performing different tasks without having to provide any in-context input-output examples. To find Task Vectors, we compute the mean activations of the attention heads in the model per task and use the REINFORCE [43] algorithm to patch into a subset of them with a new query image. The resulting Task Vectors guide the model towards performing the task better than the original model. (For code and models see www.github.com/alhojel/visual_task_vectors).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 63.34
Price includes VAT (France)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 78.06
Price includes VAT (France)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We overload the definition of \(\mathbf {D_{task_j}}\) to avoid notation clutter.

References

  1. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., Zhou, D.: What learning algorithm is in-context learning? Investigations with linear models. arXiv preprint arXiv:2211.15661 (2022)

  2. Bahng, H., Jahanian, A., Sankaranarayanan, S., Isola, P.: Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 (2022)

  3. Bai, Y., et al.: Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.00785 (2023)

  4. Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., Efros, A.: Visual prompting via image inpainting. In: Advances in Neural Information Processing Systems, vol. 35, pp. 25005–25017 (2022)

    Google Scholar 

  5. Bau, D., et al.: Gan dissection: visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597 (2018)

  6. Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  7. Dai, D., Sun, Y., Dong, L., Hao, Y., Sui, Z., Wei, F.: Why can GPT learn in-context? Language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559 (2022)

  8. Davies, D., Bouldin, D.: A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-1, 224–227 (1979). https://doi.org/10.1109/TPAMI.1979.4766909

  9. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale (2021)

    Google Scholar 

  10. Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873–12883, June 2021

    Google Scholar 

  11. Ferry, Q.R., Ching, J., Kawai, T.: Emergence and function of abstract representations in self-supervised transformers. arXiv preprint arXiv:2312.05361 (2023)

  12. Gandelsman, Y., Efros, A.A., Steinhardt, J.: Interpreting CLIP’s image representation via text-based decomposition. arXiv preprint arXiv:2310.05916 (2023)

  13. Garg, S., Tsipras, D., Liang, P.S., Valiant, G.: What can transformers learn in-context? A case study of simple function classes. In: Advances in Neural Information Processing Systems, vol. 35, pp. 30583–30598 (2022)

    Google Scholar 

  14. Hahn, M., Goyal, N.: A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971 (2023)

  15. Han, C., Wang, Z., Zhao, H., Ji, H.: In-context learning of large language models explained as kernel regression. arXiv preprint arXiv:2305.12766 (2023)

  16. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. CoRR abs/2111.06377 (2021). https://arxiv.org/abs/2111.06377

  17. Hendel, R., Geva, M., Globerson, A.: In-context learning creates task vectors. arXiv preprint arXiv:2310.15916 (2023)

  18. Jia, M., et al.: Visual prompt tuning. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13693, pp. 709–727. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_41

  19. Jin, Z., et al.: Cutting off the head ends the conflict: a mechanism for interpreting and mitigating knowledge conflicts in language models. arXiv preprint arXiv:2402.18154 (2024)

  20. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2017)

    Google Scholar 

  21. Li, X.L., Liang, P.: Prefix-tuning: optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)

  22. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What makes good in-context examples for GPT-\(3 \)? arXiv preprint arXiv:2101.06804 (2021)

  23. Liu, S., Xing, L., Zou, J.: In-context vectors: making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668 (2023)

  24. Lu, S., Schuff, H., Gurevych, I.: How are prompts different in terms of sensitivity? arXiv preprint arXiv:2311.07230 (2023)

  25. Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fantastically ordered prompts and where to find them: overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786 (2021)

  26. Luo, H., Specia, L.: From understanding to utilization: a survey on explainability for large language models. arXiv preprint arXiv:2401.12874 (2024)

  27. van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(86), 2579–2605 (2008). http://jmlr.org/papers/v9/vandermaaten08a.html

  28. Meng, K., Bau, D., Andonian, A., Belinkov, Y.: Locating and editing factual associations in GPT. In: Advances in Neural Information Processing Systems, vol. 35, pp. 17359–17372 (2022)

    Google Scholar 

  29. Moraffah, R., Karami, M., Guo, R., Raglin, A., Liu, H.: Causal interpretability for machine learning-problems, methods and evaluation. ACM SIGKDD Explor. Newsl. 22(1), 18–33 (2020)

    Article  Google Scholar 

  30. Palit, V., Pandey, R., Arora, A., Liang, P.P.: Towards vision-language mechanistic interpretability: a causal tracing tool for blip. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2856–2861 (2023)

    Google Scholar 

  31. Park, K., Choe, Y.J., Veitch, V.: The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658 (2023)

  32. Pearl, J.: Direct and indirect effects. In: Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 373–392 (2022)

    Google Scholar 

  33. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)

    Google Scholar 

  34. Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987). https://doi.org/10.1016/0377-0427(87)90125-7. https://www.sciencedirect.com/science/article/pii/0377042787901257

  35. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge (2015)

    Google Scholar 

  36. Shaban, A., Bansal, S., Liu, Z., Essa, I., Boots, B.: One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410 (2017)

  37. Singh, C., Inala, J.P., Galley, M., Caruana, R., Gao, J.: Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.01761 (2024)

  38. Todd, E., Li, M.L., Sharma, A.S., Mueller, A., Wallace, B.C., Bau, D.: Function vectors in large language models. arXiv preprint arXiv:2310.15213 (2023)

  39. Touvron, H., et al.: LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

  40. Vaswani, A., et al.: Attention is all you need. CoRR abs/1706.03762 (2017). http://arxiv.org/abs/1706.03762

  41. Wang, B., Komatsuzaki, A.: GPT-J-6B: a 6 billion parameter autoregressive language model (2021)

    Google Scholar 

  42. Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837 (2022)

    Google Scholar 

  43. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)

    Article  Google Scholar 

  44. Wu, X., Varshney, L.R.: Transformer-based causal language models perform clustering. arXiv preprint arXiv:2402.12151 (2024)

  45. Xie, S.M., Raghunathan, A., Liang, P., Ma, T.: An explanation of in-context learning as implicit Bayesian inference. arXiv preprint arXiv:2111.02080 (2021)

  46. Xu, J., et al.: IMProv: inpainting-based multimodal prompting for computer vision tasks. arXiv preprint arXiv:2312.01771 (2023)

  47. Xu, S., Dong, W., Guo, Z., Wu, X., Xiong, D.: Exploring multilingual human value concepts in large language models: is value alignment consistent, transferable and controllable across languages? arXiv preprint arXiv:2402.18120 (2024)

  48. Zhang, F., Nanda, N.: Towards best practices of activation patching in language models: metrics and methods. arXiv preprint arXiv:2309.16042 (2023)

  49. Zhang, K., Lv, A., Chen, Y., Ha, H., Xu, T., Yan, R.: Batch-ICL: effective, efficient, and order-agnostic in-context learning. arXiv preprint arXiv:2401.06469 (2024)

  50. Zhang, Y., Tiňo, P., Leonardis, A., Tang, K.: A survey on neural network interpretability. IEEE Trans. Emerging Top. Comput. Intell. 5(5), 726–742 (2021)

    Article  Google Scholar 

  51. Zhang, Y., Zhou, K., Liu, Z.: What makes good examples for visual in-context learning? In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

Download references

Acknowledgments

This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080). Prof. Darrell’s group was supported in part by DoD including DARPA’s LwLL and/or SemaFor programs, as well as BAIR’s industrial alliance programs. This work was completed in partial fulfillment for the Ph.D degree of the last author.

Author information

Authors and Affiliations

  1. UC Berkeley, Berkeley, USA

    Alberto Hojel, Yutong Bai, Trevor Darrell & Amir Bar

  2. Tel Aviv University, Tel Aviv, Israel

    Amir Globerson & Amir Bar

  3. Google Research, Tel Aviv, Israel

    Amir Globerson

Authors
  1. Alberto Hojel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yutong Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Trevor Darrell
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amir Globerson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Amir Bar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amir Bar .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 13136 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hojel, A., Bai, Y., Darrell, T., Globerson, A., Bar, A. (2025). Finding Visual Task Vectors. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15101. Springer, Cham. https://doi.org/10.1007/978-3-031-72775-7_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72775-7_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72774-0

  • Online ISBN: 978-3-031-72775-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation