The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

So if I understand the PR correctly we're moving back to prepare during generate instead of during the fa forward? 👀

yep, to be explicit + a lil faster than computing in FA2 utils per layer

Morning @zucchini-nlp. I have a question here 🖐️

yep, to be explicit + a lil faster than computing in FA2 utils per layer

What about BART-based models? They use three types of attention: encoder self-attention, decoder self-attention, and decoder cross-attention, each with different fa2 kwargs. encoder self-attention can be skipped since it’s computed only once at the start, but my concern is with decoder self-attention and decoder cross-attention.

edit for more clearer: If we initialize fa kwargs during generation, can we set different kwargs for decoder self-attention and decoder cross-attention?

@ducviet00 You're right that in this form it won't work for Bart but it also doesn't support the attention_backend flag so it should not even enter the preparation path and it has the usual calls as before

It has to be changed in the future then tho, I agree!