An Annotation class is a FancyArrowPatch and a Text. This sounds like it could be a compound Artist like that, or even just an Annotation object, perhaps with more Text placement knobs, or a child class. I don't think FancyArrowPatch on its own should grow a text method, and it almost certainly should not be called set_label since for every other Artist set_label sets the label to be used for legends.

or even just an Annotation object, perhaps with more Text placement knobs,

Looking at annotate again, this kinda already works out of the box with xycoords=Artist

fig, ax = plt.subplots(figsize=(3,3))

arr = mpatches.FancyArrowPatch((.25,.25), (.75,.75),   arrowstyle='->,head_width=.15', mutation_scale=20, )
ax.add_patch(arr)
ax.annotate("label", (.5, .5), xycoords=arr, ha='center', va='center')
ax.set(xlim=(0,1), ylim=(0,1))

so maybe my actual issue is that annotate has way way too many knobs and artists should maybe get very thin annotate method that sets xycoords to itself.

def annotate(self, label, xy, **kwargs):
     self.axes.annotate(label, xy, xycoords=self, **kwargs)

which I know seems pointless but like I struggle w/ how to communicate and make discoverable all the things annotation can do.

The core problem here is that we need all of those knobs. Reducing the number or changing the parameterization either throws out useful functionality or just masks the problem. I think we should take inspiration from https://docs.python.org/3/library/itertools.html#itertools-recipes and write up as functions a whole lot of these alternate paramaterization.

Either they happen to be what users want (and then they can copy-paste and be done), the are close enough that users can tweak them a little bit, or they are extensive enough to provide inspiration for what the users actually need to write for them selves.

The core problem here is that we need all of those knob

Yeah I'm not suggesting we take any away from annotate. Where I think I landed here is explicitly attach an annotation to an artist via Artist.annotate, which is one parameterization. It would also address the bookkeeping problem which is why I really opened the PR in the first place.

That being said, I think a good-first-issue way to address my request is to add the following section (or something like it) to Annotations: Basic Annotation

# Annotating an Artist
# ====================
#
# Annotations can be positioned relative to an artist by setting the *xycoords* to the artist. 

 
fig, ax = plt.subplots(figsize=(3,3))

arr = mpatches.FancyArrowPatch((.25,.25), (.75,.75),   arrowstyle='->,head_width=.15', mutation_scale=20, )
ax.add_patch(arr)
ax.annotate("label", (.5, .5), xycoords=arr, ha='center', va='center')
ax.set(xlim=(0,1), ylim=(0,1))

Given that you just wrote the example can you please directly open the PR 😄 .

See also #22223 (comment): I think the idea of having an easy way to specify coordinates in units relative to another artist's bounding box is something worth exploring.

AFAIK, we don't have Artists that have functions to create and add other Artists to a figure/ axes. I'm hesitant to change this as this adds complexity to the Artist (for example the function would have to determine whether the parent artist is in an Axes or figure and behave accordingly) Also, artist.annotate() conveys the impression to me that the annotation is attached to the artist, which e.g. would imply that the annotation would move with the artist. However, I don't think we can do this (or implement this without strong user demand).
IMHO the current API is good enough. It should only be documented better, which is done in #25100.

which e.g. would imply that the annotation would move with the artist.

I think draggable annotations would be really really cool, and is kinda what's happening in AnnotatedCursor I think. And more than that for me is the bookkeeping aspect, but maybe @jklymak's compound artist suggestion "AnnotatedArtist" suggestion would work for that? Basically, something like

annotated_arrow = AnnotatedArtist(FancyArrowPatch)
annotated_arrow.add_annotation(...)
annotated_arrow.add_annotation(...)

so that the complexity isn't in the artist? Granted, this is maybe more useful for something that's getting annotated a lot or something interactive/animated than my actual use case of annotating almost everything in a figure I'm making.

I think draggable annotations would be really really cool,

Can draggable annotations not already be supported by draggable text? If you can just drag the annotation wherever you like, why would it need to be attached to a parent artist?

than my actual use case of annotating almost everything in a figure I'm making.

A lot of your use cases seem to be diagraming. I'm not against diagramming in Matplotlib per-se, but it's a bit orthogonal to the main point of Matplotlib - there are lots of great diagramming GUI-based programs in the world.

Can draggable annotations not already be supported by draggable text?

The point here was rather that you modify the artist and the annotation follows along, e.g. you have

and either drag the artist (here: arrow) or modify it in an animation, then if the annotation belongs to the artist, it should follow where the artist is going.

That seems OK - Do we have an artist transform object - having that evaluated at draw time doesn't seem too hard.

What currently happens if you set xycoords=my_artist within annotate and then move the artist? Intuitively, I would expect the text moves with the artist.

@jklymak I think that's close to my IndirectTransform proposal at #22223 (comment)?

What currently happens if you set xycoords=my_artist within annotate and then move the artist?

So ok, this is totally doable 😄 I'm not a fan of having to do the bookkeeping by hand, but I can live w/ that given the lack of enthusiasm. And @anntzer's IndirectTransform proposal would probably solve most of my other use cases, so I'll close in favor of that.