fontsize can be used for figure, subfigure and axes as well? Should it be a follow-up PR to add docs for those or will you handle it here as well?

Also, I think it would be nice to a add a test for some fontsize variant, although I understand that nothing has changed.

Is scale(1) an expensive op? If not, one may consider setting the scale factor in the if-clauses.

fontsize can be used for figure, subfigure and axes as well?

I guess so, but I don't think e.g. axes fontsize (some number of fontsizes away from the lower left corner of the axes) is really worth documenting: for the lower left corner it is equivalent to annotate(xy=(0, 0), xycoords="axes fraction", xytext=some_offset, textcoords="offset font"), but the latter form also works for the other corners whereas there's no way to specify the other corners with axes fontsize. So I'd say documenting it is just an attractive nuisance. (We could rework the code to disable that use, but I'm not going to bother doing so here.)

Is scale(1) an expensive op? If not, one may consider setting the scale factor in the if-clauses.

If you're thinking tr = Affine2D().scale(... if unit == ... else ... if unit == ...) etc., I'd say (even though I usually like that kind of constructs) that this is worse 1) because of the star-unpacking in the "fraction" case and 2) because of the need to throw ValueError in the last case (I've always wanted python to have "raise expressions" so that one can write foo = bar if baz else raise Exception(...), but heh...)

I think it would be nice to a add a test for some fontsize variant

At least #25905 will exercise this in building the docs; I'll skip actual tests for now...

More something like:

        if unit == "points":
            scale = self.figure.dpi / 72  # dpi/72 dots per point
        elif unit == "pixels":
            scale = 1.
        ...

        return Affine2D().scale(scale).translate(*xy0)

But there's the unit == "fraction" case, which is a bit ugly to also handle here.

But there's the unit == "fraction" case, which is a bit ugly to also handle here.

Fair enough. Didn't see that.