I think it would be interesting to see the D2 values for the Brier score for the exact same 3 toy data cases used in the D2 code snippets for the log loss above.

I just saw that Christian said the opposite above. For educational reason, I find it interesting to see the three cases (good, bad and very bad models) to see that the value can be negative.

It's very often the case that the readers just read the code snippets in the doc and ignore the text. If they see a negative value, I suspect that will pick their interest and nudge them into reading why this can be negative.

I think it could be helpful to expand the warning message to give the use a code snippet to either turn this warning into an exception or to silence it:

#29048 (comment)

We could also expose an replace_undefined_by argument, but this can be done in a follow-up PR for all D2 scores.

I think this is the same warning we are raising for all D^2 metrics and even the R^2 score. We could change it here but wouldn't it be better to just leave this as it is for consistency. Also if we want to update it how about adding an additional statement instead of a code snippet mentioning that y_proba should have at least more than 2 samples for the D^2 score to provide anything meaningful.

Maybe we could parametrize this test "use_sample_weight", [True, False] and consistently pass the sample_weight argument everywhere, either with None or with [2, 2, 3, 1, 1, 1].

This will be a bit difficult because the null model scores or the y_proba (equaling the null model) that are hardcoded, are different with sample weight and without sample weight. If we have two parameter values in the test overall we would need to add a bunch of if and else statements which would get awkward. Should we separate this out into binary and multiclass cases?

I think this test function is a bit long. We could split the edge cases where the expected assert d2_score == 0 in a dedicated function.