@andrzejnovak implemented this in mplhep and it is really useful to be able to see what failed!

I like the idea, but instead of writing it in bash hidden in the workflow, I would modify tools/visualize_tests.py. You could add an --only-failed option to only include the failed images, and a --output option that would copy the relevant files into a subdirectory that could be used as the artifact.

I see how that would work, but that script seems to be really centred about sth else. It would actually be very convenient if GH allowed uploading the HTML file directly as artifact, but since it seems to always produce zips, having the plots in a web page would just be extra steps.

I think I find it a bit confusing since pytest-mpl has a very convenient --mpl-results-path, but I wasn't able to pass it here.

@QuLogic did you want those changes definitively, or was that just an idea. It seems gathering the failed images is helpful to me.

We already have tools to determine which files are relevant in tools/triage_tests.py and tools/visualize_tests.py. I'd rather not have a third copy of this subsetting that may fall out of sync if things change, because no-one's going to notice that the CI version is wrong until a test result is broken, which may be long after said change. Even the scripts in tools break periodically because no-one runs them as often as tests.

Filtering was necessary because the full set is ~50MB which is not practical for downloading and inspecting anyway.

So I disagree that filtering is necessary, because once downloaded, we can run the above tools on them to whittle them down to the important stuff. I'd rather we just upload the entire directory.

@QuLogic I think that's somewhat a different use case. I might want to run the tools you linked if I ran test locally and I suppose one could want to cross-check remotes by downloading the outputs from GHA and visualizing them as one would local tests.

On the other hand, publishing only the failed images lets one quickly (because it's only the affected files) download the failed result and inspect what's wrong by eye without requiring any extra setup.

I suppose you'll notice something broke next time there's a PR with a failed img comparison test (which seems to happen often enough :) ).

I guess I'm somewhat ambivalent about this. If I know a test fails, I usually need to debug it locally anyway, so getting the failed diff remotely is rarely useful. Occasionally there is something I can't repo locally, though....

@andrzejnovak Thanks a lot for this. If the tests fail we now upload all the artifacts to GitHub actions artifacts, so this PR has been superseded. Sorry we didn't catch that when it happened! Thanks...

@jklymak no worries, glad the functionality made it in one way or another :)