Should we use a std::jthread instead?

I think these variables should be tied together, with crashHandlerThreadFunction(), either into a common struct or a namespace.

Note that, since C++20, a std::atomic has methods wait(T value), notify_one() and notify_all(). And if I read https://en.cppreference.com/w/cpp/utility/program/signal correctly, calling these from the signal handler should be OK, as long as the atomic is lock free (which it has to be anyway). I think it's covered by "f is a non-static member function invoked on an object obj, such that obj.is_lock_free() yields true". But please double-check if you agree.

That would simplify this code and the wait function in the crash handler, and make it more polite/efficient.

In addition, if we use std::jthread (see my other comment), we can use stop token handling to address the thread shutdown instead of the atomic. This might be nicer to use (also see my comment on shutdown) and separate the concerns a little.

I would vote for both commenting and asserting for everything we do here that we're allowed to do it in the signal handler (or that it's not allowed and we're doing it anyway). So here I'd add

static_assert(decltype(::crashHandlerState)::is_always_lock_free);

or so, and a comment that the store is allowed under this condition.

I think it actually preferable to do it in the crash handler thread as much as possible, to reduce or eliminate the amount of UB we have in the crash handler, at least the actual logging.

Same here, I think it would be preferable to do this in the crash handler thread as much as possible (would need some refactoring/splitting of logBacktrace()).

I don't think its workable to have a function that must be called before each call to exit or similar. As of now, you've addressed an exit call in arangod.cpp (there is even a second one there) - and how does this affect the client-tools? And you've addressed to exit()s in the VersionFeature, but it seems we have at least a dozen other places calling exit.

I haven't thought it through, so there may be problems, but some ideas how we might address this:

• If we move the crashHandlerThread variable in a struct, we might be able to stop the thread in that thread's destructor
• We could install handlers with std::atexit and std::at_quick_exit (I don't think std::set_terminate is needed?)
• Don't install the thread globally, but with a scoped variable that we put in runServer, probably where we call CrashHandler::installCrashHandler(). It might then make sense to move installCrashHandler() into the constructor of the object, and maybe even remove the handlers in the destructor - that would clarify exactly in which scope the CrashHandler, including its handlers and thread, is active.

Intuitively I like the third option best, and the second the least.

I will try to do this as much as possible. Something with static lifetime will be necessary, or else the signal handler (which can be executed any time from any thread) and the std::terminate handler (which is executed on assertion failure or other catastrophic events, again essentially any time in any thread) do not have access to it. So making it into a totally scoped variable will not be possible.
I will try to avoid the explicit shutdown by additionally creating a scoped CrashHandler object (which can do the normal initialization and shutdown). But an atexit handler will be needed anyway, since we have these places which answer to --version, which use exit to stop the process right away. In this case the scoped variable is no longer cleaned up properly. Let's see what I can come up with.

If this follows (only) specific state transitions, these should be noted in a comment.

Done.