We already have k-fold cross validation (although it's not well documented, it's the folds arguments to train). We could add another param to refit_final_model_on_all_data, and have that default to true, since if you are cross validating, doing one more training run is only incrementally more expensive. I'm open to a more concise name for that param.