Htydjtk

I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with unbalanced classes (3.5% of BAD type of clients).

I'm trying to make various models to have enough of them to make ensembles, but for this purpose, let's focus on one of them, particularly XGBoost.

I was reading a lot on how to tackle overfitting, but I haven't found any good source on how to do it in practice, step-by-step. As for now, my two best models have:

1. 0.8 AUC on training data, around 0.799 AUC on holdout set and around 0.7355 and 0.7195 AUC on out-of-time batches.




2. 0.764 AUC on training, 0.7785 AUC on the holdout test set and 0,7285 AUC on both out-of-time batches.

I am worried about is that drop on out-of-time batches, since I think that 0.05-0.08 drop is huge and it might be a sign that models that I did, really are overfitting and don't generalize well. To clarify, while I was tweaking models, I didn't know about those out-of-time scores.

Could anyone share the experience what is best practice to detect overfitting? And does those two models overfit, or I am just panicking, and this drop in performance is normal?

My current pipeline in general looks like this:

1. Data cleanup




2. Feature importance using xgboost package to take best 300 features from all 2400 available.




3. Removing highly-correlated features (0.75 cutoff) - 123 features left




4. Train/test split - 0.8 vs 0.2 plus two out-of-time batches




5. Model selection using nested CV(5-fold CV in outer) with hyperparameter tuning in inner loop(5-fold CV in inner) - all done in MLR package.




6. From 5 models I get from nested CV, I'm picking the best performing one (that has the closest AUC in both train and holdout test set)

And then when I was happy with the model I performed a test on out-of-time models.

How I could improve my pipeline, in a way that I could detect overfitting? Is there any list of steps that would roughly cover what it could be done to reduce it?

Also, in highly unbalanced case, choosing a good validation set means that I only need to take care of the proper distribution of the target variable? Should I take care of something else as well?

When choosing the validation set and the test set, it is important that it reflects the actual "production environment" of your problem. Since you have "out of time" validation sets, I assume you have some time structure in your data that you need to address when making predictions. If you are developing your model, not taking this time aspect into consideration, you are likely to get issues when this model is put into a realistic environment because the models are fit to an unrealistic scenario where time does not matter.

From the scores you show here, the train and test scores are very similar, while it drops off in the "out of time"-set. This might be an indication that you are not putting enough emphasis on the time dimension while developing your model.

Another point: There is also fully possible to overfit to your validation set, when as in your case, you have a lot of variables. Since some combination of these variables might randomly fit your train and validation set well, it might not be the case for your test set. This effect is also magnified by my earlier point, where the data generating process is not really stationary across time.

Here is a nice article about how to choose a good validation set:
https://www.fast.ai/2017/11/13/validation-sets/