Optimizing decision threshold on model with oversampled/imbalanced data
$begingroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
$endgroup$
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
$endgroup$
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
$endgroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
edited Sep 24 '18 at 14:52
rayven1lk
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
666
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f38621%2foptimizing-decision-threshold-on-model-with-oversampled-imbalanced-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
add a comment |
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
add a comment |
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
1,2821413
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f38621%2foptimizing-decision-threshold-on-model-with-oversampled-imbalanced-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
