Machine Learning Validation Set
$begingroup$
I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.
machine-learning data-science-model
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.
machine-learning data-science-model
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use.
$endgroup$
– P. Esmailian
12 hours ago
add a comment |
$begingroup$
I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.
machine-learning data-science-model
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.
machine-learning data-science-model
machine-learning data-science-model
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
asked 12 hours ago
Rishab BamraraRishab Bamrara
1
1
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Rishab Bamrara is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use.
$endgroup$
– P. Esmailian
12 hours ago
add a comment |
$begingroup$
Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use.
$endgroup$
– P. Esmailian
12 hours ago
$begingroup$
Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use.
$endgroup$
– P. Esmailian
12 hours ago
$begingroup$
Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use.
$endgroup$
– P. Esmailian
12 hours ago
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.
But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.
answered 12 hours ago
user10283726user10283726
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms. One will eventually perform well on the test set just due to the nature of that particular data. You need the final holdout set to get a less biased estimate.
Comparing models can be looked at the same way as tuning hyperparameters. Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.
answered 4 hours ago
astelastel
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Rishab Bamrara is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46586%2fmachine-learning-validation-set%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.
But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.
answered 12 hours ago
user10283726user10283726
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.
But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.
answered 12 hours ago
user10283726user10283726
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.
But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.
answered 12 hours ago
user10283726user10283726
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.
But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.
answered 12 hours ago
user10283726user10283726
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 12 hours ago
user10283726user10283726
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 12 hours ago
user10283726user10283726
11
answered 12 hours ago
user10283726user10283726
11
11
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
user10283726 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
$begingroup$
Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms. One will eventually perform well on the test set just due to the nature of that particular data. You need the final holdout set to get a less biased estimate.
Comparing models can be looked at the same way as tuning hyperparameters. Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.
answered 4 hours ago
astelastel
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms. One will eventually perform well on the test set just due to the nature of that particular data. You need the final holdout set to get a less biased estimate.
Comparing models can be looked at the same way as tuning hyperparameters. Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.
answered 4 hours ago
astelastel
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms. One will eventually perform well on the test set just due to the nature of that particular data. You need the final holdout set to get a less biased estimate.
Comparing models can be looked at the same way as tuning hyperparameters. Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.
answered 4 hours ago
astelastel
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms. One will eventually perform well on the test set just due to the nature of that particular data. You need the final holdout set to get a less biased estimate.
Comparing models can be looked at the same way as tuning hyperparameters. Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.
answered 4 hours ago
astelastel
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 4 hours ago
astelastel
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 4 hours ago
astelastel
1
answered 4 hours ago
astelastel
1
1
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
astel is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
Rishab Bamrara is a new contributor. Be nice, and check out our Code of Conduct.
Rishab Bamrara is a new contributor. Be nice, and check out our Code of Conduct.
Rishab Bamrara is a new contributor. Be nice, and check out our Code of Conduct.
Rishab Bamrara is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46586%2fmachine-learning-validation-set%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use.
$endgroup$
– P. Esmailian
12 hours ago