Why increasing the number of units or layers does not increase the accuracy and decrease the loss?
$begingroup$
I have an LSTM neural network; when I increase the number of units, layers, epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:
loss: 3.5071 - acc: 0.0981 - val_loss: 6.7042 - val_acc: 0.0122
Why this happens and how can I fix it?
machine-learning neural-network deep-learning lstm loss-function
$endgroup$
add a comment |
$begingroup$
I have an LSTM neural network; when I increase the number of units, layers, epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:
loss: 3.5071 - acc: 0.0981 - val_loss: 6.7042 - val_acc: 0.0122
Why this happens and how can I fix it?
machine-learning neural-network deep-learning lstm loss-function
$endgroup$
$begingroup$
Those accuracies are so low that I wonder whether something is seriously wrong. Can you provide some more context?
$endgroup$
– Ben Reiniger
yesterday
add a comment |
$begingroup$
I have an LSTM neural network; when I increase the number of units, layers, epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:
loss: 3.5071 - acc: 0.0981 - val_loss: 6.7042 - val_acc: 0.0122
Why this happens and how can I fix it?
machine-learning neural-network deep-learning lstm loss-function
$endgroup$
I have an LSTM neural network; when I increase the number of units, layers, epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:
loss: 3.5071 - acc: 0.0981 - val_loss: 6.7042 - val_acc: 0.0122
Why this happens and how can I fix it?
machine-learning neural-network deep-learning lstm loss-function
machine-learning neural-network deep-learning lstm loss-function
edited 2 days ago
Media
7,35062161
7,35062161
asked 2 days ago
user145959user145959
1268
1268
$begingroup$
Those accuracies are so low that I wonder whether something is seriously wrong. Can you provide some more context?
$endgroup$
– Ben Reiniger
yesterday
add a comment |
$begingroup$
Those accuracies are so low that I wonder whether something is seriously wrong. Can you provide some more context?
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
Those accuracies are so low that I wonder whether something is seriously wrong. Can you provide some more context?
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
Those accuracies are so low that I wonder whether something is seriously wrong. Can you provide some more context?
$endgroup$
– Ben Reiniger
yesterday
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.
The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.
To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.
One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.
Another problem can be the weakness of LSTM
networks which cannot memorise numerous things. LSTM
models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.
Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.
You can also test Stacked LSTMs
which are powerful models.
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47362%2fwhy-increasing-the-number-of-units-or-layers-does-not-increase-the-accuracy-and%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.
The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.
To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.
One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.
Another problem can be the weakness of LSTM
networks which cannot memorise numerous things. LSTM
models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.
Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.
You can also test Stacked LSTMs
which are powerful models.
$endgroup$
add a comment |
$begingroup$
The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.
The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.
To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.
One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.
Another problem can be the weakness of LSTM
networks which cannot memorise numerous things. LSTM
models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.
Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.
You can also test Stacked LSTMs
which are powerful models.
$endgroup$
add a comment |
$begingroup$
The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.
The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.
To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.
One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.
Another problem can be the weakness of LSTM
networks which cannot memorise numerous things. LSTM
models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.
Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.
You can also test Stacked LSTMs
which are powerful models.
$endgroup$
The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.
The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.
To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.
One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.
Another problem can be the weakness of LSTM
networks which cannot memorise numerous things. LSTM
models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.
Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.
You can also test Stacked LSTMs
which are powerful models.
edited 2 days ago
answered 2 days ago
MediaMedia
7,35062161
7,35062161
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47362%2fwhy-increasing-the-number-of-units-or-layers-does-not-increase-the-accuracy-and%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Those accuracies are so low that I wonder whether something is seriously wrong. Can you provide some more context?
$endgroup$
– Ben Reiniger
yesterday