How the embedding layer is trained in Keras Embedding layer
$begingroup$
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
keras word-embeddings
asked Jan 25 '18 at 9:33
william007william007
25918
$endgroup$
add a comment |
$begingroup$
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
keras word-embeddings
asked Jan 25 '18 at 9:33
william007william007
25918
$endgroup$
add a comment |
$begingroup$
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
keras word-embeddings
asked Jan 25 '18 at 9:33
william007william007
25918
$endgroup$
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
keras word-embeddings
keras word-embeddings
asked Jan 25 '18 at 9:33
william007william007
25918
asked Jan 25 '18 at 9:33
william007william007
25918
edited Jan 25 '18 at 9:42
william007
asked Jan 25 '18 at 9:33
william007william007
25918
asked Jan 25 '18 at 9:33
william007william007
25918
asked Jan 25 '18 at 9:33
william007william007
25918
25918
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
$begingroup$
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations -> this blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.
edited 12 mins ago
Community♦
1
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
$endgroup$
add a comment |
$begingroup$
As far as I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.
If you want to use pre-trained embeddings, you can do it that way
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
$endgroup$
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
add a comment |
$begingroup$
Embedding layer uses embedding matrix for mapping data and is never updated during training.
There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f27025%2fhow-the-embedding-layer-is-trained-in-keras-embedding-layer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations -> this blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.
edited 12 mins ago
Community♦
1
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
$endgroup$
add a comment |
$begingroup$
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations -> this blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.
edited 12 mins ago
Community♦
1
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
$endgroup$
add a comment |
$begingroup$
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations -> this blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.
edited 12 mins ago
Community♦
1
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
$endgroup$
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations -> this blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.
edited 12 mins ago
Community♦
1
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
edited 12 mins ago
Community♦
1
edited 12 mins ago
Community♦
1
edited 12 mins ago
Community♦
1
1
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
answered Sep 26 '18 at 20:02
Jan van der VegtJan van der Vegt
6,3751536
6,3751536
add a comment |
add a comment |
$begingroup$
As far as I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.
If you want to use pre-trained embeddings, you can do it that way
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
$endgroup$
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
add a comment |
$begingroup$
As far as I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.
If you want to use pre-trained embeddings, you can do it that way
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
$endgroup$
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
add a comment |
$begingroup$
As far as I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.
If you want to use pre-trained embeddings, you can do it that way
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
$endgroup$
As far as I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.
If you want to use pre-trained embeddings, you can do it that way
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
answered Jan 31 '18 at 9:34
Valentin CalommeValentin Calomme
1,210423
1,210423
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
add a comment |
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
An autoencoder is a model which predicts itself. Is that what you meant?
$endgroup$
– kbrose
Apr 2 '18 at 14:14
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
Indeed, so you can view it as a neural net with a single hidden layer, and where the input and output are identical I am not certain it is what Keras uses, but given the speed and the documentation, I'd tend to think that way
$endgroup$
– Valentin Calomme
Apr 2 '18 at 21:23
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
The embedding layer itself does not do that. It only assigns a vector to each unique integer input. You could build an auto-encoding model around that to train the embeddings, but it is not part of the embedding layer.
$endgroup$
– kbrose
Apr 3 '18 at 15:06
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
$begingroup$
I was confused because your answer matches my understanding of what the Embedding layer does apart from calling it an "autoencoder".
$endgroup$
– kbrose
Apr 3 '18 at 15:08
add a comment |
$begingroup$
Embedding layer uses embedding matrix for mapping data and is never updated during training.
There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
$endgroup$
add a comment |
$begingroup$
Embedding layer uses embedding matrix for mapping data and is never updated during training.
There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
$endgroup$
add a comment |
$begingroup$
Embedding layer uses embedding matrix for mapping data and is never updated during training.
There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
$endgroup$
Embedding layer uses embedding matrix for mapping data and is never updated during training.
There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
answered Sep 26 '18 at 19:17
thanatozthanatoz
396214
396214
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f27025%2fhow-the-embedding-layer-is-trained-in-keras-embedding-layer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
