Htydjtk

I am trying to understand how the weight matrix in an LSTM cell is used. An LSTM unit has several weight matrix: Wf, Wi, Wc, Wo like below:

( from http://colah.github.io/posts/2015-08-Understanding-LSTMs/ )

At the same time, I am playing with the Keras LSTM and studying its source code:
https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L1871

In the source code, there is only one kernel mentioned. I am wondering is it referring to Wc only? Then where are the other weight matrix Wf, Wi, Wo initialized and used? Thanks!

understand how the weight matrix in an LSTM cell is used

In LSTM you have a cell vector that keeps track of necessary information for the task at hand.

To backpropagate the errors from far away time steps, LSTM by design has simple linear operations (*/+) that update the cell vector, thus it’s very easy for the gradient to just flow.

The key idea is we manipulate the cell vector by linear interactions through gates, where we introduce some non-linearity between the current input and the previous hidden features to obtain new features that range between zero to one, indicating how much we manipulate (add/remove/update) across each element of the new feature vector.

1. 
   

   how much information we need to keep track of?

   
   
   

   w_f: guess from the interactions between the previous hidden features and the current input by concatenating both vectors and multiplying them with a weight matrix and applying the sigmoid non-linearity to get values between [0,1] (different for each element) to see how much should we forget from the old cell vector.

   
   
   

   modified_old_cell = old_cell * forgetting_some_dimensions (forget gate f_t).

   
   


2. 
   

   how much information we need to extract from the current input.

   
   
   

   we introduce two feature vectors, by concatenating both vectors (input,previous hiddens) and applying tanh, sigmoid to get proposed cell features, input gate features respectively.

   
   
   

   proposed_cell = non_linearity(previous_hiddens,input), hence we used tanh to get values between [-1,+1].

   
   
   

   modified_proposed_cell = proposed_cell * input_gate. hence we don't need to add all information from the current time step.

   
   
   

   current_cell = integration (modified_old_cell,modified_proposed_cell). hence the integration operation is simply the plus operation.

   
   


3. 
   

   how much information do we need to pass to the next time step, hence if the problem is sequence tagging, maybe some of the output features are only important for the current time step.

   
   
   

   the same mechanism as above:
   current_hidden_features = current_cell * outputting_some_dimensions (output gate o_t).

   
   

They use this variable to save all the weight matrices by concatenating them.

In the call function of the LSTMCell you can see, how they are unpacked:

self.kernel_i = self.kernel[:, :self.units]

self.kernel_f = self.kernel[:, self.units: self.units * 2]

self.kernel_c = self.kernel[:, self.units * 2: self.units * 3]

self.kernel_o = self.kernel[:, self.units * 3:]