Htydjtk

While using keras's multi-input model, the model just doesn't train at all. The accuracy skyrockets to near 100% and the loss plummets, so I think there's something wrong with the data generation.

I'm using a multi-input keras model, with two images of the same object, just rotated. The plan is to run each image through it's own CNN, then concatenate the two flattened layers and classify the object.

I prepare the data using the method found (here)[https://github.com/keras-team/keras/issues/8130]. The images are in separate directories but with the same seeding, they get loaded correctly. The labels are also correct, I've checked by looking at the filenames and the directories that the ImageDataGenerator generates.

The model is simple enough, I don't think it's causing the problem

def multiInput_model():

    #create model - custom



    input_1 = Input(shape=(img_width,img_height,1))

    input_2 = Input(shape=(img_width,img_height,1))   



    output_1 = Conv2D(32,(5,5), activation='relu')(input_1)

    output_1 = BatchNormalization()(output_1)

    output_1 = MaxPooling2D(pool_size=(2,2))(output_1)

    output_1 = Dropout(0.4)(output_1)

    output_1 = Flatten()(output_1)



    output_2 = Conv2D(32,(5,5), activation='relu')(input_2)

    output_2 = BatchNormalization()(output_2)

    output_2 = MaxPooling2D(pool_size=(2,2))(output_2)

    output_2 = Dropout(0.4)(output_2)

    output_2 = Flatten()(output_2)



    inputs = [input_1,input_2]

    outputs = [output_1,output_2]

    combine = concatenate(outputs)



    output = Dense(32,activation='relu')(combine)

    output = Dense(num_classes,activation='softmax')(output)





    model = Model(inputs,[output])





    model.compile(loss='categorical_crossentropy',

                 optimizer='RMSprop',metrics=['accuracy'])



    return model

The image generators are as follows

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_width,img_height,subset):

    genX1 = generator.flow_from_directory(dir1,

                                      color_mode='grayscale',

                                      target_size= 

                                      (img_width,img_height),

                                      batch_size=batch_size,

                                      class_mode='categorical',

                                      shuffle=False,

                                      subset=subset,

                                      seed=1)

                                      #Same seed for consistency.



    genX2 = generator.flow_from_directory(dir2,

                                      color_mode='grayscale',

                                      target_size= 

                                      (img_width,img_height),

                                      batch_size=batch_size,

                                      class_mode='categorical',

                                      shuffle=False,

                                      subset=subset,

                                      seed=1)

    while True:

        X1i = genX1.next()

        X2i = genX2.next()

        yield [X1i[0],X2i[0]],X1i[1]    #Yields both images and their mutual label







train_generator = 

generate_generator_multiple(generator=train_datagen,

                                          dir1=train_data_dirA,

                                          dir2=train_data_dirB,

                                          batch_size=batch_size,

                                          img_width=img_width,

                                          img_height=img_height,

                                          subset='training')



validation_generator =  

generate_generator_multiple(generator=train_datagen,

                                               dir1=train_data_dirA,

                                               dir2=train_data_dirB,

                                               batch_size=batch_size,

                                               img_width=img_width,

                                               img_height=img_height,

                                               subset='validation')

The output is always like this

20/20 [==============================] - 4s 183ms/step - loss: 0.1342 - acc: 0.9500 - val_loss: 1.1921e-07 - val_acc: 1.0000

Epoch 2/20

20/20 [==============================] - 0s 22ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 8.0590 - val_acc: 0.5000

Epoch 3/20

20/20 [==============================] - 0s 22ms/step - loss: 1.1921e-07 - acc: 1.0000 - val_loss: 16.1181 - val_acc: 0.0000e+00

Epoch 4/20

20/20 [==============================] - 0s 22ms/step - loss: 8.0590 - acc: 0.5000 - val_loss: 16.1181 - val_acc: 0.0000e+00