Problem

Keras does not have any direct implementation of region of interest pooling. I am aware of how to perform maxpooling, but I don't know how to get bounding boxes from feature maps passed from convolutional layer.

Is there any way to directly implement a region proposal algorithm?

Example

Let's say there is an architecture like this:

So we have a multi-input neural network architecture that eventually leads to the ROI MaxPool layer. We have three inputs, screenshot, textmaps and candidates, let's take candidates out. Then we would have such code in Keras:

from keras.models import Model

from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate

from keras.activations import relu

from keras.initializers import RandomUniform, Constant, TruncatedNormal



#  Network 1, Layer 1

screenshot = Input(shape=(1280, 1280, 0),

                   dtype='float32',

                   name='screenshot')

# padded1 = ZeroPadding2D(padding=5, data_format=None)(screenshot)

conv1 = Conv2D(filters=96,

               kernel_size=11,

               strides=(4, 4),

               activation=relu,

               padding='same')(screenshot)

# conv1 = Conv2D(filters=96, kernel_size=11, strides=(4, 4), activation=relu, padding='same')(padded1)

pooling1 = MaxPooling2D(pool_size=(3, 3),

                        strides=(2, 2),

                        padding='same')(conv1)

normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn



# Network 1, Layer 2



# padded2 = ZeroPadding2D(padding=2, data_format=None)(normalized1)

conv2 = Conv2D(filters=256,

               kernel_size=5,

               activation=relu,

               padding='same')(normalized1)

# conv2 = Conv2D(filters=256, kernel_size=5, activation=relu, padding='same')(padded2)

normalized2 = BatchNormalization()(conv2)

# padded3 = ZeroPadding2D(padding=1, data_format=None)(normalized2)

conv3 = Conv2D(filters=384,

               kernel_size=3,

               activation=relu,

               padding='same',

               kernel_initializer=TruncatedNormal(stddev=0.01),

               bias_initializer=Constant(value=0.1))(normalized2)

# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',

#               kernel_initializer=RandomUniform(stddev=0.1),

#               bias_initializer=Constant(value=0.1))(padded3)



# Network 2, Layer 1



textmaps = Input(shape=(160, 160, 128),

                 dtype='float32',

                 name='textmaps')

txt_conv1 = Conv2D(filters=48,

                   kernel_size=1,

                   activation=relu,

                   padding='same',

                   kernel_initializer=TruncatedNormal(stddev=0.01),

                   bias_initializer=Constant(value=0.1))(textmaps)



# (Network 1 + Network 2), Layer 1



merged = concatenate([conv3, txt_conv1], axis=-1)

merged_padding = ZeroPadding2D(padding=2, data_format=None)(merged)

merged_conv = Conv2D(filters=96,

                     kernel_size=5,

                     activation=relu, padding='same',

                     kernel_initializer=TruncatedNormal(stddev=0.01),

                     bias_initializer=Constant(value=0.1))(merged_padding)

If you look at the end of the code (and architecture itself), we pass concatenated activations from two different Conv+ReLu layers and then pass it to ROI MaxPool layer.

Thank you!