Htydjtk

I'm working on a project in which I should classify System calls sequences, my dataset is represented as sequences of integers (from 1 to 340). To do the classification I have inspired from Text classification projects. I'm trying to use on of them but I found a problem in my dataset shape, the code is:

df = pd.read_csv("data.txt") 

#df_test = pd.read_csv("validation.txt")



#split arrays into train and test data (cross validation)

train_text, test_text, train_y, test_y = train_test_split(df,df,test_size = 

0.2)



#train_text, train_y = (df,df)

#test_text, test_y = (df_test, df_test)

MAX_NB_WORDS = 5700



texts_train = train_text.astype(str)

texts_test = test_text.astype(str)





tokenizer = Tokenizer(nb_words=MAX_NB_WORDS, char_level=False)

tokenizer.fit_on_texts(texts_train)

sequences = tokenizer.texts_to_sequences(texts_train)

sequences_test = tokenizer.texts_to_sequences(texts_test)



word_index = tokenizer.word_index

#print('Found %s unique tokens.' % len(word_index))

type(tokenizer.word_index), len(tokenizer.word_index)

index_to_word = dict((i, w) for w, i in tokenizer.word_index.items()) 

" ".join([index_to_word[i] for i in sequences[0]])



seq_lens = [len(s) for s in sequences]

#print("average length: %0.1f" % np.mean(seq_lens))

#print("max length: %d" % max(seq_lens))



MAX_SEQUENCE_LENGTH = 100

# pad sequences with 0s

x_train = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)  # former des 

sequence de meme taille 150, en ajoutant des 0

x_test = pad_sequences(sequences_test, maxlen=MAX_SEQUENCE_LENGTH)

#print('Shape of data train:', x_train.shape)  #it gives (1,100)

#print('Shape of data test tensor:', x_test.shape)

y_train = train_y

y_test = test_y

#if np.any(y_train):

  #y_train = to_categorical(y_train)

print('Shape of label tensor:', y_train.shape)

EMBEDDING_DIM = 50

N_CLASSES = 2

# input: a sequence of MAX_SEQUENCE_LENGTH integers

sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='float32')



embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM,

                        input_length=MAX_SEQUENCE_LENGTH,

                        trainable=True)

embedded_sequences = embedding_layer(sequence_input)



average = GlobalAveragePooling1D()(embedded_sequences)

predictions = Dense(N_CLASSES, activation='softmax')(average)



   model = Model(sequence_input, predictions)

model.compile(loss='categorical_crossentropy',

          optimizer='adam', metrics=['acc'])

model.fit(x_train, y_train, validation_split=0.1,

      nb_epoch=10, batch_size=100)

output_test = model.predict(x_test)

print("test auc:", roc_auc_score(y_test,output_test[:,1]))

I got the error: ValueError Error when checking target expected dense_1 to have shape(2,), but got array with shape(1,)

Any suggestion, cause I don't know how to proceed .
Thank you

The problem seems to arise in this line.

predictions = Dense(N_CLASSES, activation='softmax')(average) 

Here, N_CLASSES has the value of 2. Meaning you have 2 classes. But the y_train vector has shape ( 1, ).

• You need to convert the y_train vector to a one hot vector. Here's any example.

If y_train = [ [ 0 ] , [ 1 ] ] then its one hot vector would be [ [ 1 , 0 ] , [ 0 , 1 ] ]. The one hot vector has shape ( 2 , ) which is needed in the above code snippet.
So make this edit.

y_train = keras.utils.to_categorical( y_train , N_CLASSES )