Htydjtk

Can someone give me a tip on how I could incorporate MSE & loss plots? I have been following some machinelearningmastery posts to plot this but the application is classification and I am attempting regression. Also what is different in my script is I am defining the model thru calling a function, so I am curious if my script could be re-written without the function def wider_model() that defines the model.

This script below works except what is commented out on the bottom for the plt plots. In the machinelearningmastery post, someone does ask this question how to do for regression, and supposedly if you print print(history.history.keys()) two values are returned for dict_keys([‘mean_absolute_error’, ‘loss’])...

Any tips help, there isn't a lot of wisdom here... Thank you

import numpy

import pandas

from keras.models import Sequential

from keras.layers import Dense

from keras.wrappers.scikit_learn import KerasRegressor

from sklearn.model_selection import cross_val_score

from sklearn.model_selection import KFold

from sklearn.preprocessing import StandardScaler

from sklearn.pipeline import Pipeline

import matplotlib.pyplot as plt

import math







# load dataset

dataset = pandas.read_csv("prepdSPdata.csv", index_col='Date', parse_dates=True)



# shuffle dataset

dataset = dataset.sample(frac=1.0)





# split into input (X) and output (Y) variables

X = numpy.array(dataset.drop(['Demand'],1))

Y = numpy.array(dataset['Demand'])



print(dataset.shape)

print(dataset.dtypes)

print(dataset.columns)





def wider_model():

    # create model

    model = Sequential()

    model.add(Dense(20, input_dim=11, kernel_initializer='normal', activation='relu'))

    model.add(Dense(10, kernel_initializer='normal', activation='relu'))

    model.add(Dense(1, kernel_initializer='normal'))

    # Compile model

    model.compile(loss='mean_squared_error', optimizer='adam')

    return model





# fix random seed for reproducibility

seed = 7

numpy.random.seed(seed)

estimators = 

estimators.append(('standardize', StandardScaler()))

estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=1, batch_size=5, verbose=0)))

pipeline = Pipeline(estimators)

kfold = KFold(n_splits=10, random_state=seed)

results = cross_val_score(pipeline, X, Y, cv=kfold)

print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))

print("RMSE", math.sqrt(results.std()))







# list all data in history

#print(wider_model.wider_model.keys())



# summarize history for MSE

#plt.plot(history.history['acc'])

#plt.plot(history.history['val_acc'])

#plt.title('model MSE')

#plt.ylabel('MSE')

#plt.xlabel('epoch')

#plt.legend(['train', 'test'], loc='upper left')

#plt.show()



# summarize history for loss

#plt.plot(history.history['loss'])

#plt.plot(history.history['val_loss'])

#plt.title('model loss')

#plt.ylabel('loss')

#plt.xlabel('epoch')

#plt.legend(['train', 'test'], loc='upper left')

#plt.show()

cross_val_score does not return the history of the training. You can use fit instead:

history = model.fit( ...

See this example.

As you mentioned, the history object holds the results of the training for each epoch.

Here is the relevant bit:

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)

pyplot.plot(history.history['mean_squared_error'])

pyplot.plot(history.history['mean_absolute_error'])

pyplot.plot(history.history['mean_absolute_percentage_error'])

pyplot.plot(history.history['cosine_proximity'])

pyplot.show()

I was actually working on the same example that you referenced yesterday. I think it is hard to understand because it introduces many functions and concepts: estimators, StandardScaler, KerasRegressor, Pipeline, KFold and cross_val_score.

However, I did like the approach to creating and testing models, and Cross Validation would produce more robust models.

You're only training your model for 1 epoch so you're only giving it one data point to work from. If you want to plot a line of loss or accuracy you need to train for more epochs.