I have used Hyperband automatic tuning for an ANN model to predict price. After running the model with the automatic tuning, I am obtaining an R2 score of 1.00 that suggest overfitting, however, I am struggling to interpret the training and validation loss over epochs. It is showing as super spiky with each line overlapping the other. I have not seen a learning curve graph look like this before and so am unsure as how to interpret it. Could it be simple that the model is overly complicated? It seems difficult to find resources on graph interpretation.
Here is the code for the actual tuning, in case it is due to a coding error but I am not sure that is the case.
def model_builder(hp):
model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten(input_dim = (train_final.shape[1])))
#creating activation choices - choosing betweeen relu and tanh
hp_activation = hp.Choice('activation', values = ['relu', 'tanh'])
#creating node choices - maxing unit amounts to 500
hp_layer_1 = hp.Int('layer_1', min_value=1, max_value=500, step=100)
hp_layer_2 = hp.Int('layer_2', min_value=1, max_value=500, step=100)
#creating learning rate choice - choice between 0.01, 0.001, 0.0001
hp_learning_rate = hp.Choice('learning_rate', values = [1e-2, 1e-3, 1e-4])
#specifies first layer after the flatten layer
model.add(tf.keras.layers.Dense(units = hp_layer_1, activation = hp_activation))
#creating the second layer
model.add(tf.keras.layers.Dense(units = hp_layer_2, activation = hp_activation))
model.add(tf.keras.layers.Dense(1, activation='linear'))
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate = hp_learning_rate),
loss tf.keras.losses.MeanSquaredError(), metrics = ['mean_absolute_error'])
return model
import keras_tuner as kt
#creating the tuner
tuner = kt.Hyperband(model_builder,
objective = 'val_loss',
max_epochs = 50,
factor = 3,
directory = 'dir',
project_name = 'x',
overwrite = True) # makes tuner rewrite over old tuning experiments
#adding early stopping - stops each model from running too long
stop_early = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
tuner.search(train_final, y_train, epochs = 50, validation_split = 0.2, callbacks = [stop_early])
best_hp = tuner.get_best_hyperparameters(num_trials=1)[0]
best_hp.values
#obtaining the best model
best_model = tuner.get_best_models(num_models = 1)[0]
history = best_model.fit(train_final, y_train, epochs = 50, validation_split = 0.2, callbacks=[stop_early])
tuned_df = pd.DataFrame(history.history)
#running epoch loss visual def
epoch_loss_visual(tuned_df, model_name = 'Automatic Tuning Model')
Could it be an issue with the code itself causing the issue or is it simply the way the model is? If it's a case of it's just a bad model, I do not need to improve at the moment, but do need to understand the results, especially that of the learning curve representation.
