Htydjtk

I am trying to overfit to the maximum a random forest classifier using scikit-learn to make some tests.

Does somebody know what hyperparameters I can tune to do that? Or does somebody know which other model I could apply to achieve a overfitted to the maximum a non-linear model?

Decision Trees are definitely easier to overfit than Random Forests. The averaging effect (see bagging) is meant to combat overfitting.

Other than that I think the default parameters will overfit.

Example:

from sklearn.tree import DecisionTreeRegressor



# Create a dataset

x = np.linspace(0, 10 * np.pi, 50).reshape(-1,1)

y = x + 3 * np.sin(x)

noise = np.random.random(50).reshape(-1,1)

noise -= noise.mean() # center noise at 0

noisy = y + noise * 2



# Define a Decision Tree (with default parameters)

dtr = DecisionTreeRegressor()

dtr.fit(x, noisy)

y_dtr = dtr.predict(x)



# Draw the two plots

plt.figure(figsize=(14, 4))

ax1 = plt.subplot(121)

ax1.plot(np.linspace(0, 10 * np.pi, 100), 

         np.linspace(0, 10 * np.pi, 100) + 3 * np.sin(np.linspace(0, 10 * np.pi, 100)),

        color='gray', label='desired fit', zorder=-1, alpha=0.5)

ax1.plot(x, y_dtr, color='#ff7f0e', label='decision tree', zorder=-1)

ax1.scatter(x, noisy, label='data')

ax1.set_xlabel('x')

ax1.set_ylabel('y')

ax1.set_title('Model Overfit')

ax1.spines['right'].set_visible(False)

ax1.spines['top'].set_visible(False)

ax1.yaxis.set_ticks_position('left')

ax1.xaxis.set_ticks_position('bottom')

ax1.legend()



ax2 = plt.subplot(122)

ax2.plot(np.linspace(0, 10 * np.pi, 100), 

        np.linspace(0, 10 * np.pi, 100) + 3 * np.sin(np.linspace(0, 10 * np.pi, 100)),

        color='gray', label='desired fit', zorder=-1, alpha=0.5)

ax2.plot(x, y_dtr, color='#ff7f0e', label='decision tree', zorder=-1)

ax2.set_xlabel('x')

ax2.set_ylabel('y')

ax2.set_title('Same graph')

ax2.spines['right'].set_visible(False)

ax2.spines['top'].set_visible(False)

ax2.yaxis.set_ticks_position('left')

ax2.xaxis.set_ticks_position('bottom')



ax2.legend()

Running the code below will produce the following figure:

I was doing very similar exercise. I've generated the synthetic dataset:

y = 10 * x + noise

and fitted one Random Forest model with full trees and one with pruned:

# ranadom forest with full trees

rf = RandomForestRegressor(n_estimators=50)

# random forest with pruned trees

rf = RandomForestRegressor(n_estimators=50, min_samples_leaf=25)

I got following predictions on test data:

As you can see the Random Forest with full trees clearly overfit while Random Forest with pruned trees generalize much better. Here is a link for my full experiment.