Up

Relation of Number of Parameters and Error:

Contradicting trends depend on how to measure

Brunton & Kutz (2019) explain that error becomes larger as number of parameters increases beyond the true number of parameters. But, this increase is against the naïve intuition, because a model with larger number of parameters includes that with smaller number of parameters. To investigate their counter-intuitive explanation, I wrote a Python script shown in Listing 1, which is based on Code 4.14 in Brunton & Kutz (2019). In the script, the true model is given by a quadratic function:

Data are given with a Guassian noise

Models are

In explaining the counter-intuitive trend, Brunton & Kutz (2019) defined the error as difference between the true model and the estimated model.

But, is unobservable, so cannot be calculated. Empirically, we can calculate error by the following

By running the script in Listing 1, we can get relations between number of parameters of and the errors En and Err.

Run the script in Listing 1, then the graphs in Figure 1 will be shown. (The script was run in Python 3.11.8/Anaconda)

Figure 1 Relations between number of parameters and errors.

Errors En between the true values and the estimated models are shown by red triangles, which show the increasing trend explained by Brunton & Kutz (2019).

Green inverted triangles show observable errors and exhibit decreasing trend.

Criteria used in model selection are explained by Brunton & Kutz (2019). To study on the criteria, simple scripts were prepared at this website.

Listing 1 Script to investigate relation between number of parameters and error.

import numpy as np

import matplotlib.pyplot as plt

M = 20

n = 1000

L = 4

n_pnts = 100

x = np.linspace(0, L, n_pnts)

En = np.empty((n, M))

Err = np.empty((n, M))

for jj in range(M):

A = np.empty((n_pnts, jj+1))

for j in range(jj+1):

A[:,j] = x ** j

f = x**2 # True value

for j in range(n):

# True value + Noise

fn = f + 0.1 * np.random.normal(size=n_pnts)

An = np.linalg.pinv(A) @ fn

fna = A @ An

# Error from the true values

En[j, jj] = np.linalg.norm(f - fna) / np.linalg.norm(f)

# Error fron the givendata

Err[j, jj] = np.linalg.norm(fn - fna) / np.linalg.norm(fn)

plt.figure(figsize=(12,5))

plt.subplot(1,2,1)

Mean_En = np.mean(En, axis=0)

plt.plot(range(M), Mean_En, '^', c='r', lw=0, label='w.r.t. True values')

Mean_Err = np.mean(Err, axis=0)

plt.plot(range(M), Mean_Err, 'v', c = 'g', lw=0, label='w.r.t. Data')

plt.boxplot(En, positions=np.arange(0,M))

plt.boxplot(Err, positions=np.arange(0,M))

plt.xticks(np.arange(0, M))

plt.xlabel('Polynomial Degree',fontsize=14)

plt.ylabel('Error', fontsize=14)

plt.title('Degree >= 0', fontsize=16)

plt.legend(fontsize=14)

#lt.tight_layout()

#plt.show()

plt.subplot(1,2,2)

plt.plot(np.arange(M)[2:], Mean_En[2:], '^-', c='r', lw=1, label='w.r.t. True values')

plt.plot(np.arange(M)[2:], Mean_Err[2:], 'v-', c = 'g', lw=1, label='w.r.t. Data')

plt.xticks(np.arange(2, M))

plt.xlabel('Polynomial Degree', fontsize=14)

plt.ylabel('Error', fontsize=14)

plt.boxplot(En[:,2:], positions=np.arange(2, M))

plt.boxplot(Err[:,2:], positions=np.arange(2, M))

plt.title('Degree >= 2', fontsize=16)

plt.legend()

plt.tight_layout()

plt.show()

Reference

S. L. Brunton and J. N. Kutz (2019) Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press.

Home