Non-Identifiable Parameters and Bayesian Analysis
Yasuharu Okamoto
A model where the number of parameters is larger than the degree of the data is not identifiable, and point estimation of some parameters cannot be given. In this case, Bayesian analysis can be applied to the model. An example is presented for a model with a multinomial distribution.
A multinomial distribution is given by the following equation:
Consider the case of 
. We can set the probability distribution as follows
where 
are weight parameters.
The number of parameters 
and 
is 2, but the degree of
freedom of the data is 1 as shown by this:
Set the frequencies of the two categories,
category-1 and category-2, 
and 
. These values satisfy the following equation
where 
is the total number of
data. That is, the two data and is constrained by one
equation, so the degree of freedom of the data is
The number of parameters is 2, so is larger than the degree of freedom of the data, which is 1.
In this case, point estimates of the
parameters and cannot be given by the
method of maximum likelihood, or least squared criterion.
However, Bayesian analysis gives estimation of non-identifiable parameters as posterior distributions, as shown by the following example.
Stan script for the above model is as this:
data {
int
f[2];
}
parameters {
real
w1;
real
w2;
}
transformed parameters {
vector[2]
p;
p[1]
= w1 / (w1 + w2);
p[2]
= w2 / (w1 + w2);
}
model {
w1 ~
uniform(0.0, 1.0);
w2 ~
uniform(0.0, 1.0);
f
~ multinomial(p);
}
The following Python script uses the above Stan script saved in the file named BinCat.stan.
import numpy as np
import matplotlib.pyplot as plt
import pystan
import seaborn as sb
Data = {'f': [30, 70]}
sm = pystan.StanModel(file =
'BinCat.stan')
fit = sm.sampling(data = Data, n_jobs
= 1)
print(fit)
fit.plot()
plt.show()
w1 = fit['w1']
w2 = fit['w2']
plt.plot(w1, w2, 'b.', alpha = 0.3)
plt.plot([0.0, 3/7], [0.0, 1.0], 'g-',
linewidth = 3,
label = 'w1:w2 = 3:7')
plt.xlabel('w1', fontsize = 20)
plt.ylabel('w2', fontsize = 20)
plt.legend(loc = 4, fontsize = 20)
plt.show()
w1w2 = np.array(w1) / (np.array(w1) +
np.array(w2))
plt.xlabel('w1', fontsize = 15)
plt.ylabel('w1/(w1+w2)', fontsize =
15)
plt.plot(w1, w1w2, 'b.', alpha = 0.3)
plt.show()
sb.kdeplot(w1, w1w2)
plt.xlabel('w1', fontsize = 15)
plt.ylabel('w1/(w1+w2)', fontsize =
15)
plt.show()
In the above Python script, the frequencies
of categories are 
and 
.
Run the script, we get trace graphs in Figure 1.
Figure 1
Posterior distributions of , , 
and 
are obtained.
Figure 2 shows a joint distribution of and , which are non-identifiable parameters.
Figure 2
Sample points from the joint posterior
distribution of and gather along the line 
, which corresponds to probabilities 
.
A scattergram of 
is shown in Figure 3.
Figure 3
A kernel distribution estimate of the scattergram Figure 3 is shown in Figure 4.
Figure 4
We see that the ratio 
gathers near 0.3
irrespectively of .
In the above model, probabilities are given by ratios of weights, which induces non-identifiability of parameters. If we use a Dirichlet distribution, this problem of non-identifiability can be avoided.
The above model uses uniform distributions as priors so that the posterior distribution reflect the likelihood. When we have some information about the parameters, we set a prior according to the prior information. In the case of a softmax model, we set priors for components of the model.