Proceedings of the 15th Annual Conference of the Japan Association for Research on Testing, 2017,152|155.
Three Aspects of
Reliability in Item Response Theory
Visual
representations of reliability
Yasuharu
Okamoto
Faculty of
Integrated Arts and Social Sciences, Japan Womenfs University
1. Introduction
Reliability has mainly
been discussed in classical test theory (CTT), and coefficients have been
defined according to three aspects, precision, stability, and predictive power.
Precision comes from the ratio of the variance of true scores to that of observed
scores and is considered the mathematical definition of reliability. Stability
is denoted by correlation coefficient between two parallel tests. Predictive
power is represented by strength of relation between observed values and true
values and is obtained by the regression modelfs coefficient of determination
for observed values and true values as
dependent and predictor variables, respectively. These three coefficients of
reliability coincide in the case of continuous items. However, in the case of
categorical items, they differ. Results of simulation (Okamoto, 2016) showed
that alpha coefficient tends to be larger than strength of relation, i.e.,
coefficient of determination, between true values and observed values in the
case of categorical items. These results indicate the necessity of considering
reliability in item response theory (IRT) from the three points of view.