Up

Posterior Probability of “Effective” Depends on the Power of the Test

High power test gives you more information than low power test

Yasuharu Okamoto, 2017/6

When we find that the result of the statistical test is significant, the prior probability of “Effective” changes to the posterior probability. The way how the prior probability changes to the posterior one depends on the power of the test, which can be described by the Bayesian approach.

Table 1 shows prior probabilities of “Effective” and “Not effective”, and probabilities of “Significant” and “Not significant” under the condition “Effective”, or under the condition “Not effective” (Null hypothesis).

Table 1 Probabilities of “Effective”, “Significant”, and “Not significant”.
	Significant (+)	Not significant (-)
Effective:	Power:	Type II error:
Not effective:	Type I error:	Accept Null Hypoth.:

The posterior probability can be calculated as conditional probability as follows:

Notice that when

holds, we have

In this case, the posterior probability does not change after we find that the result of the test is significant. Condition (1) holds when the probability distributions for the null hypothesis and alternative one is the same.

Figure 1 shows graphs of as a function of for various powers s of tests. The blue curve represents the graph for power 0.9, the dotted red curve represents the graph for power 0.06, and so on.

Figure 1

When the power of the test is high, the posterior probability increases more than for the test of low power from the prior probability .

Test of high power gives you more information also in the case that the result of the test is not significant.

The posterior probability conditional that the results of the test is not significant is this.

When the condition

holds, as in the case of significant result, we have

That is, not significant result of the test does not change the probability of “”.

Curves in Figure 2 show relations of posterior probability and prior probability for various powers of tests.

図２

The blue curve represents the relation for power 0.9, the dotted red curve represents the relation for power 0.06, and so on. Amount of the change from the prior probability to the posterior probability depends on the power of the test. When the power is high, i.e. 0.9, the difference between the prior and posterior probabilities is large. For the test of low power, i.e., 0.06, the difference between the prior and posterior probabilities is negligible.

When the power is high, we can get more information from the result of the test, i.e. the change in probability of hypothesis “” is large, but test of low power would give you little information as to the probability of hypothesis “”.

The script for Figure 1 is this.

import matplotlib.pyplot as plt

import numpy as np

def P_after( g, a, b ):

return g * (1 -b) / (g * (1 - b - a) + a)

x = np.arange(0, 1.001, 0.01)

b = [ 0.1, 0.2, 0.5, 0.8, 0.94 ]

y = []

for i in range(5):

y.append([])

for v in x:

for i in range(5):

t = P_after( v, 0.05, b[i] )

y[i].append(t)

plt.plot(x, y[0], 'b-')

plt.plot(x, y[1], 'b--')

plt.plot(x, y[2], 'g-')

plt.plot(x, y[3], 'g--')

plt.plot(x, y[4], 'r:')

plt.plot([0, 1], [0, 1], 'k--')

plt.legend(['0.9', '0.8', '0.5', '0.2', '0.06', 'y = x'], loc = 4)

plt.title('Probablity of "Effective" conditional on "Significant"')

plt.xlabel('P(Effective)')

plt.ylabel('P(Effective|Sig.)')

plt.show()

The script for Figure 2 is this.

import matplotlib.pyplot as plt

import numpy as np

def P_after_n( g, a, b ):

return g * b / (g * (b + a - 1) + (1 - a))

x = np.arange(0, 1.001, 0.01)

b = [ 0.1, 0.2, 0.5, 0.8, 0.94 ]

y = []

for i in range(5):

y.append([])

for v in x:

for i in range(5):

t = P_after_n( v, 0.05, b[i] )

y[i].append(t)

plt.plot(x, y[0], 'b-')

plt.plot(x, y[1], 'b--')

plt.plot(x, y[2], 'g-')

plt.plot(x, y[3], 'g--')

plt.plot(x, y[4], 'r:')

plt.plot([0, 1], [0, 1], 'k--')

plt.legend(['0.9', '0.8', '0.5', '0.2', '0.06', 'y = x'], loc = 2)

plt.title('Probablity of "Effective" conditional on "Not Significant"')

plt.xlabel('P(Effective)')

plt.ylabel('P(Effective|Not Sig.)')

plt.show()

The files of these scripts are archived in ZIP file SigStatPrg.zip.

Up