Confidence intervals and P-values
In an inference setting for a single parameter, the calculations for a confidence interval and the π-value are commonly based on the same underlying statistical theory. When that is the case, there is a link between them. It is useful to know how this relationship works.
A confidence interval contains a range of values for the parameter of interest that are plausible, given the data. This should resonate with testing; when we carry out a hypothesis test, we are asking whether the data are consistent with a particular parameter value: is it plausible?
Often the focus in testing is on a “no effect” null hypothesis, such as π = 0. But in any context, we can test the hypothesis that the parameter equals some other fixed value, not necessarily null or zero. The process is the same.
For inference on a single parameter using the same method for both the interval estimate and the hypothesis test, we can state the connection between hypothesis tests and confidence intervals as follows.
- A 95% confidence interval for an unknown parameter consists of all parameter values which, if tested, give π > 0.05.
More simply and informally, the 95% confidence interval consists of all parameter values with which the data are consistent. We operationalise “consistent” as meaning π > 0.05.
In this connection, the value “0.95” is connected to “0.05”: 0.05 = 1 – 0.95. Also, the test and interval need to be both two-sided (the usual case), or one-sided in the right way.
This app allows you to explore the relationship between confidence intervals and the π -value in the context of taking a random sample from a normally distributed population. You can:
- Vary the sample size
- Change the confidence level
- Change the value of the null hypothesis
Activities
Here are some activities to support your understanding of the relationship between the confidence interval and the π-value.
Activity 1
- With the confidence level set at 0.95 and the null hypothesis at zero, choose a sample size.
- Take a New random sample, and record the P-value and the location of the null hypothesis (inside or outside the confidence interval).
- Repeat this many, many times.
- What range of P-values did you observe when the null hypothesis was outside the 95% confidence interval?
- What range of P-values did you observe when the null hypothesis was inside the 95% confidence interval?
Activity 2
- With your preferred sample size and the null hypothesis at zero, choose a confidence level other than 0.95, say 0.70.
- Take a New random sample, and record the P-value and the location of the null hypothesis (inside or outside the confidence interval).
- Repeat this many times.
- What range of P-values did you observe when the null hypothesis was outside the confidence interval?
- What range of P-values did you observe when the null hypothesis was inside the confidence interval?
Activity 3
- Make the confidence level equal to 1. What is the confidence interval?
- Make the confidence level equal to 0. What is the confidence interval?
Activity 4
- With the confidence level set at 0.95 and the null hypothesis at zero, choose a sample size.
- Take a New random sample and record the bounds of the confidence interval.
- Record the value of the null hypothesis and the P-value.
- Now change the null hypothesis to a value other than zero and record the value of the null hypothesis and the P-value.
- Repeat this many times.
- Consider the null hypotheses that fall inside the 95% confidence interval. What is the P-value for these hypotheses?
- Consider the null hypotheses that fall outside the 95% confidence interval. What is the P-value for these hypotheses?
- What is the P-value when the null hypothesis is closest to a boundary of the confidence interval?
Of course, in practice the null hypothesis is not varied; these activities are for educational understanding.