Interactive app: Confidence intervals and P-values

Confidence intervals and P-values

In an inference setting for a single parameter, the calculations for a confidence interval and the 𝑃-value are commonly based on the same underlying statistical theory. When that is the case, there is a link between them. It is useful to know how this relationship works.

A confidence interval contains a range of values for the parameter of interest that are plausible, given the data. This should resonate with testing; when we carry out a hypothesis test, we are asking whether the data are consistent with a particular parameter value: is it plausible?

Often the focus in testing is one a “no effect” null hypothesis, such as πœ‡ = 0. But in any context, we can test the hypothesis that the parameter equals some other fixed value, not necessarily null or zero. The process is exactly the same.

For inference on a single parameter using the same method for both the interval estimate and the hypothesis test, we can state the connection between hypothesis tests and confidence intervals as follows.

  • A 95% confidence interval for an unknown parameter consists of all parameter values which, if tested, give 𝑃 > 0.05.

More simply and informally, the 95% confidence interval consists of all parameter values with which the data are consistent. We operationalise “consistent” as meaning 𝑃 > 0.05.

In this connection, the value “0.95” is connected to “0.05”: 0.05 = 1 – 0.95.  Also, the test and interval need to be both two-sided (the usual case), or one-sided in the right way.

This app allows you to explore the relationship between confidence intervals and the 𝑃 -value in the context of taking a random sample from a Normally distributed population.  You can:

  • Vary the sample size
  • Change the confidence level
  • Change the value of the null hypothesis

Activities

Here are some activities to support your understanding of the relationship between the confidence interval and the 𝑃-value.

Activity 1

  • With the confidence level set at 0.95 and the null hypothesis at zero, choose a sample size.
  • Take a New random sample, and record the P-value and the location of the null hypothesis (inside or outside the confidence interval).
    • Repeat this many, many times.
  • What range of P-values did you observe, when the null hypothesis was outside the 95% confidence interval?
  • What range of P-values did you observe, when the null hypothesis was inside the 95% confidence interval?

Activity 2

  • With your preferred sample size and the null hypothesis at zero, choose a confidence level other than 0.95, say 0.70.
  • Take a New random sample, and record the P-value and the location of the null hypothesis (inside or outside the confidence interval).
    • Repeat this many times.
  • What range of P-values did you observe, when the null hypothesis was outside the confidence interval?
  • What range of P-values did you observe, when the null hypothesis was inside the confidence interval?

Activity 3

  • Make the confidence level equal to 1.  What is the confidence interval?
  • Make the confidence level equal to 0.  What is the confidence interval?

Activity 4

  • With the confidence level set at 0.95 and the null hypothesis at zero, choose a sample size.
  • Take a New random sample and record the bounds of the confidence interval.
    • Record the value of the null hypothesis and the P-value.
    • Now change the null hypothesis to a value other than zero and record the value of the null hypothesis and the P-value.
      • Repeat this many times.
  • Consider the null hypotheses that fall inside the 95% confidence interval.  What is the P-value for these hypotheses?
  • Consider the null hypotheses that fall outside the 95% confidence interval.  What is the P-value for these hypotheses?
  • What is the P-value when the null hypothesis is closest to a boundary of the confidence interval?

Of course, in practice the null hypothesis is not varied; these activities are for educational understanding.