Using a simulation to estimate how unlikely is it that the difference you observed could be due to chance alone

The t test works by modeling what would happen if the treatment had no effect.
That is, the t test models what would happen if differences between the means in your study were do entirely to chance.
The general question it asks is, "If you took two samples from the same population, took the mean of each sample, and then subtracted those two means to get the difference between means,
what percent of the time would you get a difference as big or bigger than the one you observed in your study?"
The specific question it asks is, "If you took two samples of the same number of observations as you had in each of your groups and from a population that had a standard deviation was almost the same as the standard deviation of your groups,
what percent of the time would you get a difference as big or bigger than the one you observed in your study?"

Rather than looking at the mathematics and theory that the t test uses to model what would happen if the treatment had no effect,
you can run the simulation below to see what happens. This simulation produces results that will approximate but not duplicate a t test's by doing the following 100 times:

1. One random sample of a certain size is taken from a population that has a certain mean and standard deviation, and the mean of that sample is calculated.
2. A second random sample of a certain size is taken from a population that has a certain mean and standard deviation, and the mean of that sample is calculated.
3. The mean of the second random sample is subtracted from the mean of the first random sample

You can then see how often the groups differ by a certain amount by chance alone.

To run the simulation, enter a mean, an estimate of the population's standard deviation, the number of scores in each group, and a value for the difference between means.
Then, click the "Run the simulation" button.
If you have data from an experiment, you can use your experiment's data to fill in these values. Use you can use one of your group means (or, even better, the average of your group means) for the mean,
the standard deviation of one of your groups (or, even better, the average of your group's standard deviations) for your estimate of the population's standard deviation,

the number of participants in one of your groups for sample size, and
the difference between your group means for the observed difference.

If you don't have any data, play around and see what happens when you vary

1. The sample size
2. The estimate of the population's standard deviation, and
3. The observed difference

Enter an estimate of the population mean (for this simulation, the actual value is not that important):

Enter your estimate of the population's standard deviation:

Sample size: How many scores will each sample mean be based on?

Observed Difference: How big is the difference between the two sample means?

© 2025 Mark L. Mitchell All rights reserved.