Some Definitions of Statistical Terms
Sampling Error - The sampling process will always result in differing composition of a sample from one replication to the next. Sampling error is the fluctuation in a sample statistic over repeated samples, or replications, that is due to this differing composition of samples.
Sampling Distribution - The distribution of a sample statistic over repeated samples of a specific size. It shows the behavior of the statistic over a theoretically infinite number of replications of a randomly selected sample of a particular size.
Standard Error - The standard deviation of a sampling distribution. It is an estimate of the extent to which you expect a sample statistic to vary from a population parameter. It is a measure of the amount of sampling error we can expect there to be in a sample statistic.
Type I Error - Concluding falsely that your study shows evidence for the truthfulness of the hypothesis under investigation when the apparent effects are really due to sampling error. The hypothesis is really false but you conclude that it is true because you were fooled by sampling error. It is like saying there is a relationship, difference, or effect when there really isn’t one. You say the treatment works when it really does not work in the population condition. Your sample data just makes it appear like it works because of sampling error.
Type II Error - Concluding falsely that your study shows evidence for the falsehood of the hypothesis under investigation when the apparent lack of effects is really due to sampling error. The hypothesis is really true but you conclude that it is false because you were fooled by sampling error. It is like saying there in not a relationship, difference, or effect when there really is one. You say the treatment does not work when it really does work in the population condition. Your sample data just makes it appear like it does not work because of sampling error.
Test Statistic - A quantitative index of the magnitude of the treatment effects, differences, or relationship, that is used to make decisions about the truthfulness of a hypothesis.
p Value - The probability that the effects (as measured by the test statistic) you found are really due to sampling error under the condition that the null hypothesis is true. It answers the question: If this treatment really does not work, how likely is it that the effects observed could have been due to sampling error?
Statistical Significance - Are the results beyond what would be expected due to sampling error alone? Are the results large enough that it is very unlikely (small p value) that they are due to sampling error?
Practical Significance - Are the differences found in a particular study large enough to be of any practical value? Do the results translate into any change in actual practice? Do they translate into any change in policy?