SampleDesign.pdf

Sample Design

The two major decisions in designing your sampling plans are the sampling method and the

sample size.

Given the desire to generalize the results of a quantitative study, researchers will use a

probability procedure, if at all possible. This includes a simple random sample, systematic

sample, stratified sample, cluster sampling. A common form of cluster sampling is area sampling

where the clusters are the geographical area. The decision of what method is used is dependent

on a number of factors including practicality, methodology, and ethics.

Sample size determination for a probability sample is largely based on statistical theory. The

factors that have to be specified to determine the appropriate size is the variability in the

population, the degree of acceptable error, and the confidence interval. Note that it is not the size

of the population that is important but the degree of heterogeneity of the population. For

instance, if everyone in the population was exactly the same with respect to what you are

measuring, a sample of 1 would tell you all you need to know. While a researcher can determine

the sample size necessary statistically, this may have to be modified due to other factors. For

instance, if a survey was being conducted, to obtain the desired sample size given, the level of

precision, and confidence desired; the initial sample may have to be larger due to the completion

rate (the number of selected respondents who actually complete the interview or questionnaire)

as well as the incidence rate (the percentage of people eligible for participation) in the

population.

Unless you design your study adequately and select a sample of sufficient size, your design may

be a set-up for a Type II error—failing to find a difference or a relationship that is really there—

and your study may be largely a waste of time! You want to have a large enough sample to find a

relationship among constructs that is really there and to be able to argue that the relationship is

meaningful. At the same time, cost, and the ability to collect the desired number of sample

elements have to be considered.

There are four factors involved in calculating sample size:

1. Statistical test – Your sample size is partly a function of the statistical test you use. Some tests
(e.g., Chi-squared) require larger samples to detect a difference than others (e.g., ANCOVA).

2. Expected/estimated effect size – The effect size is potency of the strength of the relationship you
are investigating. In the language of statistics, an effect size is a difference between the mean
scores of two groups divided by the pooled standard deviation. This is called Cohen’s d. You will
calculate an effect size as part of the analysis of your data in order to determine that you have
found something meaningful (not merely statistically significant). However, in advance of doing
your study, you must estimate the effect size in your study.

3. Alpha. The alpha level is the probability of a Type I error—of rejecting the null, no difference,
hypothesis when it is true—that you are familiar with. By convention, this is set at p=.05. The
convention may not be your best guide. The null hypothesis is always false and can always be

rejected with a large enough sample, so a .05 level may unnecessarily require you to have a
larger sample than you need. It is best to use the literature and your judgment to justify an
alpha level that makes sense for your study. This justification will involve looking at the danger
of a Type I error versus the cost in resources of avoiding it.

4. Beta. The beta level is the probability of a Type II error—of accepting the null, no difference,
hypothesis when it is false, in other words, of failing to detect a difference when it is there. The
main point of an a priori power analysis is to estimate enough subjects and no more to detect a
difference. As with alpha, you set beta based on a judgment. The convention is .2, which yields a
power of .8 (1-beta). This is the lowest acceptable level.

Launch in a separate window

https://ncuone.ncu.edu/shared/Course%20Interactives/SB/BUS-7320/week_3_sample_design/story.html