MMC 9002 (500) Researching Communication
Fall 2007 Lombard
Validity and Reliability, Sampling, Levels of Measurement
Validity and Reliability
Types of variables
•Independent variable vs. dependent variable
•Extraneous
•Control
•Uncontrolled
Relationship between variables
•Qualitative variables
•If one variable changes, other does
•Quantitative variables
•Positive or direct
•Negative or inverse
•Curvilinear
•Qualitative and quantitative
•Differences between various levels
•Causal
•Association
•Time-order
•Non-spuriousness
Ways to measure variables
•Direct
•Self-report
•Single or multiple items
•Observation
•Indirect
•Unobtrusive: collect data without individual’s knowledge
Validity
•Internal (causal): Did A really produce B?
•External: How well does the sample generalizes to a larger population?
•Measurement: Extent to which operation measures the concept as intended
Threats to internal validity
•History: events in real world affect people between 1st and 2nd test.
•Maturation: change due to time of task; aging, hungry, tired
•Testing: effect of being tested again
•Instrumentation: wording reactions
•Regression: effect of extreme scores moving to middle on retest
•Selection: bias in sample
•Mortality: drop out
Threats to external validity
•Reactivity: test situation different from real world
Reliability
•Deals with the operational level
•Extent to which measure obtains same result again and again
•Or the extent to which multiple measures measure the same thing
Ways to examine reliability
•Stability: consistency of instrument over time
•Test-retest
•Internal consistency: consistency of item performance
•Split-half & Cronbach’s alpha
•Equivalency: correlation between two forms of a test or different judges on the same test
•Intercoder reliability
•Alternative forms
Tradeoff between validity and reliability:
Sampling
The basic idea of sampling is that we seek knowledge about an entire population based on some cases - because it is impossible to measure everyone in a short amount of time.
Terms:
1. Population: A group or class of subjects, variables, concepts, etc.
- Must define precisely before you can sample
- This is the conceptual definition - the group in abstract.
2. Sampling Frame: The operational definition of the population - what you will actually choose the sample from.
- Hopefully sampling frame and population are identical.
- Differences between the two could mean systematic biases.
e.g., how good are phone books as a sampling frame?
3. Census: The case of measuring every member of a population.
4. Sample: A subset of the population, selected in any fashion. Ideally, a sample should be representative of the population, but this is seldom possible or even knowable.
Two types of sampling designs: Rules for selecting for sample.
1. Probability sampling: All elements have an equal chance for selection in the study.
- This allows researchers to calculate the amount of sampling error present in the study.
- A systematic selection procedure.
- No bias on part of investigator.
- Can apply statistical methods to the results.
a. Simple random sample: Every unit of sampling frame has an equal chance of being selected.
- e.g., items selected by random number table.
- No cases are favored.
- Random start, then systematic sampling.
b. Stratified sample: A segment of the population is defined as important, and a random sample is take from each level of the segment.
- Used to ensure sufficient representation of small segments.
- This method is better, more efficient (smaller sample needed) when the strata are directly related to the dependent variable.
e.g., men and women as strata for drinking.
c. Cluster sample: A multistage design
- e.g., use tracts or clusters: Randomly select county, districts within county, blocks within districts, households within blocks, etc.
- Very cost efficient - don’t have to send people everywhere.
Simple random sample:
Systematic random sample:
Stratified random sample:
Cluster sample:
2. Non-probability sampling: Selection based on means other than chance.
a. Convenience: Researcher uses people most available.
- people in mall, a class of students
- used in market research, experiments
b. Purposive sample: Selection based on specific criteria
- Used in field observation
c. Quota sample: Selection based on known, predetermined percentages that exist in real world.
Factors to consider in determining what sample size to use:
1. Project type and purpose: New, exploratory vs. fine-tuning previous research.
2. Project complexity: How will data be used, how many variables, how much precision needed?
3. Resources: Money, time, etc.
4. Population heterogeneity: More heterogeneousrequires more cases.
5. Desired precision: Larger sample provides more precision.
6. Sampling design: Some methods more efficient (e.g., stratified over S.R.S.)
Generally 400-500 is sufficient if you don’t have too complex of analyses or too many variables.
Eliminate non-random error from:
Incomplete sampling frames
Incomplete data collection
Definitions in describing samples:
Statistic: A characteristic of a sample.
Parameter: A characteristic of a population.
Measures of central tendency:
Mode: Most frequent answer
Median: Mid point
Mean: Average
Measure of dispersion:
Range: Difference between highest and lowest value
Variance: A measure of how condensed or spread out the values are around the mean.
Standard deviation: Square root of the variance.
[See example in Singleton, pages 141-3.]
Levels of measurement
Four levels at which variables can be measured
•Nominal
•Categorical, classification
•Property of equivalence
•Exhaustive and mutually exclusive
•Ordinal
•Rank along a specific dimension
•Properties of above and order
•Interval
•Properties of above and equal spacing
•Ratio
•Properties of above and a true zero point
Discrete versus continuous data:
1