Validity and Reliability, Sampling, Levels of Measurement

MMC 9002 (500) Researching Communication

Fall 2007 Lombard

Validity and Reliability, Sampling, Levels of Measurement

Validity and Reliability

Types of variables

•Independent variable vs. dependent variable

•Extraneous

•Control

•Uncontrolled

Relationship between variables

•Qualitative variables

•If one variable changes, other does

•Quantitative variables

•Positive or direct

•Negative or inverse

•Curvilinear

•Qualitative and quantitative

•Differences between various levels

•Causal

•Association

•Time-order

•Non-spuriousness

Ways to measure variables

•Direct

•Self-report

•Single or multiple items

•Observation

•Indirect

•Unobtrusive: collect data without individual’s knowledge

Validity

•Internal (causal): Did A really produce B?

•External: How well does the sample generalizes to a larger population?

•Measurement: Extent to which operation measures the concept as intended

Threats to internal validity

•History: events in real world affect people between 1st and 2nd test.

•Maturation: change due to time of task; aging, hungry, tired

•Testing: effect of being tested again

•Instrumentation: wording reactions

•Regression: effect of extreme scores moving to middle on retest

•Selection: bias in sample

•Mortality: drop out

Threats to external validity

•Reactivity: test situation different from real world

Reliability

•Deals with the operational level

•Extent to which measure obtains same result again and again

•Or the extent to which multiple measures measure the same thing

Ways to examine reliability

•Stability: consistency of instrument over time

•Test-retest

•Internal consistency: consistency of item performance

•Split-half & Cronbach’s alpha

•Equivalency: correlation between two forms of a test or different judges on the same test

•Intercoder reliability

•Alternative forms

Tradeoff between validity and reliability:

Sampling

The basic idea of sampling is that we seek knowledge about an entire population based on some cases - because it is impossible to measure everyone in a short amount of time.

Terms:

1. Population: A group or class of subjects, variables, concepts, etc.

Must define precisely before you can sample
This is the conceptual definition - the group in abstract.

2. Sampling Frame: The operational definition of the population - what you will actually choose the sample from.

Hopefully sampling frame and population are identical.
Differences between the two could mean systematic biases.

e.g., how good are phone books as a sampling frame?

3. Census: The case of measuring every member of a population.

4. Sample: A subset of the population, selected in any fashion. Ideally, a sample should be representative of the population, but this is seldom possible or even knowable.

Two types of sampling designs: Rules for selecting for sample.

1. Probability sampling: All elements have an equal chance for selection in the study.

This allows researchers to calculate the amount of sampling error present in the study.
A systematic selection procedure.
No bias on part of investigator.
Can apply statistical methods to the results.

a. Simple random sample: Every unit of sampling frame has an equal chance of being selected.

e.g., items selected by random number table.
No cases are favored.
Random start, then systematic sampling.

b. Stratified sample: A segment of the population is defined as important, and a random sample is take from each level of the segment.

Used to ensure sufficient representation of small segments.
This method is better, more efficient (smaller sample needed) when the strata are directly related to the dependent variable.

e.g., men and women as strata for drinking.

c. Cluster sample: A multistage design

e.g., use tracts or clusters: Randomly select county, districts within county, blocks within districts, households within blocks, etc.
Very cost efficient - don’t have to send people everywhere.

Simple random sample:

Systematic random sample:

Stratified random sample:

Cluster sample:

2. Non-probability sampling: Selection based on means other than chance.

a. Convenience: Researcher uses people most available.

people in mall, a class of students
used in market research, experiments

b. Purposive sample: Selection based on specific criteria

Used in field observation

c. Quota sample: Selection based on known, predetermined percentages that exist in real world.

Factors to consider in determining what sample size to use:

1. Project type and purpose: New, exploratory vs. fine-tuning previous research.

2. Project complexity: How will data be used, how many variables, how much precision needed?

3. Resources: Money, time, etc.

4. Population heterogeneity: More heterogeneousrequires more cases.

5. Desired precision: Larger sample provides more precision.

6. Sampling design: Some methods more efficient (e.g., stratified over S.R.S.)

Generally 400-500 is sufficient if you don’t have too complex of analyses or too many variables.

Eliminate non-random error from:

Incomplete sampling frames

Incomplete data collection

Definitions in describing samples:

Statistic: A characteristic of a sample.

Parameter: A characteristic of a population.

Measures of central tendency:

Mode: Most frequent answer

Median: Mid point

Mean: Average

Measure of dispersion:

Range: Difference between highest and lowest value

Variance: A measure of how condensed or spread out the values are around the mean.

Standard deviation: Square root of the variance.

[See example in Singleton, pages 141-3.]

Levels of measurement

Four levels at which variables can be measured

•Nominal

•Categorical, classification

•Property of equivalence

•Exhaustive and mutually exclusive

•Ordinal

•Rank along a specific dimension

•Properties of above and order

•Interval

•Properties of above and equal spacing

•Ratio

•Properties of above and a true zero point

Discrete versus continuous data: