How to Design and Evaluate Research in Education
By
Jack R. Fraenkel and Norman E. Wallen
Chapter 1The Nature of Research
Ways of knowing
Sensory experience (incomplete/undependable)
Agreement with others (common knowledge wrong)
Experts’ opinion (they can be mistaken)
Logic/reasoning things out (can be based on false premises)
Why research is of value
Scientific research (using scientific method) is more trustworthy than expert/colleague opinion, intuition, etc.
Chapter 1 - continuedThe Nature of Research
Scientific Method (testing ideas in the public arena)
Put guesses (hypotheses) to tests and see how they hold up
All aspects of investigations are public and described in detail so anyone who questions results can repeat study for themselves
Replication is a key component of scientific method
Chapter 1 - continuedThe Nature of Research
Scientific Method (requires freedom of thought and public procedures that can be replicated)
Identify the problem or question
Clarify the problem
Determine information needed and how to obtain it
Organize the information obtained
Interpret the results
All conclusions are tentative and subject to change as new evidence is uncovered (don’t PROVE things)
Chapter 1 - continuedThe Nature of Research
Types of Research
Experimental (most conclusive of methods)
Researcher tries different treatments (independent variable) to see their effects (dependent variable)
In simple experiments compare 2 methods and try to control all extraneous variables that might affect outcome
Need control over assignment to treatment and control groups (to make sure they are equivalent)
Sometimes use single subject research (intensive study of single individual or group over time)
Chapter 1 - continuedThe Nature of Research(Types of Research continued)
Correlational Research
Looks at existing relationships between 2 or more variables to make better predictions
Causal Comparative Research
Intended to establish cause and effect but cannot assign subjects to trtmt/control
Limited interpretations (could be common cause for both cause and effect…stress causes smoking and cancer)
Used for identifying possible causes; similar to correlation
Chapter 1 - continuedThe Nature of Research(Types of Research continued)
Survey Research
Determine/describe characteristics of a group
Descriptive survey in writing or by interview
Provides lots of information from large samples
Three main problems: clarity of questions, honesty of respondents, return rates
Ethnographic research (qualitative)
In depth research to answer WHY questions
Some is historical (biography, phenomenology, case study, grounded theory)
Chapter 1 - continuedThe Nature of Research(Types of Research continued)
Historical Research
Study past, often using existing documents, to reconstruct what happened
Establishing truth of documents is essential
Action Research (differs from above types)
Not concerned with generalizations to other settings
Focus on information to change conditions in a particular situation (may use all the above methods)
Each of these methods is valuable for a different purpose
Chapter 1 - continuedThe Nature of Research
General Research Types
Descriptive (describe state of affairs using surveys, ethnography, etc.)
Associational (goes beyond description to see how things are related so can better understand phenomena using correl/causal-comparative
Intervention (try intervening to see effects using experiments)
Chapter 1 - continuedThe Nature of ResearchQuantitative v. Qualitative
Quantitative (numbers)
Facts/feelings separate
World is single reality
Researcher removed
Established research design
Experiment prototype
Generalization emphasized
Chapter 1 - continuedThe Nature of Research
Meta-Analysis
Locate all the studies on a topic and synthesize results using statistical techniques (average the results)
Critical Analysis of Research (some say all research is flawed)
Question of reality (are only individual perceptions of it)
Question of communication (words are subjective)
Question of values (no objectivity only social constructs)
Question of unstated assumptions (researchers don’t clarify assumptions that guide them)
Question of societal consequences (research serves political purposes that are conservative or oppressive; preserve status quo)
Chapter 1 - continuedThe Nature of ResearchOverview of the Research Process (Fig. 1.4)
Introduction chapter
Problem statement that includes some background info and justification for study
Exploratory question or hypothesis (relationship among variables clearly defined); goes last in Ch.
Definitions (in operational terms)
Review of related literature (other studies of the topic read and summarized to shed light on what is already known)
Chapter 1 - continuedThe Nature of ResearchOverview of the Research Process (Fig. 1.4)
Methods chapter
Subjects (sample, population, method to select sample)
Instruments (tests/measures described in detail and with rationale for their use)
Procedures (what, when, where, how, and with whom);
Give schedule/dates, describe materials used, design of study, and possible biases/threats to validity
4. Data analysis (how data will be analyzed to answer research questions or test hypothesis)
Chapter 2The Research Problem
Statement of the Problem (identify a problem/area of concern to investigate)
Must be feasible, clear, significant, ethical
Research Questions (serve as focus of investigation, see p. 28 list)
Some info must be collected that answers them (must be researchable)
Cannot research “should” questions
See diagram, p. 29
Chapter 2 - ContinuedThe Research Problem
RQ should be feasible (can be investigated with available resources)
RQ should be clear (specifically define terms used…operational needed, but give both)
Constitutive definitions (dictionary meaning)
Operational definitions (specific actions/steps to measure term; IQ=time to solve puzzle, where <20 sec. is high; 20-40 is med.; 40+ is low)
RQ should be significant (worth investigating; how does it contribute to field and who can use info)
RQs often investigate relationships (two characteristics/qualities tied together)
Chapter 3 Variables and Hypotheses
Important to study relationships
Sometimes just want to describe (use RQ)
Usually want to look for patterns/connections
Hypothesis predicts the existence of a relationship
Variables (anything that can vary in measure; opposite of constant)
Variables must be clearly defined
Often investigate relationship between variables
Chapter 3 - ContinuedVariables and Hypotheses
Variable Classifications (Fig. 3.4, p. 42)
Quantitative (variables measured as a matter of degree, using real numbers; i.e. age, number kids)
Categorical (no variation…either in a category or not; i.e. gender, hair color)
Independent: the cause (aka the manipulated, treatment or experimental variable)
Dependent: the effect (aka outcome variable)
Extraneous: uncontrolled IVs (see Fig. 3.2, p. 46)
All extraneous variables must be accounted for in an experiment
Chapter 3 - ContinuedVariables and Hypotheses
Hypotheses – predictions about possible outcome of a study; sometimes several hypotheses from one RQ (Fig 3.3)
RQ: Will athletes have a higher GPA that nonathletes?
H: Athletes will have higher GPAs that nonathletes
Advantages to stating a hypothesis as well as RQ
Clarifies/focuses research to make prediction based on previous research/theory
Multiple supporting tests to confirm hypothesis strengthens it
Disadvantages
Can lead to bias in methods (conscious or un) to try to support hypothesis
Sometimes miss other important info due to focus on hypothesis (peer review/replication is a check on this)
Chapter 3 - ContinuedVariables and Hypotheses
Some hypothesis more important than others
Directional v. nondirectional
Directional says which group will score higher/do better
Nondirectional just indicates there will be a difference, but not who will score higher/do better
Directional more risky, so be careful/tentative in using directional ones
Chapter 4 Ethics and Research
Examples of unethical practices
Requiring participation from powerless (students)
Using minors without parental permission
Deleting data that don’t agree w/ hypothesis
Invading privacy of subjects
Physically or psychologically harming subjects
APA statement of ethical principles in research
Each student must sign one and have it signed by workplace supervisor
Chapter 4 - Continued Ethics and Research
Protecting participants from harm requires informed consent
Subjects must know the purpose of the study, possible benefits/harm; participation is voluntary and they can w/draw without penalty any time (Fig. 4.3, p. 59)
Researchers should ask: Could subjects be harmed? Is there another way to get the info? Is the info valuable enough to justify study?
Researchers must ensure confidentiality of data (limit access; no names if possible; tell subjects confidential or anonymous)
Deceiving subjects is sometimes necessary (Milgram study), ask if results justify ethical lapse
When deception used subjects they should be okay with it after (and they can refuse use of their data)
Chapter 4 - Continued Ethics and Research
Research with children
Parental consent required (signed permission from parents
APA Ethics in Research Form addresses this also
Regulation of Research (National Research Act of 1974)
If federal funding received must have an IRB to check: risks to subjects, informed consent guidelines met, debriefing plans for subjects
HHS made changes in 1981 so that educational research is exempt under certain conditions
Video 1
Chapter 5 Review of the Literature
Value of the Literature Review
Glean ideas from others interested in topic
See results of related studies (must be able to evaluated those objectively)
Types of sources
General References – indexes (of primary sources and abstracts (ERIC, Psych Abstracts)
Primary Sources – publications where researchers report their results (peer reviewed/refereed journals)
Secondary Sources – publications where authors describe works of others (encyclopedias, tradebooks, textbooks)
Chapter 5 - Continued Review of the Literature
Steps in the Literature Review (manual or electronic) See examples p. 74
Define problem precisely as possible
Review some secondary sources*
Review some general reference works*
Formulate search terms (keywords/descriptors)
Search general references for primary sources
Obtain and read primary sources (make notes/summarize)
*May be based on existing knowledge or previous reading
Chapter 5 - Continued Review of the Literature
Making notes
Include problem/purpose; hypotheses/RQ; procedures w/ subjects/methods; findings/conclusions; citation!
Searching strategies…use Boolean operators (AND, OR, NOT)
Searching www…be careful of reliability
Writing up the Literature Review
Introduction - describes problem and justification for study;
Body – discuss related studies together (#2, p.88)
Summary – ties literature together/give conclusions arising from literature
Reference list
Don’t replace a review of primary sources with meta-analysis (a combined review of all available research on a topic w/ results averaged)
End Part 1
Chapter 6 Sampling
Sample – any group on which info is obtained
Population – group that researcher is trying to represent
Population must be defined first; more closely defined, easier to do, but less generalizable
Study a subset of the population because it is cheaper, faster, easier, and if done right, get same results as a census (study of whole pop)
Accessible population – the group you are able to realistically generalize to…may differ from target population
Chapter 6 - Continued Sampling(Random v. Nonrandom Sampling)
Random – every population element has an equal and independent chance to participate
Uses names in a hat or table or random numbers
Elimination of bias in selecting the sample is most important (meaning the researcher does not influence who gets selected)
Ensuring sufficient sample size is second most important
Nonrandom/purposive - troubles with representativeness/generalizing
Chapter 6 - Continued Sampling(Random Sampling Methods)
Simple random sampling
Names in a hat or table of random numbers--p.99
Larger samples more likely to represent pop.
Any difference between population and sample is random and small (called random sampling error)
Stratified random sampling
Ensures small subgroups (strata) are represented
Normally proportional to their part of pop.
Break pop into strata, then randomly select w/in strata
Multistage sampling (see p. 94)
Chapter 6 - Continued Sampling(Random Sampling Methods, cont.)
Cluster random sampling
Select groups as sample units rather than individuals
REQUIRES a large number of groups/clusters
Multistage sampling (see p. 94)
Systematic (Nth) sampling
Considered random is list if randomly ordered or nonrandom if systematic w/ random starting point
Divide pop size by sample size to get N (ps/ss=N)
Chapter 6 - Continued Sampling(Non-Random Sampling Methods)
Systematic can be nonrandom if list is ordered
Convenience sampling
Using group that is handy/available (or volunteers)
Avoid, if possible, since tend not to be representative due to homogeneity of groups
Report large number of demographic factors to see likeliness of representativeness
Purposive sampling
Using personal judgment to select sample that should be representative (i.e., this faculty seems to represent all teachers) OR selecting those who are known to have needed info (interested in talking only to those in power)
Snowball is a type (used with hard to identify groups such as addicts)
Chapter 6 - Continued Sampling
Sample size affects accuracy of representation
Larger sample means less chance of error
Minimum is 30; upper limit is 1,000 (see table)
External validity – how well sample generalizes to the population
Representative sample is required (not the same thing as variety in a sample)
High participation rate is needed
Multiple replications enhance generalization when nonrandom sampling is used
Ecological generalization (gen to other settings/conditions, such as using a method tested in math for English class)
Video 17
Chapter 7 Instrumentation(Measurement)
Data – information researchers obtain about subjects
Demographic data are characteristics of subjects such as age, gender, education level, etc.
Assessment data are scores on tests, observations, etc. (the device used to measure these is called the measurement instrument)
Key questions in data measurement/ instrumentation
Where and when will data be collected
How often will data be collected
Who will collect the data
Chapter 7 - Continued Instrumentation
Validity – measures what it is supposed to (accurate)
Reliability – a measure that consistently gives same readings (repeatable)
Objectivity – absence of subjective judgments (need to eliminate subjectivity in measuring)
Usability of instruments
Consider ease of administration; time to administer; clarity of directions; ease of scoring; cost; reliability/validity data availability
Chapter 7 - Continued Instrumentation(Classifying Data Collection Instruments)
By the group providing the data
Researcher instruments (researchers observes student performance and records)
Subject instruments (subjects record data about themselves, such as taking test)
Others/Informants (3rd party reports about subjects such as teacher rates students)
By where instrument came from
Preference is for existing ones ( MMY
Can develop your own (requires time, effort, skill, testing; see p. 125)
By response type
Written response – preferred – objective tests, rating checklist
Performance instruments – measure procedure, product
Chapter 7 - Continued Instrumentation(Examples of Data Collection Instruments)
Researcher Completed Instruments
Rating scales (mark a place on a continuum for example numeric rating 1=poor to 5= excellent)
Interview schedules (complete scales as interview takes place; use precoding; beware of dishonesty)
Tally sheets (for counting/recording frequency of behavior, remarks, activities, etc.)
Flow charts (to record interactions in a room)
Anecdotal records (need to be specific and factual)
Time/Motion logs (record what took place and when)
Chapter 7 - Continued Instrumentation(Examples of Data Collection Instruments)
Subject Completed Instruments
Questionnaires (question clarity to reader essential)
Self checklists
Attitude scales (Likert is one type, how much subject agrees/disagrees with descriptive statements about a topic indicates a positive/negative attitude toward topic)
Semantic differential (good/bad; poor/excellent ratings)
Personality profiles
Achievement/Aptitude tests
Performance tests
Projective devices (Rorschach Ink Blot Test)
Sociometric devises (peer ratings)
Chapter 7 - Continued Instrumentation
Item Formats
Selection items or closed response (T/F; Yes/No; Right/Wrong; Multiple choice)
Supply items or open ended (short answer; essay)
Unobtrusive measures (no intrusion into event… usually direct observation and recording)
Types of Scores
Raw scores (initial score or count obtained…w/out context)
Derived scores (raw scores translated to meaningful usage with standardized process)
Age/Grade equivalence; Percentile ranks; Standard scores (how far a score is from a given reference point, i.e. z and T scores);
Which to use depends on the purpose; usually standard scores used
Chapter 7 - Continued Instrumentation
Norm Referenced v. Criterion Referenced Tests
Norm referenced scores give a score relative to a reference group (the norm group)
Criterion referenced scores determine if a criterion has been mastered
These are used to improve instruction since they indicate what students can or cannot do or do or do not know
Chapter 7 - Continued Instrumentation(Measurement Scales)
Nominal (in name only)
Numbers are only name tags, they have no mathematical value (gender: 1=male and 2= female OR race: 1= Blk, 2=Wht, 3=other)
Ordinal (in name, plus relative order)
Numbers show relative position, but not quantity (grade level, finishing place in a race)
Interval (in name w/ order AND equal distance)
Numbers show quantity in equal intervals, but an arbitrary zero (can have negative numbers; degrees C or F)
Ratio (in name, w/ order, eq. distance AND absolute zero)
Numbers show quantity with base of zero where zero means the construct is absent
Higher levels more precise…collect data at highest level possible; some statistics only work with higher level data
Chapter 7 - Continued Instrumentation(Preparing for Data Analysis)
Scoring data – use exact same format for each test and describe scoring method in text
Tabulating and Coding – carefully transfer data from source documents to computer
Give each test an ID number
Any words must be coded with numerical values
Report codes in text of research report
Video 18
Chapter 8 Validity and Reliability(Quality of instruments is important)
Validity is most important aspect of measures
Means accuracy, correctness, usefulness of instrument
Validation is the process of collecting and analyzing evidence to support inferences based on an instrument
Test publishers usually give a statement of intended use as well as evidence to support validity
Reliability (consistency in scoring) is part of validity
Chapter 8 - ContinuedValidity and Reliability(Three ways to establish validity)
Content validity – is entire content of construct covered by test, are important parts emphasized?