PRR 475 Topic OutlinesPage 1
PRR 475. Evaluation in Parks, Recreation and Tourism
D. Stynes
Topic Outlines
TABLE OF CONTENTS
I. Evaluation and Research Concepts
II. Research And Evaluation Processes : The Study Plan/Proposal
III. Definition and Measurement
IV. Study Population and Sampling
V. Survey Research Methods
VI. Experimental Designs
VII. Other Methods
VIII. Statistics and Data Analysis
IX. Ethical, Social, and Political Issues in Research and Evaluation
X. Communicating Results of Research and Evaluation Studies
I. EVALUATION & RESEARCH CONCEPTS
1.1. DEFINITION-Evaluation - process of judging merit or worth of something
Evaluation is the systematic assessment of the worth or merit of some object. Joint Committee on Standards for Educational Evaluation.
Evaluation research is the systematic application of social research procedures in assessing the conceptualization and design, implementation, and utility of social intervention programs (Rossi & Freeman, p. 20).
Evaluation is the process of ascertaining value by comparing results with objectives and judging how well objectives have been met in a qualitative and quantitative sense. (Lundegren & Farrell 1985).
Evaluation is a set of procedures to appraise a program's merit and to provide information about its goals, expectations, activities, outcomes, impacts, and costs. (Kosecoff & Fink 1982).
The process of determining the value or amount of success in achieving a predetermined objective. It includes at least the following steps: formulation of objectives, identification of the proper criteria to be used in measuring success, recommendations for further program activity.
The process of collecting and analyzing information about a social program, its audience, or its impacts on an audience for the explicit purpose of improving its ability to serve the audience in the intended ways. (Ham 1988)
Evaluation is the process of judging the merit, worth, or value of something. (Carpenter & Howe, Introd to leisure pgmming & pgmming cycle)
1.2. PURPOSES OF EVALUATION - Why evaluate ?
The "Academic List" "The Real List"
To assess merits of alternative programs. 1. Because we are required to
To discover whether & how well objectives are being fulfilled2. To make better decisions.
To determine the reasons for successes & failures.3. To learn from experience
To uncover the principles underlying a successful program.4. To justify programs
To refine, revise, update or track a program5. To kill programs
2.1. DEFINITION - Research - application of scientific methods to answer questions, controlled inquiry directed at increasing knowledge and establishing truth.
Scientific research is systematic, controlled, empirical, and critical investigation of natural phenomena guided by theory and hypotheses about the presumed relations among such phenomena. (Kerlinger)
Marketing research is a formalized means of obtaining information to be used in making marketing decisions. (Tull & Hawkins).
Social research is the systematic examination of empirical data, collected by someone first hand, concerning the social or psychological forces operating in a situation (Monette, Sullivan and DeJong 1990)
Marketing research is the function which links the consumer, customer, and public to the marketer through information -- information used to identify and define marketing opportunities and problems; generate refine, and evaluate marketing actions; monitor marketing performance; and improve understanding of marketing as a process. (American Marketing Assoc.)
2.2. Purposes of research
Set A Set B
1. Answer questions of fact arising from management - applied research1. Exploratory
2. Answer questions for sake of knowing - basic or pure research2. Descriptive
3. Answer questions arising during research - methodological research3. Explanatory, predictive
4. Develop & test new alternatives for management - developmental
5. Assess the worth, merit, or effectiveness of programs - evaluation
3.SCIENCE
3.1. Science may be defined as a body of knowledge or a method of inquiry.
AS A BODY OF KNOWLEDGE, science is
systematic - propositions related within a body of theory
abstract - don't explain everything, extract most important features of phenomona under study,
general - seek general laws vs knowledge of isolated events
parsimonious - prefer simpler explanation
AS A METHOD OF INQUIRY, science is
logical - hypothetico-deductive system, deductive & inductive logic
self-corrective - iterative search for knowledge, everything open to systematic questioning.
empirical - ultimate test of truth are observations in "real" world.
3.2. Scientific Management is the application of scientific principles to management. It entails systematic study of the system(s)one is managing and application of this information to improve management peformance or to "steer" the system toward a set of goals and objectives. See any text in management for a summary of scientific management theory. Howard & Crompton's Financing, Managing & Marketing Recreation & Park Resources, Chapter 10 summarizes the basic theories of management. As originally formulated by Taylor in early 1900's, scientific management involved experimentation to find the most effective way of doing things, working to standards, planning out work tasks, and systems of inspection and control. A scientific approach is also implicit in human relations, organizational and behavioral approaches to management. Scientific management is also associated with the systems approach,and the fields of management science/operations research, which apply mathematical models to help solve management problems.
3.3. Leisure Science is a term used to describe scientific investigations into leisure behavior. Leisure research includes systematic study of people's leisure behavior, as well as examination of recreation, park and tourism organizations and resources. Leisure science is an interdisciplinary field dominated largely by social science, ie. study of people. Leisure organizations and behaviors have been investigated from psychological, sociological, economic, geographic, political, communications, physiological, anthropological, management, marketing, and natural resource perspectives. Leisure research is largely applied research and borrows heavily from disciplines and related applied fields.
3.4 Tourism Science similarly characterizes a body of knowledge about tourism while expressing a scientific orientation of tourism management and scholarly work on tourism.
Major leisure and tourism research areas include:
Outdoor recreation research
Therapeutic recreation
Recreation and tourism marketing
Community recreation needs studies
Impact studies (economic, social, environmental)
Evaluation research (evaluating programs and management/marketing alternatives)
Leisure time budget studies
Consumer behavior studies
Research to support recreation planning: supply, demand, forecasts
Studies into the nature of leisure & leisure experiences
Recreation, Park and Tourism Management studies
4. Evaluation research - simply combine a definition of research with one of evaluation.
4.1. Distinguishing between research & evaluation studies:
RESEARCH / EVALUATIONDefine problems or questions / Describe Program to be evaluated
Objectives or Hypotheses / Evaluation criteria
Literature Review / Program scoping
Research Methods / Evaluation Procedures
Data Gathering & Analysis / Data Gathering & Analysis
Conclusions & Recomendations / Conclusions & Recomendations
5. Discussion Questions & Exercises:
1. Formulate a definition of evaluation research.
2. Can you give an example of:
a. An evaluation study that is not research.
b. A research study that isn't an evaluation study.
c. A study that is both evaluation and research.
d. A study that is neither.
3. Explain the relationship between evaluation, research, and evaluation research.
4. Contrast the meanings and use of the words evaluation, assessment, diagnosis, monitor and audit. Are there any differences in your use of these terms, popular use, and use within the park,recreation, and tourism fields?
5. Contrast the meanings and use of the terms research, science, survey, experiment, study, investigation, analysis, discovery, explanation?
6. Browse periodicals and journals in the parks, recreation, leisure, and tourism fields.
a. Identify articles that are evaluations, scientific research, evaluation research.
b. Distinguish between opinion pieces and reporting of scientific information.
c. Observe differences in the format and style of research papers.
SELECTED REFERENCES
Agnew, N.M & Pyke, S.W. (1978). The Science game.: An introduction to research in the behavioral sciences. Englewood Cliffs, NJ: Prentice Hall.
Babbie, E. (1995). The practice of social research. Belmont, CA: Wadsworth.
Barnett, L.A. (Ed). (1988). Research about leisure: Past, present and future. Champaign, IL: Sagamore.
Katzer, J., Cook, K.H., & Crouch, W.W.(1991). Evaluating information: A guide for users of social science research. New York: McGraw Hill.
Kraus, R. & Allen, L. (1987). Research and evaluation in recreation, parks, and leisure studies. Columbus, OH: Publishing Horizons.
Lundegren, H.M. & Farrell, P. (1985). Evaluation for leisure service managers. Philadelphia: Saunders.
Ng, D. & Smith, S.L.J. (1982). Perspectives on the nature of leisure research.Waterloo, Ontario: Ontario Research Council on Leisure, University of Waterloo.
Rossi, P. & Freeman, H. (1985). Evaluation: A systematic approach. Beverly Hills, CA: Sage.
Ritchie, J.R.B & Goeldner, C.R. (1994). Travel, tourism and hospitality research: A handbook for managers. New York: John Wiley.
Suchman, E.A. (1967). Evaluation research. New York: Russell Sage Foundation.
6. TYPES OF EVALUATION / KINDS OF RESEARCH
6.1. Types of evaluation: Approaches from "Is everyone happy?" to experimental designs.
6.1.1. by program stage: new program, fine tune, revise established pgm
1. formative (conceptualization & design),
When2. process (implementation),
/Why3. summative (impacts & efficiency)
6.1.2. by approach:
How1. standards
a. norm-based
b. criterion referenced
2. goals,
3. impacts
6.1.3. by method
How1. qualitative---quantitative
2. secondary data, experiment, survey
6.1.4. by who evaluates:
Who1. internal, external;
2. peers, superiors, subordinants, participants, scientists.
6.1.5. by type of program: TR, interpretation., commercial, operations, programming, planning
6.1.6. by program area: lands, facilities, personnel, budgets, marketing, public relations, maintenance, programs, policies, plans.
6.1.7. by evaluation criteria (Suchman Ch 4)
1. effort: qty & qlty of inputs
2. performance: qty & qlty of outputs
3. adequacy of performance: meet needs?
4. efficiency: costs/benefits
5. process: how & why program works
a. attributes that make it more or less successful
b. recipients who are more or less affected.
c. conditions
d. effects
1. single or multiple
What2. intended or side-effects
3. duration
4. type: cognitive, affective, behavioral
(6). equity: distributional issues, "fairness".
6.1.8. by hierarchy of effects/generality (Suchman Ch 4)
immediate, intermediate, ultimate
level of objective: from abstract to specific, whole to parts to smaller parts.
6.1.9. Some examples of different kinds of evaluation studies
cost benefit analysiscost effectiveness analysis
Management by objectives (MBO)program audits
peer reviewsimpact assessment
needs assessmentfeasibility study
6.2. Kinds of Research
6.2.1. Applied - Basic
6.2.2. Exploratory, Descriptive, Explanatory, Predictive
6.2.3. Classified by application to evaluation, planning, policy analysis, marketing, management, etc.
6.2.4. By Discipline - social science, physical science, management science, biological science
Social Sciences: Sociology, Psychology, Economics, Geography, Political Science, Anthropology, Applied Sciences: Marketing, Natural Resources, Management, ...
6.2.5. Research topic - boating, wilderness, benefits, leisure constraints, tourism, urban...
6.2.6. In-house/ out-house
6.2.7. By method : survey, experiment, observation, secondary data, qualitative,...
TOPIC II. RESEARCH AND EVALUATION PROCESSES : THE STUDY PLAN/PROPOSAL
Research and evaluation are processes - i.e. they involve a systematic series of steps.
A. PROBLEM DEFINITION: Work from broad problem down to the focus of the proposed study. Provide background on the problem or program and show where the proposed study fits into a program of research or a program of evaluation studies. Use objective language - to test vs to prove, to assess or evaluate vs to show or demonstrate (that the program is effective). Keep in mind the client for study. Who will use results for what kinds of decisions? State problem in terms users can relate to.
a. Identify a problem(s) or question(s), usually begin as “messes” = systems of interrelated problems
b. Refine this to one or more researchable questions or evaluable programs. This often requires that we simplify (make simplifying assumptions) or delimit the problem (reduce its scope).
c. Immerse yourself in the problem or program : literature review, program scoping, etc.
d. Feasibility/evaluability assessment : Is question answerable/program evaluable given current knowledge, conditions, resources, political climate, etc.?
Research - Define the problem to be addressed. Draw from previous research and theory from the literature as well as secondary data and observations.
Evaluation - Define the program to be evaluated by reviewing written documents (program proposals, plans, budget, organization chart, program history,...) , talking with people (program administrators, managers, participants, & others), and observing (observe various parts of the program in operation).
B. OBJECTIVES: Develop a set (list) of concise, precise statements of study objectives. These translate the problem into concrete study objectives(achieving the objectives should solve the problem or answer questions, as stated in A). Usually two or three clear objectives is sufficient. The objectives guide the study not to mention helping to structure the methods and write ups of study proposal and final reports. Methods or procedures should not be listed as objectives ( e.g. to conduct a survey of visitors is not an objective, it is a method). Objectives should answer why you are conducting the survey and not jump to conclusion about the best approach.
Research - Phrased as questions to be answered or hypotheses to be tested. List objectives in a logical order.
Evaluation - Evaluation study objectives should help define the criteria by which the program will be evaluated (prioritize these). e.g. Determine the degree to which program objectives are met, determine if program benefits exceed costs, assess the positive and negative impacts of the program. While the program objectives should be taken into account in writing evaluation study objectives, the research objectives are not the same as the program objectives. E.g., If program strives to satisfy the customer, one evaluation objective may be to assess levels of satisfaction.
C. LITERATURE REVIEW/ PROGRAM SCOPING This is the place for more extended discussion of literature and background for the study. A “literature review” section is more common in research studies than evaluations. "Literature" can include research articles, previous studies, internal documents and plans, laws and policy statements as well as more popular magazine and newspaper articles. This section of the proposal demonstrates you have done your homework and are familiar enough with the topic (program) to study it (evaluate it). Key here is to review related research and theory, not everything.
Research - Review scholarly literature related to the topic. Key is linking literature to present study.
Evaluation - In program evaluations, provide a more extensive history of the program, review previous evaluation studies of this or related programs, and discuss key concepts, theories or research/evaluation approaches that are relevant here.
D. METHODS: Describe the procedures to be carried out to achieve each of the study objectives. Procedures cover what you intend to do to whom, when, where and how. Also cover why by linking procedures to your objectives. Longer proposals may also justify the proposed procedures by discussing the advantages and disadvantages of alternative approachess. Methods should be indicated for each objective. Typical sub-sections for the procedures section are:
Study Population
Sampling Procedures
Definition of key concepts
Measurement procedures
Overall study design, controlling for error
Field Procedures
Data Processing
Analysis
Reporting of results
Limitations and potential sources of error
E. EVALUATION STUDY PLAN - RESEARCH PROPOSAL. Ideally, the results of Steps A-D are assembled into a written document, which may be called a research proposal, study plan, or evaluation study plan. This document is then reviewed by the study client, other researchers, a committee, etc. The idea is to plan the study out from the beginning and make sure the procedures are appropriate, feasible, and will address the questions asked, and the intended problem. A complete proposal could be turned over to any competent researcher/evaluator to execute and presumably any investigator following the procedures/plan should arrive at the same results. Plan should be reviewed, evaluated and revised as needed before beginning the data gathering. Pre-testing of instruments and procedures may be part of the prescribed procedures.
F. COLLECT DATA, FIELD PROCEDURES. Empirical research & evaluations will propose specific data to be gathered using specific procedures/instruments. This generally involves making measurements of variables on a set of entities. The entities are often individuals or social groups when studying people, although visits, trips, lands, trees, wheelchairs, swingsets etc. are also things that can be measured.
G. PROCESS AND ANALYZE THE DATA. The proposed procedures for analyzing the data are carried out in accordance with the study plan. Possibly distinct sets of analyses may be carried out for each study objective. Analysis frequently involves applying statistical procedures to estimate “population parameters”, test hypotheses, or estimate predictive models. Many applied research and evaluation studies require only simple computations of percentages and averages, perhaps supplemented by a few crosstabulations of the gathered data. In either case, the analysis procedures should be prescribed in advance, rather than adjusted to obtain the “desired results”.
H. REPORTING and COMMUNICATING THE RESULTS. Results should be communicating in one or more forms to relevant audiences. Reports may vary from short summaries or oral presentations for lay audiences to long technical reports for specialists. Managers usually fall in between. Researchers generally present the results in a factual manner in the “Results” section. An implications, discussion, or recommendation section provides more room for the researchers interpretation of the findings. Limitations and problems with the study should be openly discussed in a “limitations” section. Suggestions for further study are often made, acknowledging that most studies provide as many questions for further study as answers.
I. COMPLETING THE CYCLE. The researcher/evaluator finishes by documenting and storing files and other information that could be useful should someone want to do further analysis or repeat the study. In applied research and evaluation studies, researchers/evaluators may or may not be involved in implementing recommendations or carrying the study through additional steps.
PRR 475 Topic OutlinesPage 1
PRR 475 Topic OutlinesPage 1
Table 1. Alternative Research Designs
Where Data is Collected
How Data Is Gathered / Household / On-Site / Laboratory / OtheraPersonal Interview / Surveys / Surveys & Field Experiments / Focus Groups / Surveys & Field Experiments
Telephone or Computer Inteview / Surveys / Computer Interviews / Computer Interviews
Self-Administered Questionnaire / Surveys & Field Experiments / Experiments / Surveys
Observation & Traces / NA / Observable Characteristics / Observable Characteristics / Observable Characteristics
Secondary Sources / NA / Internal Records / NA / Gov't , Industry & Other external sources
a. Other locations include highway cordon studies, mall intercept surveys, surveys at outdoor shows & other special events