WaldenUniversity

Project Study Rubric with Explanations

July, 2010

Abstract
(Ed.D. Project Studies) / Explanations in this column added 6/18/10
  1. Abstract includes all of the following:
  1. Describes the overall study problem being addressed in the first couple of sentences and indicates why it is important (e.g. who would care if the problem is solved).
  2. Identifies the rationale for the problem and the project.
  3. Identifies the theoretical foundations, if appropriate.
  4. Summarizes the key research/guiding question
  5. Describe the overall research design, methods and specific data analytic procedures.
  6. (For the final project study): Identifies the key results, the project, and recommendations that capture the heart of the study.
  7. Concludes with a statement on the implications for positive social change and applications to the local problem
Comments: (click here) /
  • Proposal abstract focuses on problem and research methods
  • Proposal stage: Must include the research methods. The project must not be included.
  • Final project study: Must include research findings and project.
  • Follow the Abstract rubric
  • Note thatprogram and policy evaluation are possible designs for the project study.

Section 1: The Problem (Proposal)
(Ed.D. Project Studies)
Quality Indicators /
Overview
  • Focus is on the local problem
  • The “project” must NOT be determined until the research has been completed
  • Documentation is needed supporting the existence of and the nature of the local problem

1.1 The local problem that prompted the study is clearly defined and is discussed in terms of the local setting and the larger population or education situation.
Comments: (click here) /
  • Local problem is required.
  • Local setting must be clearly defined and described. This is usually the student’s place of employment. Exceptions should be carefully considered
  • Discuss the relationship of the problem to the local setting
  • Discuss the relationship of the problem to the larger educational setting as discussed in the scholarly literature.
  • Does NOT need to be expressed as a gap in the research, but needs to note a gap in practice.
  • Must NOT be worded as in the practice problem statement template from the Residencies: “There is a problem at XXX. That problem specifically is YYY.”
  • Must be written in appropriate scholarly language using their own words

1.2 The rationale for choosing this problem is included.
Comments: (click here) /
  • Rationale or justification for the problemchoiceconsists of evidence that the problem exists and why we need to study and address this problem.
  • It must NOT include the rationale for the research design or method, nor for the project.
  • Requires support from data, and may include appropriate personal communications if no other data available. Answers the question, “Who thinks this is a problem other than you?”
  • Does NOT need to be expressed as a gap in the research, but needs to note a gap in practice.
  • Concludes with the purpose or intent.

1.3 Any special terms associated with the problem are defined and cited.
Comments: (click here) /
  • Special terms: Include variables or conceptual terms used in the particular study.
  • Omit common terms not used in a special way.
  • Must be supported by scholarly literature or local document.
  • Definitions: NOdictionary, NOencyclopedia, NOWikipedia, etc.
  • Terms related to methodology belong in Section 2.

1.4 The significance of the problem is clearly discussed.
Comments: (click here) /
  • Significance is about the problem, and not about the project or the research methods/design.
  • For example, it may answer the question: How might studying this problem be useful to the local educational setting?

1.5 The problem is stated clearly in the form of a guiding or research question.
This would include one-two paragraphs briefly summarizing past research on the topic (no references needed), specifying the local problem, a gap in past research if applicable and the type of research needed to address the local problem.
Comments: (click here) /
  • Lists guiding/research question(s).
  • Questions investigate the nature of the problem if uncertain and the best solution to the problem.
  • Includes a statement to frame questions in relation to problem and purpose/rationale.
  • Check to make sure questions are aligned well with problem, purpose/rationale, and methodology.

1.6 Review of Literature addressing the problem:
a.The theoretical base/conceptual framework related to the problem is discussed in a manner that justifies the investigation of this problem as a worthwhile scholarly endeavor.
b.Critical review that documents the broader problem associated with the local problem addressed in the study is drawn primarily from recent articles published in acceptable peer-reviewed journals or sound academic journals and texts, or there is a justification for using other sources. Literature from diverse perspectives, cultures, gender, etc. is included as appropriate.
c. Saturation has been reached in the literature review. Search terms (including Booleans) and efforts to find related research are explained.
d. Any relevant public data are discussed.
Comments: (click here) /
  • Section 1 review must focus only on the problem.
  • Any references to the methodology belong in Section 2
  • Any references to the project belong in Section 3
  • Any literature themes related to the project belong in Section 3 literature review.
  • Must focus on the problem.
  • Demonstrate saturation and indicate how search was conducted.
  • 25-40 current, peer-reviewed sources (mostly primary sources)for the Section 1 review in addition to conceptual references and seminal works as needed.
  • If there are exceptions to the minimum number, an explanation of why they have not reached this number must be provided.
  • Current is defined as within 5 years of study completion date.
  • Includes grasp of theoretical base/conceptual framework.
  • Refers to the scholarly literature of the broader problem, including the empirical/research aspects of the problem.
  • Includes critical analysis of the body of the literature.
  • It is NOT an annotated bibliography or a series of article summaries.

1.7 Implications
Discusses implications for possible project directions based on anticipated findings of the data collection and analysis.
Comments: (click here) /
  • Includes sometentative directions for the project, but it is the outcomes/findings of the research that must inform decisions about development of the project.
  • Tentative language needs to be used such as may, might, possible
  • Must NOTbe stated as definite: “A desk manual for teachers will be created.”

1.8 Section 1 ends with a transition statement that contains a summary of key points of the section (for the proposal) or study (for the final paper) and an overview of the content of the remaining sections.
Comments: (click here) /
  • A transition must be included.

Section 2: The Methodology (Proposal)
(Ed.D. Project studies)
Quality Indicators /
Overview
  • Project Study Section 2, Methodology, is similar to Doctoral Study Section 3 at the proposal stage.
  • For the final Project Study, the findings are added to Section 2 (comparable to Section 4 in the Doctoral Research Study).
  • All Project Study proposals MUST include original research or evaluation.
Please note the following concerns:
  • If quantitative design: Valid, reliable measures are needed.
  • If qualitative design: Case study design will generally be used. Methods to address quality/validity are needed.
  • If a program evaluation design: Identify type of program evaluation design and cite sources.
  • Program Evaluation and Policy Analysis are considered types of analytic models for conducting research

QUANTITATIVE DESIGN
2.01 Research Design and Approach, to include:
  1. Introduction describes the research or evaluation design and approach,
  2. a justification for using the design and approach, and
  3. an explanation showing how the design derives logically from the problem
If the research question involves an evaluation, the research design and approach include:
  1. The description of the type of evaluation that will be conducted (goal-based, outcomes based, formative or summative)
  2. Justification for using this type of evaluation
  3. The overall goal(s) for a goal-based evaluation
  4. The outcomes and performance measures that will be utilized as indicators (for an outcomes-based evaluation)
  5. The overall evaluation goals.
Comments: (click here)
2.02 Setting and Sample
Presents description and defense, to include:
  1. the population from which the sample will be or was drawn,
  2. the sampling method used,
  3. the sample size (support by using a power analysis or a reference to a statistical text)
  4. the eligibility criteria for study participants, and
  5. the characteristics of the selected sample.
Comments: (click here)
2.03 Instrumentation and Materials
Presents descriptions of instrumentation or data collection tools to include:
  1. name of instrument,
  2. type of instrument,
  3. concepts measured by instrument,
  4. how scores are calculated and their meaning,
  5. processes for assessment of reliability and validity of the instrument(s) and the results of those processes published in previous studies (test-retest reliability, internal consistency, validity),
  6. processes needed to complete instruments by participants; (full version of multi-item instrument included as an appendix; single-item measures included in the body of this section.)
  7. where raw data are or will be available (appendices, tables, or by request from the researcher), and
  8. explanation of the data used to measure each variable in the study
Comments: (click here)
2.04 Data Collection and Analysis
Includes an explanation of descriptive and/or inferential analyses to be used in the study, such as :
a.description of data collection processes
b.nature of the scale for each variable (nominal, ordinal, interval)
c.statement of hypotheses related to the research/guiding question, and description of parametric, nonparametric, or descriptive statistics.
Comments: (click here)
2.05 Assumptions, Limitations, Scope and Delimitations
Provides descriptions of:
  1. facts assumed to be true but not actually verified (assumptions),
  2. potential limitations of the study (weaknesses), and
  3. the variables (scope) and the boundaries of the study(delimitations).
These can be presented as a numbered list or in paragraph format.
Comments: (click here)
2.06 If this is an evaluation, limitations of the evaluation are reported
Comments: (click here)
2.07 Measures that will be taken for protection of participants’ rights are summarized, including issues of confidentiality, informed consent, and protection from harm,
Comments: (click here)
QUALITATIVE DESIGN
2.11 Qualitative Research Design and Approach:
  1. Introduction describes how the research or evaluation design derives logically from the problem and research (guiding) question.
  2. Description of the qualitative tradition or research design that will be used.
  3. Justification of the choice of research design, with explanations why other likely choices would be less effective.
If this is an evaluation, the design and approach also include:
  1. The description of the type of evaluation that will be conducted.
  2. Justification for using this type of evaluation
  3. The overall goal(s) (for a goal-based evaluation)
  4. The outcomes and performance measures that will be utilized as indicators (for an outcomes-based evaluation)
  5. The overall evaluation goals
Comments: (click here)
2.12 Participants:
Provides description and justification to include:
  1. Criteria for selecting participants
  2. Justification for the number of participants, balanced with depth of inquiry. (In general, the fewer the participants the deeper the inquiry per individual.)
  3. Procedures for gaining access to participants
  4. Methods of establishing a researcher-participant working relationship.
  5. Measures for ethical protection of participants: measures taken for protection of participants’ rights are summarized including issues of confidentiality, informed consent, and protection from harm.
Comments: (click here)
2.13 Data Collection
Provides description and justification to include:
  1. Choices about which data to collect are justified.
  2. Data to be collected are appropriate to the type of evaluation and to the qualitative tradition chosen.
  3. A specific plan around the number and anticipated duration of interviews/observations/focus group sessions.
  4. How and when the data are to be collected and recorded.
  5. The process by which the data will be generated, gathered and recorded is described.
  6. The systems for keeping track of data and emerging understandings (research logs, reflective journals, cataloging systems).
  7. Procedures for gaining access to participants.
  8. The role of the researcher (including past/current professional roles at the setting, past/current professional relationships with the participants, how these past/current roles and relationships are likely to affect data collection, and the researcher’s experiences or biases that are related to the topic.)
Comments: (click here)
2.14 Data Analysis:
Provides discussion and description, including:
  1. How and when the data will be analyzed
  2. If a software program will be used in the analysis,
  3. The coding procedure for reducing information into categories and themes
  4. Evidence of quality and procedures to assure the best possible accuracy and credibility of the findings. (e.g. member checks, triangulation, peer debriefing, clarifying researcher bias, etc.).
  5. Procedures for dealing with discrepant cases.
(a)Comments: (click here)
2.15 If this is an evaluation, limitations of the evaluation are reported
Comments: (click here)
MIXED METHODS DESIGN
2.21 Mixed Method Research Design and Approach:
Provides explanation to include:
  1. Introduction describes the intent of mixing qualitative and quantitative data in a single study and identifies the qualitative and quantitative components
  2. The strategy for data collection (concurrent, sequential, or transformative)
  3. The multiple forms of data collection and analysis
  4. Justification for using the design and approach
  5. When and where (data collection, analysis, interpretation) the integration of the approaches will occur.
If this is an evaluation, the mixed method design and approach also include:
  1. The description of the type of evaluation that will be conducted (goal-based, outcomes based, formative or summative)
  2. Justification for using this type of evaluation
  3. The overall goal(s) for a goal-based evaluation)
  4. The outcomes and performance measures that will be utilized as indicators (for an outcomes-based evaluation)
  5. The overall evaluation goals
Comments: (click here)
2.22 Setting and Sample
Provides description and justification to include:
  1. Population from which the sample will be or was drawn,
  2. Sampling method including the sampling frame used,
  3. Sample size,
  4. Eligibility criteria for study participants, and
  5. Characteristics of the selected sample
Comments: (click here)
2.23 The context and sequential or concurrent strategies
Provides description of strategies, to include:
• For the qualitative sequence:
  1. Procedures for gaining access to participants
  2. A specific plan around the number and anticipated duration of interviews/observations/focus group sessions
  3. Methods of establishing a researcher-participant working relationship
  4. Data triangulation is built into the data collection and analysis
  5. Role of the researcher in the data collection process. (including past/current professional roles at the setting, past/current professional relationships with the participants, how these past/current roles and relationships are likely to affect data collection, and the researcher’s experiences or biases that are related to the topic.)
• For the quantitative sequence of the design:
a.Descriptions of instrumentation or data collection tools to include:
  1. Name of instrument and/or data to be collected
  2. Type of instrument and/or data to be collected
  3. Concepts measured by instrument and/or data to be collected
  4. How scores/responses are calculated/rated and their meaning
  5. Processes for assessment of reliability and validity of the instrument(s)
  6. Processes needed to complete instruments by participants
  7. Where raw data are or will be available (e.g., appendixes, tables, questions, or by request from the researcher)
b.b. Explanation of the data used to measure each variable in the study
Comments: (click here)
2.24 Data analysis and validation procedures:
Presents description and explanation to include:
  1. Analysis procedures within the design
  2. Analysis which occurs within the quantitative approach and the qualitative approach, and/or between the two approaches
  3. Validity and trustworthiness of both the quantitative data and the qualitative findings
  4. Procedures for integration of qualitative and quantitative data.
  5. Integration of the findings from the two approaches
Comments: (click here)
2.25 Measures taken for protection of participants’ rights are summarized including issues of confidentiality, informed consent, and protection from harm.
Comments: (click here)
2.26 If an evaluation, the limitations of the evaluation are reported clearly
Comments: (click here)
Section 2: The Methodology (Final)
(Ed.D. Project studies)
Quality Indicators /
Overview
Findings are added to Section 2 (comparable to Section 4 in the Doctoral Study).
Before the student writes Section 3--The Project, the student and committee must meet to discuss the findings and the most appropriate project based on the findings.
Committee approval is required before the student may proceed to write Section 3--The Project.
Quantitative Results
2.01. Measures obtained are reported clearly, following standard procedures.
Comments: (click here)
2.02. Data analyses
  1. Overall, data analysis (presentation, interpretation, explanation) is consistent with the project and its underlying theoretical/conceptual framework
  2. Do not contain any evident statistical errors.
Comments: (click here)
2.03. Tables and Figures
  1. Self-descriptive, informative, and conform to APA format
  2. Referred to within the text included in the section
  3. Comments are immediately adjacent
  4. Properly titled (APA)
  5. Show copyright permission if not in public domain
Comments: (click here)
2.04. Outcomes are logically and systematically summarized and interpreted in relation to the problem and research (guiding) question. The project as an outcome is described.
Comments: (click here)
Qualitative Results:
2.11 The process by which the data were generated, gathered, and recorded is clearly described.
Comments: (click here)
2.12Data analyses
  1. The systems used for keeping track of data and emerging understandings (research logs, reflective journals, cataloging systems) are clearly described.
  2. The findings build logically from the problem and/or the project.
  3. Patterns, relationships, and themes described as findings are supported by the data. All salient data are accounted for in the findings. Any outlying or disconfirming data are included and explained.
  4. A discussion on Evidence of Quality shows how this study followed procedures to address accuracy of the data (e.g., member checks, triangulation, etc.). Appropriate evidence is included in the appendixes (sample transcripts, researcher logs, field notes, etc.).