2

ED 861 Qualitative Methods Applied to Educational Research

Critique of Qualitative Research Articles: Rubric and Guidelines

prepared by Jane M. Gangi, Ph.D., with input from Drs. Heilbronner and Delcourt

January 20, 2011

In class, Dr. Nancy Heilbronner will join us to review how to critique articles. As a touchstone, we will use:

Angrosino, M. V. (1994). On the bus with Vonnie Lee: Explorations in life history and metaphor. Journal of Contemporary Ethnography, 23, 14-28.

which is an example of Narrative Inquiry. It is recommended and included in Creswell’s (2007) Qualitative Inquiry and Research Design: Choosing Among Five Approaches. On the syllabus, you will see as optional readings Creswell’s other recommended studies, except for the article exemplifying grounded theory, which is required.

While they are discussing how to evaluate studies that might become part of your literature review for your dissertation, Merriam and Simpson’s guidelines are helpful in evaluating articles as well:

·  Is the author of the source an authority on the topic, one who has done much of the empirical work in the area, or one who has offered seminal theory upon which subsequent research and writing has been based? If so, that author’s work will be quoted by others and listed in bibliographies on the topic.

·  When was the article or book or report written? As a rule, the most recent work in an area should be included in a review.

·  What exactly was written about or tested? If a particular resource or research study is highly relevant to your present research interest, it should be included even if the “who” and “when” criteria are not met.

·  What is the quality of the source? A thoughtful analysis, a well-designed study, or an original way of viewing the topic is probably a significant piece of literature. In historical or documentary analysis, the quality of primary and secondary sources is a major criterion for the inclusion into the data base. (as cited in Merriam, 1998, p. 54)

The literature review in your dissertation (Chapter 2) would include only articles deemed high-quality. Your article critique, however, can include articles you might not include in your literature review.

Keep in mind that the purpose of qualitative research is to discover the meanings participants hold—their thoughts, feelings, intentions, experiences, and how they make sense of their world. Qualitative research is analytic and interpretive, and seeks to illuminate. In the case of critical ethnography and some action research, the purpose is also to reveal the way institutions (schools) maintain the status quo.

From Dr. Heilbronner’s Checklist for Critique of an Article (adapted for qualitative research):

ABSTRACT
1.  Is the abstract intelligible?
2.  Does the abstract accurately describe the participants?
3.  Does the abstract include information not presented in the paper?
INTRODUCTION
1.  Did the author(s) indicate why the study was undertaken?
2.  Was the background information provided adequate to understand the aims of the study?
3.  Was the study grounded in a review of the literature?
4.  Were the research questions presented, either in this section or the Methods section?
5.  Does the author state how he or she is qualified to write the study?
METHODS
1.  Did the author(s) outline how the study was conducted, who was in it, and why they were selected?
2.  What type of design was used? (case study, grounded theory, phenomenology, narrative inquiry, and so on)
3.  Was the design appropriate for qualitative inquiry?
4.  Did the author(s) indicate any potential problems with the method?
DISCUSSION AND INTEPRETATION
1.  Were the objectives of the study met?
2.  If the objectives were not met, does the author(s) have an explanation?
3.  Does the author(s) discuss their results in relation to literature in the field?
4.  Is the interpretation and/or analysis supported by the data?
5.  Does the author(s) discuss the limitations of the methods used?
CONCLUSION
1.  Do the conclusions make sense, given the data provided?
2.  Are recommendations for future studies made?
REFERENCES
1.  Does the author(s) cite appropriate papers for the comments made?
2.  Does the author(s) cite their own publications needlessly?
3.  Are the sources published?
4.  Are the sources dated?
5.  Does the author(s) include articles from peer-reviewed journals?

Before reviewing articles, please review this excellent resource, Guidelines for Critical Review of Qualitative Studies, which shows how phenomenology differs from ethnography, differs from case study, and so on. Give evidence of having read the Guidelines in your review.

http://www.usc.edu/hsc/ebnet/res/Guidelines.pdf

ED 861 Qualitative Methods Applied to Educational Research

Critique of Qualitative Research Articles: Rubric

1= Needs Improvement: Information not included or inadequate,

2= Acceptable: Meets Expectations: adequate information,

3= Target: Above expectations, insightful

Needs Work = 1 / Acceptable = 2 / Target = 3
Abstract (3 points possible)
-Critique does not evaluate whether the abstract is intelligible, accurately describes the participants, and whether information is included that is not present in the article. / -Critique somewhat evaluates whether the abstract is intelligible, accurately describes the participants, and whether information is included that is not present in the article. Specific example(s) that support the writer’s viewpoint may or may not be provided. / -Critique evaluates whether the abstract is intelligible, accurately describes the participants, and whether information is included that is not present in the article. Specific example(s) that support the writer’s viewpoint are provided.
Introduction (3 points possible)
-Critique does not examine whether the author(s) indicated why the study was undertaken; whether the background information was adequate; if the study was grounded in a review of the literature; and, how the author is qualified to undertake the study. / -Critique somewhat examines whether the author(s) indicated why the study was undertaken; whether the background information was adequate; if the study was grounded in a review of the literature; and, how the author is qualified to undertake the study. Specific example(s) that support the writer’s viewpoint may or may not be provided. / -Critique examines whether the author(s) indicated why the study was undertaken; whether the background information was adequate; if the study was grounded in a review of the literature; and, how the author is qualified to undertake the study. Specific example(s) that support the writer’s viewpoint are provided.
Methods (3 points possible)
-Critique does not evaluate how the author(s) explain how the study was conducted, and if the type of design was clearly explained.
-Critique does not state the methodology (phenomenology, grounded theory, case study, and so on) and does not evaluate (using the Guidelines) how well the article adhered to the methodology / -Critique somewhat evaluates how the author(s) explain how the study was conducted, and if the type of design was clearly explained. Specific example(s) that support the writer’s viewpoint may or may not be provided.
-Critique somewhat states the methodology (phenomenology, grounded theory, case study, and so on) and somewhat evaluates (using the Guidelines) how well the article adhered to the methodology / -Critique evaluates how the author(s) explain how the study was conducted, and if the type of design was clearly explained. Specific example(s) that support the writer’s viewpoint are provided.
-Critique clearly states the methodology (phenomenology, grounded theory, case study, and so on) and evaluates (using the Guidelines) how well the article adhered to the methodology
Discussion and Interpretation (3 points possible)
-Critique does not determine whether the objectives of the study were met and, if not, whether the author(s) include an explanation.
-The interpretation and analysis is not critiqued, positively or negatively.
-Critique does not whether the author(s) discuss the limitations of the study. / -Critique somewhat determines whether the objectives of the study were met and, if not, whether the author(s) include an explanation.
-The interpretation and analysis is somewhat critiqued, positively or negatively.
-Critique includes somewhat whether the author(s) discuss the limitations of the study.
Specific example(s) that support the writer’s viewpoint may or may not be provided. / -Critique determines whether the objectives of the study were met and, if not, whether the author(s) include an explanation.
-The interpretation and analysis is critiqued, positively or negatively.
-Critique includes whether the author(s) discuss the limitations of the study.
Specific example(s) that support the writer’s viewpoint are provided.
Conclusion (3 points possible) / Conclusion (3 points possible)
-Critique does not determine whether the conclusions make sense, given the data provided, and whether indications of future studies are made. / -Critique somewhat determines whether the conclusions make sense, given the data provided, and whether indications of future studies are made. / -Critique determines whether the conclusions make sense, given the data provided, and whether indications of future studies are made. Specific example(s) that support the writer’s viewpoint are provided.
References (3 points possible)
-Critique does not the author(s)’s use of references (citing appropriate papers; citing their own publications excessively; including published sources; including peer-reviewed articles). / -Critique somewhat evaluates the author(s)’s use of references (citing appropriate papers; citing their own publications excessively; including published sources; including peer-reviewed articles). / -Critique evaluates the author(s)’s use of references (citing appropriate papers; citing their own publications excessively; including published sources; including peer-reviewed articles).
Format (3 points possible)
-Conventions of Standard English are not adhered to
-APA style is not observed / -Conventions of Standard English are somewhat adhered to.
-APA style is somewhat observed / -Conventions of Standard English are adhered to
-APA style is observed

Scale: 19-21=A 17-18=A- 15-16=B+ 14=B 13=B- 12=C+

11=C 10=C- 9=D 8=F

Gangi 2011

References

Creswell, J. W. (2007). Qualitative inquiry and research design: Choosing among five approaches (2nd ed.). Thousand Oaks, CA: Sage Publications.

Merriam, S. B. (1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass.