Dissecting Research Articles

This activity focuses on developing the thinking skills associated with becoming critical consumers of research literature. Through these learning experiences, you will be able to identify key research concepts and utilize these research concepts to begin evaluating research articles more critically. These critical thinking competencies are directly applicable to your coursework and to each aspect of the research process that you will continue to encounter as you advance in your program of studies.

Residency Competencies

These activities directly support the following competencies:

Apply advanced critical thinking skills to research methods and design.
Support ideas and concepts with evidence from the literature in proposing research methods and design.
Develop research questions that align with an identified problem.
Justify approach to research problem by analyzing and applying current research literature.
Analyze the various methods available for data analysis.
Apply ethics and academic integrity in research.

Appendix Resources

Throughout this activity, you will be using two research articles. They will be referred to as Article 1 and Article 2. Article 1 is an example of a quantitative research study and Article 2 is an example of a qualitative research study. You will find links to the articles in the courseroom activities, Dissecting a Quantitative Article and Dissecting a Qualitative Article.

Objectives

Upon completion of these activities, you will be able to:

Assess the problem statement and research questions articulated in quantitative and qualitative research studies.
Assess how well literature reviews familiarize readers with research topics and provide information about how the studies fill a gap in the knowledge base.
Assess the research designs in quantitative and qualitative research studies.
Assess the sampling strategies and the process for recruiting and selecting participants in quantitative and qualitative research studies.
Evaluate authors' findings in quantitative and qualitative research studies.
Assess the implication and application sections of quantitative and qualitative research studies.

Identifying a Peer-Reviewed Journal

When evaluating research articles, there may be varying opinions on the credibility of the evidence presented. A beginning step for graduate learners is to determine if the article under scrutiny is a peer-reviewed research article.

Peer review refers to the formal process that determines whether an article will be published. A journal is peer reviewed or refereed when a group of reviewers and an editor read and evaluate manuscripts (Beebe, 1993).
According to the Publication Manual of the American Psychological Association (2010), peer review is conducted by a panel of scholars in the subspecialty of the submitted article. The peer review process focuses on the quality of a submitted manuscript in regards to the work being "original, valid, and significant” (p. 225).

Although one could make a case for only using peer-reviewed journals, there are many important published studies that have not gone through the rigorous process of peer review. Therefore, a graduate learner needs to understand how to judge the credibility of research studies.

Developing Your Research Skills

Knowing how to evaluate research does not happen naturally. Throughout life, individuals make decisions about truth based on nonscientific methods. Examples of nonscientific evidence are opinions, value judgments, personal experiences, unsupported assertions, and second-hand reports. Therefore, learning to distinguish between nonscientific evidence and scientific evidence is a learning process that requires self-awareness and practice. As a graduate student, recognizing nonscientific methods and learning how to apply critical thinking skills to the dissection and evaluation of research articles is an essential tool for your educational success.

In this activity, obtaining a comprehensive understanding of quantitative and qualitative research methods and data analysis is not the objective. The focus is on helping you achieve a basic understanding of key research concepts that form the foundation of the research process. This activity will help you to develop the foundation of knowledge and understanding that you will continue to build upon as you complete coursework in your doctoral program.

You will apply your knowledge of the research process when reading research articles for your courses and comprehensive exam papers, and when designing your dissertation proposal. As you continue in the transformational process, you will become increasingly sophisticated in critically evaluating the claims of authors in research articles, and be able to separate out scientifically verified claims from unsubstantiated claims. "Research can be regarded as a process of asking a question (or a related series of questions) and then initiating a systematic process to obtain valid answers to that question" (Meltzoff, 2001, p. 13). A research article describes the research design and the results of the study conducted.

Components of Research Articles

Abstract — A 120-150 word summary of the entire article. It captures the essence of the entire article.
Introduction (Background) — An introduction sets the backdrop of the specific research topic and the study, and identifies the research question(s).
Literature Review — This section provides a summary of the literature on key themes of the specific research topic. It also gives readers a sense of what has been written in the field about the specific research topic.
Methodology — This section is a detailed description of how data was collected. For example, it describes the overall blueprint of the study (research design), how the research participants were recruited (sampling), what measurements or instruments were used, and how the data was collected once the participants were recruited (data collection procedures).
Data Analysis — This section explains how the data was analyzed.
Findings — This section is a write up of the findings. After the researchers collect the data, they write up the findings or the results.
Discussion — In this section, the researchers interpret the results that were presented in the previous section and the implications of the research findings.
References — In this section, the articles and literature used throughout the article are cited.

In activities Dissecting a Quantitative Article and Dissecting a Qualitative Article, you will evaluate each of these components of a research article. The activity documents will identify the evaluation questions you will ask. For now, remember that the key things to look for when evaluating research are:

1.Whether the terms and methods are clearly described.

2.Whether the literature review shows a need for the study and the background of the methodology.

3.Whether the methods and procedures used were consistent with the literature and with the research question.

4.Whether the findings—including their implications—are clearly presented.

5.Whether there are any ethical challenges in the study that may have affected the results.

Abstract, Introduction, Hypothesis, Research Question, and the Literature Review

The first step to dissecting and evaluating a research article is the review for clarity and consistency. The abstract, the introduction, the problem statement and the research hypothesis or research question should not only be clearly stated, they should be consistent. The reader should be able to understand the relationship between each component. These components set the stage for the rest of the document.

Quantitative Methodology – Research Design, Methods and Procedures, and Sampling

The method section describes in detail how the study was conducted. The evaluation of the methodology consists of reviewing the research design, how the research participants were recruited, and the data collection procedures. Different research methodologies have different reporting requirements. The evaluation of a research article will vary somewhat depending on whether the research article is quantitative or qualitative. Article 1 is a quantitative study.

According to Creswell (2003, p. 18), "A quantitative approach is one in which the investigator primarily uses positivistic claims for developing knowledge (i.e., cause and effect thinking, reduction to specific variables and hypotheses and questions, use of measurement and observation, and the test of theories), employs strategies of inquiry such as experiments and surveys, and collects data on predetermined instruments that yield statistical data. The goal of quantitative research is to support or disprove the hypotheses."

Quantitative research designs attempt to find relationships between and among variables. The essence of quantitative research studies involves numbers and measurement. So the data collected involves numbers and, ultimately, statistics are used to analyze those numbers. Quantitative research is aimed at verification. Examples of quantitative designs are:

Experiment — An attempt to determine a cause-and-effect relationship. It involves the manipulation of an independent variable and measures the effect(s) on the dependent variable(s).
Quasi-experimental design — A research design used when you cannot randomly assign your participants to the groups, but the researcher does manipulate an independent variable and measure the effect(s) on the dependent variable(s).
Non-experimental design — A research design used when one is interested in measuring the difference in some variable between a number of groups, whether one or a combination of variables predicts a certain outcome (another variable), or the relationships (correlation) between two or more variables.

In order to dissect and evaluate research articles, there is a need to have knowledge of research methodology. Being able to critique a research study is not based on your personal beliefs or ideas, but on the scientific soundness of the study. As you progress through your graduate program, you will develop the skills necessary to critique a research article. As you continue to read research articles, the following questions will guide you in developing your skills for critically reading and evaluating research articles.

Variables in Quantitative Studies

Successful evaluation of a research article, in the case of quantitative studies, requires a basic understanding of variables—what they are and how they are described. Variables are specific concepts that are being studied. They vary along a continuum in terms of its characteristics (continuous variables) or they have at least two categories (categorical variables). For example, let us say the study is to examine gender differences in levels of morality. Gender is one variable in the study, and the other variable is frequency of exercise.

Gender has two categories: (1) male and (2) female.

Frequency of exercise can vary along a continuum. In this case, let us say frequency scores can range from zero to 10 times per week, with higher scores reflecting higher frequency of exercise.

In some studies, the variables can be divided into two categories: independent and dependent variables. The independent variable causes or influences some change in the dependent variable. The dependent variable is the outcome variable and must be measurable. You can use this formula to help you differentiate between the independent and dependent variable:

X → Y

X influences Y where X is the independent variable (IV) and Y is the dependent variable (DV). For example, a researcher might ask whether a particular intervention causes a desired result. The intervention would be the IV and the desired result the DV.

In the case above, gender might be considered the independent variable and frequency the dependent variable, but since gender cannot really be a cause of something like one's exercise regime, most methodologists do notuse the term IV and DV in such cases, preferring terms such as criterion variable and outcome variable.

An important class of variables is named extraneous variables. An extraneous variable can create problems for the researcher. Extraneous variables are undesired variables that can influence the dependent variable and change or invalidate the results of an experiment. For example, suppose a researcher is investigating whether a particular training experience improves performance on some task. She assigns participants from an organization randomly to either the experimental group or the control group. Training is then provided to the experimental group, but not the control group. But the fact that some of the participants may have more experience with the task than others could be an extraneous variable—it could confuse the results because the more experienced participants score higher, not as a result of the training, but as a result of their having more experience.

Extraneous variables are the chief reason for controlled studies—studies designed to reduce or eliminate the influence of extraneous variables.

Qualitative Methodology – Research Designs, Methods and Procedures, and Sampling

Creswell (2003, p.18) describes qualitative research as follows:

"…a qualitative approach is one in which the inquirer often makes knowledge claims based primarily on constructivist perspectives (i.e., the multiple meanings of individual experiences, meanings socially and historically constructed, with an intent of developing a theory or a pattern)…The researcher collects open-ended, emerging data with the primary intent of developing themes from the data."

Essentially, qualitative research provides descriptions of experiences in the participants' own words, which are then analyzed and interpreted by the researcher. Put the simplest way, where quantitative data are numbers, qualitative data are words.

The goal of qualitative research is to gain an understanding of the experiences of the individuals who participate in the research study. Qualitative research is aimed at discovery, understanding, meaning, and developing knowledge from the perspective of first person accounts of the experience under inquiry. Examples of qualitative research methodologies are:

Phenomenology concentrates on the study of phenomena as experienced by the individual, with the emphasis on exactly how a phenomenon reveals itself to the experiencing person in all its specificity and concreteness. The participant experiencing a phenomenon is required to attend to it exactly as it appears in consciousness, without prejudgment, bias, or any predetermined set or orientation. A prominent expert in phenomenological research is Amedeo Giorgi (2009).
Grounded theory is a descriptive research approach that attempts to develop theories of understanding of processes based on data from the real world. The ultimate goal of this approach is to derive theories that are grounded in the data collected. Kathy Charmaz (2006) is an emerging leader in grounded theory research.
Case study is an in-depth description, using many kinds or sources of data, of a single research subject - a program, an event, an activity or group of individuals. Stake (1995) is a good introduction to qualitative case study research.
Ethnography is a descriptive research approach that involves becoming a part of the culture you study. This model is based in the anthropological tradition of research. In this approach, the researcher would spend a long time becoming immersed in the “culture” of the population being studied.
Heuristics—a research model that places special emphasis on knowing through the self, by becoming one with the topic and experiencing it, as it exists in the world. Eric Craig (1978) defines heuristics in The Heart of the Teacher as: "A private discovery oriented approach to understanding how individuals experience themselves and their world" (p. 22). A foremost proponent of heuristic research is Clarke Moustakas (1990). Note that not all schools or departments approve heuristic research.
Generic or basic qualitative inquiry—basic qualitative methodology—an approach to eliciting opinions, attitudes, beliefs, or descriptions of experiences that is oriented toward issues external to the participants. An open-ended subjective opinion survey about people’s political beliefs would be an example. Caelli, Ray, and Mill (2003) provide an excellent introduction to generic qualitative inquiry.
Other qualitative designs are approved by different schools. To find acceptable qualitative designs, click your school's link on the iGuide PhD page, expand the School of Public Service Leadership heading and then view the "SoPSLAcceptable Methods Document."

When presenting the results of a qualitative research study, there needs to a clear and detailed description that includes direct quotes from data collected from the participants. One way in which the quality of the research results can be judged is by the reaction of the reader. Are the results believable? The reader should be able to say, "I understand," or "That makes perfect sense." The presentation of the data should convince the reader of the credibility of the data analysis.

The results section of a qualitative study is a written presentation that includes the voices of the participants, the reflections of the researcher and an interpretation of the answer to the research question and problem (Creswell, 2007).

"A weak or unconvincing analysis can also stem from a failure to provide adequate examples from the data... The researcher needs to make sure that their interpretations and analytic points are consistent with the data extracts" (Braun & Clarke, 2006, pp. 94–95).

Analysis, Findings, Discussion, and Ethics

When evaluating the research findings, first look for the scientific research evidence. Examples of nonscientific evidence are opinions, value judgments, personal experiences, unsupported assertions, and second-hand reports.

When you are dissecting the findings of the study or what the researcher wants you to believe, how do you decide what evidence to believe? The following questions are helpful:

What is the proof?
Where is the evidence?
How do you know that is true?
Why do you believe that?
Can you prove it?
Is the research evidence relevant to the key points?
Does the researcher draw the correct conclusion from the evidence?
Is information missing?

Researchers are susceptible to confirmation bias, meaning that there can be a tendency to find one's personal beliefs confirmed by the data. This is true of both quantitative and qualitative research. When evaluating a research article it is important to be aware of possible confirmation bias, and of your own personal beliefs, so that you can objectively decide whether the results have been reasonably presented and interpreted.

The results section must thoroughly explain how the data was collected and what statistics were used to analyze the data. The results must be described in enough detail to justify the conclusions.