Critical Appraisal 1
Critical Appraisal of a Meta-analysis
Jill Radtke
University of Pittsburgh
Worksheet for Critical Appraisal of Systematic Reviews/Meta-Analyses
Citation:
Martin, R.M., Gunnell, D., Owen, C.G., & Smith, G.D. (2005). Breast-feeding and
childhood cancer. International Journal of Cancer, 117(6), 1020-31.
What type of article is this (e.g., research/data-based, clinical paper, review, editorial)?
Research/data-based
If this is a research/data-based article, what makes it this type of article? Identify 2-3 key characteristics of the article.
- Methodology: The article/study seeks to obtain data in a systematic fashion (e.g., the comprehensive, systematic search for articles; the choosing of articles for inclusion, based on criteria determined a priori).
- Style: The article’s results and discussion are presented in an objective and frank manner (also discussing the limitations) in order that the reader may judge, implement, question, and/or disregard the evidence.
- Original Findings: The study is a meta-analysis, combining results of many individual studies for new statistics that offer a more comprehensive answer as to whether breastfeeding is associated with a decreased risk of childhood cancers.
State the research question posed by the authors.
In epidemiologic studies examining the association between breastfeeding and childhood cancer, selected through systematic review for this study, what is the association between any breastfeeding and childhood cancer across studies, as compared to the never-having-been-breastfed child’s development of childhood cancer, and are these associations different depending upon type of cancer?
What is my clinical question?
In breast milk samples taken weekly in an industrialized area from postpartum women exclusively breastfeeding starting two days after birth and lasting until 26 weeks postpartum, is there any biochemical marker or element, in presence, or concentration, associated with an increased rate of childhood cancer development by 5 years of age in these women’s offspring, as compared to a similar group of breastfed children in a rural area?
Using PICO, identify the following if applicable:
P (= population): Children aged 0-19 whose cancer status/mortality was known, and whose breastfeeding status as an infant could be classified as “never breastfed” or “any breastfeeding”
I (= intervention): Breastfeeding
C (= comparison group): Children never having been breastfed
O (=outcome): Childhood cancer development
Appraisal Guide / COMMENTSI. Are the results valid/trustworthy?
1. Are the specific aims/research questions clearly focused for this systematic review/meta-analysis?
a) What is being reviewed?
b) What is the population?
c) What are the independent variables (i.e., factors, exposures, interventions)?
d) What are the dependent variables (i.e., outcomes responses)?
e) Were all clinically relevant (patient-important) outcomes considered? / The main research question, whether breast milk is associated with a decrease in the risk of childhood cancer, is clear both in the abstract and introduction. Additionally, the specific aims, to determine “the consistency of associations [between childhood cancer development and breastfeeding] across studies and to determine whether associations differed by cancer types” is clear in the introduction. However, the types of childhood cancer the authors are looking at and how breastfeeding is being classified (never vs. ever been) is not clearly delineated. Rather, we are given this information in pieces throughout the article. This may be appropriate for a meta-analysis, as the type of articles included should provide more direction and specificity for specific aims. Nevertheless, giving this information up front would have made the article more readable.
This meta-analysis reviewed the Medline database from 1966 (inception) until June 2004 for articles, papers, letters, abstracts and review articles pertaining to infant feeding and cancer. They also employed the auto-alert system and reviewed relevant reference lists (manually searched). In the end, of 1,415 “hits,” 46 articles were selected as relevant. Of these 46, 26 were selected for study inclusion (the discarded studies were discarded for a variety of reasons, including that some articles were commentary or review articles or did not focus on childhood cancer). Of these 26 articles, 2 were cohort/nested case-control studies, and the rest were case-control studies. All examined the association between childhood cancer and breastfeeding. The authors reviewed the individual studies for quality and characteristics, as shown in the tables in the article. The types of childhood cancers reviewed in the 26 studies included leukemia, acute lymphoblastic leukemia, acute nonlymphoblastic leukemia, Hodgkin’s disease, non-Hodgkin’s lymphoma, central nervous system cancers, neuroblastoma, malignant germ cell tumors, juvenile bone tumors, and other solid cancers. The meta-analysis looked at the associations of these cancers with the classification of never having been breastfed as compared to any or exclusive breastfeeding. The authors employed pooled odds ratios to determine if any breastfeeding was associated with any of the childhood cancers mentioned above, as well as childhood cancer in general. They also looked at heterogeneity for between study variation among many factors, chosen a priori, that might have impacted study results (i.e., study size, study design) using Cochran’s Q-statistic and the I2 statistic. If too much heterogeneity were found, it would not have been meaningful or possibly even valid to pool the odds ratios between the studies. The authors also looked at breastfeeding specifically for greater than 6 to 8 months as compared to never having been breastfed to determine any impact of prolonged breastfeeding on childhood cancer. Finally, the authors looked at the Egger test for small study bias, in order to determine if the small studies utilized in the review arrived at exaggerated effect sizes.
The actual populations used in the individual articles included children from 0-19 years whose breastfeeding status could be classified as “ever” or “never.” Additionally, their development of childhood cancer or mortality from cancer was known.
The independent variables are the child’s breastfeeding classifications (e.g., never, ever, greater than 6-8 months); this includes the duration of breastfeeding. Age of the child might be considered an independent variable, but year of birth is similar, and this was considered a possible confounder in the study.
The dependent variable was the rates of childhood cancer: i.e., the number of children with each type of cancer.
I believe the relevant outcome is considered in the context of this study’s goal. We are interested in childhood cancer development and its association with breastfeeding in order to ascertain whether breastfeeding should be recommended to mothers for this potential benefit. The authors addressed odds ratios of childhood cancer development for each major type of childhood cancer and childhood cancer in general when any or no breastfeeding occurs. They found that breastfeeding’s effect on childhood cancer is negligible, but should be recommended anyway for its other known benefits. Truly, the most important outcome here is cancer development, in order to establish or negate the relationship between childhood cancer and breastfeeding, but other outcomes, such as mortality and age at diagnosis of cancer and its association with breastfeeding, might have been considered.
2. Is the search for studies comprehensive (i.e., did the authors look for the appropriate sort of studies)?
a) What bibliographic databases were used?
b) What years were searched?
c) What languages were searched? d) Were the study inclusion criteria appropriate (i.e., study design, participants, intervention, and outcomes of interest)?
e) Was the inclusion process discussed?
f) Did at least 2 individuals review articles for selection?
g) What process did they use to reach agreement on article selection?
h) Was agreement achieved?
i) Were all relevant studies included in the review?
j) Were any important studies missed? Explain. / The Medline database was used, combining keywords for infant feeding and cancer. Additionally, auto-alerts were employed on Medline, and the authors manually searched the reference lists of eligible studies and previous meta-analyses for the specific cancers.
Medline was searched from its inception in 1966 to June 2004.
It was not mentioned what languages were searched.
Yes, the study inclusion criteria appeared appropriate. No specifications of specific study design were mentioned under “Material and methods,” however it was noted that several studies were excluded because they were reviews and commentaries. Thus, we see that the authors are ensuring the quality of the meta-analysis by including studies that are primary sources, and excluding secondary sources. The age criterion, 0-19 years, seems appropriate, as we are studying “childhood” cancer. Finally, the meta-analysis criterion included studies comparing those ever breastfed to those never breastfed, and estimates of the association of having ever been breastfed to cancer outcomes had to be able to be derived or available in the study. This is appropriate, given the main objective of our study: to determine the association of breastfeeding with the development of childhood cancer.
The inclusion process was discussed, as noted above, under “Material and methods,” and reasons for excluding certain studies were also discussed under “results” on page 1021.
It was not mentioned how many individuals reviewed the articles for selection. (Thus, “g” and “h” are n/a.)
There may be some relevant studies not included in the review. The authors mentioned that they had to exclude studies matching their inclusion criteria because they were already mentioned in their article previously. Additionally, they had to exclude some studies because estimates of the odds ratios could not be derived. This does not mean, however, that these studies did not offer valuable insight and evidence for our research question. For this particular meta-analysis, though, they did not fit. Additionally, the authors might have tried to search other databases, confer with “experts” in the area to uncover unpublished sources of information, search other languages, etc. It was not mentioned if these techniques were employed or not, but they might have contributed relevant and valuable data to our meta-analysis.
Additionally, I searched OVID Medline for the same study years as the article’s authors used, using the keywords, “infant feeding” and “cancer.” I had 15 “hits”, but I did not find any relevant articles fitting the study’s inclusion criteria that the authors did not use in their systematic review. However, one study uncovered did seem to offer an important opposing viewpoint. It looked at breastmilk samples’ contamination with heavy metals and how this can contribute to cancer in later life for infants who breastfeed(Lutter, Lyengar, Barnes, Chuvakova,Kazbekova, & Sharmanov, 1998). While not meeting study inclusion criteria completely, it does make an important and valid point about environmental influence on breastmilk and subsequent cancer development in offspring. In my opinion, this should be considered in future studies looking at childhood cancer risk associated with breastfeeding.
3. Is the validity of the included studies adequately assessed?
a) What criteria were used to evaluate validity of individual studies (e.g., use of intention-to-treat, if an intervention study; efforts to obtain missing information; efforts to minimize publication bias; assessment of methodologic quality)?
b) Were these criteria appropriate? Explain.
c) Did at least 2 reviewers examine individual studies for validity?
d) What process did reviewers use to reach agreement on the validity of individual studies?
e) Was agreement achieved? / The article mentions that, unlike randomized control trials, there are no widely accepted criteria used to judge the validity and quality of observational studies (which this meta-analysis used). Therefore, the study employed quality criteria chosen a priori from previous meta-analyses and systematic reviews examining infant feeding. They seem to be appropriate, as they are chosen a priori. However, the model criteria were derived from articles whose titles implied they were looking at the association between breastfeeding and indicators of overweight. We are examining the relationship between breastfeeding and childhood cancer. So, although the criteria may be easily transferable to our meta-analysis, we cannot be sure. It would have been appropriate to also consider criteria from systematic reviews based on childhood cancer.
Some of the criteria used and accounted for by the authors included study size (less than or equal to 500, or greater than 500—cited by the authors as the number used for effect size in meta-analyses for breastfeeding), study design (cohorts considered more robust than case-control), whether breastfeeding was the main outcome measure, whether maternal recall was used to assess breastfeeding after 1 year (memory problems), early exposure to infectious disease in the child (because this has been associated with breastfeeding and cancer, according to the authors), whether effects estimates controlled for social factors in the child or adult, reproductive factors, if the study was population-based (potential for selection bias), if the control or case response was less than 80% (selection bias), whether prevalence of breastfeeding was at least 70% (authors did not justify this criteria), the region the study was done in (as different views of breastfeeding, different cancer rates in different regions), and the year of birth of the child participants (as breastfeeding is viewed differently in different generations, and formula has changed dramatically over the years). This information can be found on p.1026, Table II.
Throughout the analysis/results, several studies are mentioned by the authors as having limited validity for one reason or another, and the analysis is done with and without these studies for this reason. For example, one study was only given only in abstract form and was thus considered poor quality for the meta-analysis. Removing it changed the association between Hodgkin’s disease and breastfeeding quite significantly (p.1026).
It was not mentioned whether more than one reviewer arrived at the validity criteria or assessed the studies for validity.
4. What are the study characteristics?
a) How many individual studies were included?
b) What types of studies were included (e.g., randomized controlled trials, case-control studies, cohort studies, cross-sectional studies)?
c) What were the sample sizes for the studies? If applicable, what are the sample sizes for the study groups?
d) In what countries were the studies conducted?
e) If an intervention was the primary independent variable under investigation: What was the duration of treatment?
What was the adherence to the study treatment?
Were adverse results reported? / 26 individual studies were included. These included 2 cohort/nested case-control studies, and 24 case-control studies.
The study sample sizes were in a wide range. Some studies looked at individual cancer types and gave number of cases with each type. Some studies looked one cancer type or childhood cancer in general and gave the total sample size. The smallest sample was composed of 33 subjects and looked at childhood cancer in general. The largest study included over 3,000 subjects, and broke down childhood cancer into several specific cancer types (and included number of subjects with each type of cancer).
The studies were collected from a wide variety of countries and areas, including among them, the United States, England, Ireland, Greece, Australia, Italy, United Kingdom, United Arab Emirates, Shanghai, Canada, New Zealand, etc. The meta-analysis was very diverse in study location.
The “intervention” was breastfeeding, and the classifications used in the meta-analysis were “ever” or “never” breastfed. Additionally, the authors did a second small analysis comparing “never” breastfed to breastfed greater than 6-8 months. This was the only reference to duration of breastfeeding. We are not told how long those who “ever” breastfed nursed. Because these studies were observational in nature, breastfeeding was not classified whether it was “adhered” to. Instead, retrospectively, these children’s parents classified whether they did or did not ever breastfeed.
Although some studies showed no significant decrease in risk for certain childhood cancers associated with breastfeeding, it was not noted that breastfeeding was ever positively associated with childhood cancer significantly.
5. What are the results of systematic review/meta-analysis?
a) Were the results in terms of effect sizes of the individual studies clearly reported?
b) Were these results consistent from individual study to individual study? Explain.
c) Were the results of the different studies similar (e.g., Were tests of homogeneity conducted? What did they show?)
d) Were the individual study findings combined statistically? (If results were combined, did it make sense to do so? Explain.
e) If results were not combined, should they have been? Explain.)
f) Were the reasons for study variations discussed? / The odds ratios and 95% confidence intervals (CI’s) for the individual studies werereported for childhood cancer (individual types and overall, depending on study), comparing never breastfed children to breastfed children. This was shown in Figure 1 (p.1027) in the “Results” section, but was difficult to read. Some studies’ odds ratios were discussed individually, but most were not. Additionally, for the “other” cancers in childhood, such as germ cell tumors (in which only one study was used), no odds ratios were provided for any studies. We are simply told that breastfeeding and that cancer are not significantly associated.