Tips of the Trade: Developing High Quality Research Syntheses

Online Workshop

Tips of the Trade: Developing High Quality Research Syntheses

Presenters: Chad Nye and Oliver Wendt

Part 2: Research Synthesis Process

(Slide 24) OK, so the rules of research synthesis, it’s got to be transparent, documentable, and replicable. In laymen’s terms, it means anything that you do, I ought to be able to do it and get the exact same result. Now I might interpret it differently, but I get the same result. It’s that documenting transparency, replicability that sort of adds an underpinning of credibility to the end evidence, the end result. It also helps us to focus in on the, “what is it that we’re doing that needs to be understood for what’s known, what’s not known, what needs to be done,” that sort of thing.

(Slide 25) So, if we look at the steps, there are several products here, a couple of products here that we can deal with, different kinds of reviews. One is a scoping review. I think, in the reference list that we sent you, I listed a scoping review completed, one or two of them.

(Slide 26) A scoping review has a couple of functions, or differences from a more involved review. Kind of think of it as the idea of, “is there enough information here, that I want to spend more time and resources, in understanding the nature of the treatment, the intervention?” So, we’re kind of looking at the big picture. We’re at the 30,000 feet level kind of thing. We’re probably not overly interested in trying to extract quantitative information. We’re trying to say, “is there enough literature here, that has been recognized someplace as important in a given discipline or content area?” Can we then turn that into something that will guide us in the development of a more substantive outcome?

(Slide 27) So, two or three things here that are common to the exploratory side of this. Sometimes it’s the literature mapping. Finding that literature and knowing where it looks like it comes from. Sometimes conceptual mapping. You’re looking for the vocabulary that’s used to describe the process. I’m working on one now having to do with personal health records, and electronic medical records, as used by families of children with autism. What I’m really struggling for here, is what other terms go with “public health record?” We’re all going to get this, folks, if you didn’t know that. By next year, Obamacare is going to have physicians, have to have electronic records of patients. In 2016, physicians who receive Medicare or Medicaid funding, have to have patient health records. The two right now don’t have to necessarily meet, but that’s where this is driving me. I don’t know the discipline, I don’t know the literature. So I’m in the process now of trying to figure out how I can find the terminology that will get me to those kinds of studies that have to do with the development, use and application of electronic medical records.

Sometimes scoping reviews are done, and maybe more often, for policy purposes. Somebody wants to know, does street lighting make a difference in reducing crime? They don’t what to go through and calculate how many murders occur or didn’t occur or how many burglaries occur or didn’t occur and that sort of thing. It’s largely done through government agency documents and that sort of thing. That may serve as the basis for then advancing the idea to the stage of developing a real review, to access whether or not certain interventions and methodologies work.

So lastly, it’s summarizing research. It is a systematic review, but it is not a meta-analysis. It doesn’t have that quantitative component to it. I believe one that I sent had to do with a medical and social area example, but worth taking a look at. What we do with these five topics or five steps that are kind of the steps that we go through to get this mapping of the literature, getting into the scoping review of the thing.

(Slide 28-29) Another review, another level up, is the rapid evidence review. Here we’re trying to say, let’s assume for a moment, we’ve done the scoping, we see that there is probably enough literature out there, there’s certainly is some that’s worth looking at, to try to organize a summary statement about what works. It does entail though, the extraction of the quantitative side of the ledger. That is, pre- and a post-test of the experimental and control condition, or some measure of quantitative outcome.

So we think of it as, you’re going to calculate an effect size of some sort. We’re not going to do a statistical run here. There’s no multivariate analysis, nof-test, t-test. Simple effect size, but we will do it on the major outcomes of the studies. So we have a question that is of particular importance to us, and fairly well focused and narrow; what are the outcomes that the literature in the scoping review is pointing to, to address that issue? Address that outcome or that intervention? The rapid evidence review would say, let’s collect those studies, let’s analyze those studies for their quality. Then, does it look like effects are being generated from those studies that suggest there’s something worthy here of a serious full-fledged kind of analysis and review? I believe I also put a rapid evidence review in there for you.

Now, there are some advantages to this, partly on time and resources, but that are worth taking note of. You can kind of think of it as a brief systematic review. It’s certainly systematic, a scoping review is systematic, a rapid evidence review is systematic, but it is in a truncated form. We get the primary outcomes and we look at the measures of effect. You know, I didn’t put the reference on here, but I stole this from somebody, I have to go back and look at the reference. (Laughter)

There’s going to be a comparison, in terms of the rapid evidence review, time is the big saver here. Usually 4-6 weeks is considered a reasonable amount of time to do a rapid evidence review. You have to have the question, find the literature, have a selection process, and then evaluate or extract the information from that study that is relevant, by participants, or instrumentation, or whatever. So you’re still doing all the systematic eight steps, but what you’re not doing is the big number crunching part, as in the synthesis. You’re going to write basically a descriptive summary that would be a presentation of an outcome of a study. It gives us some ability to suggest causation of effect. It’s limited and you don’t sort of want to “hang your hat on it” if you please, for that purpose, but it gets us to the place. You may also say, “I’ve got 3 studies, I don’t know, that may not be worth the time, and the outcomes are all different, and the intervention is not really well prescribed.” So doing a full-fledged review might not tell you any more than a rapid evidence review would at this stage.

(Slide 30) The third product here is the full systematic review. Would you say that What Works Clearinghouse is kind of a form of rapid evidence assessment?

Cindy: What they are, there are a number of products. There is one called quick turnaround, quick review. I think that probably it would fit under rapid evidence, but still that is a single study review.

Chad: They don’t synthesize across the studies.

Cindy: But there are intervention reports that emphasize cross-studies, for example, on an intervention. Now we also have another product called practice guide that we synthesize across a number of practices.

Chad: Right, so the What Works Clearinghouse products tend to follow this rapid evidence mode of operation. Where they don’t take 5-10 studies and aggregate them together in a single unit of expression of effect. But they do give really good definition to the outcomes and the effects of the individual studies, and have a very rigorous selection and inclusion criteria.

Well, the systematic review is the one that we read that is pages long, either in a journal or 50 pages long in a Cochrane or a Campbell review. Where what we’re really trying to do is identify all of the relevant literature that we can, go through that process of systematically assessing how it functions to collect this information and report it, and statistically analyze it. In the comparison, this is where it shows up for us that the real difference in terms of time, effort, and resources lie. My first Cochrane review took 6 years. Fortunately it wasn’t 6 years, 40 hours a week. Will Shadish was giving a presentation, he said it took him 9 years to do his first, I said “ I got that beat by a bunch”. They take a long time, that’s the moral of the story. They’re labor intensive and it’s certainly a challenging process. So, the evidence comes down to the synthesis then, is both a qualitative summary and a quantitative summary, the meta-analysis of the data. The evidence based conclusions then become causal in interpretation. Lots of potential resource differences in terms of the costs of what is going to be resulting.

OK, let me stop there. Thoughts, gripes, complaints, questions about that? The whole basics of “systematic reviewness,” whether it’s narrative, vote count, or the scoping, rapid evidence, systematic review. All of them, if they are quality productions, will use those steps in one way or another, some of those steps in one way or another, just building on each other.

(Slide 31) OK, so let me take a couple minutes. We thought we should at least give at least a little service here to the searching issues—finding the studies. I’m assuming most people here at least have had some experience with this, so I’m going to whip through this in about three seconds.

If you don’t find the studies you can’t do a quality review, and you can’t do a quality synthesis, because you may have missed really important material. Now, depending on whether you’re looking for a narrative to just describe or a full systematic review, that description, kind of falls along those lines. The more that is put in to the identification, searching, and retrieval of studies, the more likely it is you have a higher level of functioning. There are increasingly, a number of tools that are helpful in being able to be more precise. Instead of going into ERIC and putting in your terms and coming up with 300,000 hits, the indexing of the systems is changing all the time and they are getting substantially better.

(Slide 32) We typically want to look at published and unpublished literature, the grey literature. The idea being that just because it isn’t published doesn’t mean it isn’t science. We know there’s a bias in favor of significant findings in publications. Disciplines have specific tools and subjects that are appropriate for your area or interests. Access to them—most of our institutions have quite good database availability. Format, the more web based operations we use, it seems to me, the more that becomes a factor. I don’t go to the library any more, I don’t have any reason to with most of it online.

How relevant is the search in terms of the times? In some cases the interventions may only be 20 years old, so if you’re searching back to 1948, that is sort of…You’ll pick up terms, examples, illustrations, but it probably won’t be relevant. The language [criterion] is one that sort of gets left out often. There is a lot of material that is not in the English language or not published in English or in the U.S., Canada, Australia, and England. I’m just suggesting that if that is a function of your interest in the body of the work, that it should be accounted for. It’s also sometimes a little more difficult to locate but can none the less be found.

Cindy: I have a question.

Chad: Yeah.

Cindy: Could you go back please, a slide (Slide 32)? Thank you. The second point, Availability of indexing tools and expertise. What is the main point of that,

Chad: Well there are some sources that we have to pay for. You have to make a decision, is it worth paying $200 for an article for you to include it. That is less and less of a problem; at least it is for me for accessing research. So, sometimes you just can’t get to the databases that have studies.

Cindy: But then you if you want the systematic review published in a prestigious journal and the reviewers say that you missed a couple important studies…

Chad: Yeah, I mean it isn’t that this process doesn’t have subjective decisions to it. One of those subjective decisions is how much resource do you have to put into this?

If you have 3 studies, it might be more important to spend the money to get them than if you have 20. One more study that’s not in the group probably won’t change the results. Now, it doesn’t make it as comprehensive, but that’s just sort of the reality of it. Doing a search, if you have somebody who is trained and can do that efficiently, and you’re having to pay them, it can get expensive. I haven’t been able to find anybody to support me on that. I had to drag through that myself and be up on my reference librarian people. I think though that it’s getting so much better every year. Access is becoming less and less of an issue. Even in foreign language sources. Getting an article in Chinese is one thing. Being able to use it is another. But there are ways around that.

Oliver: I can also comment on that. If you go through a lot of the larger systematic review producing institutions such as Cochrane or such as Campbell, there is an increasing trend that they ask you to screen the international literature. Not just North American, Anglo American parts of literature, but European Asia and Australia and so forth. Then the question is, one point that Chad said is, if you go with the European databases, does it really add anything else. What is the yield? If there is a good reason to assume that it will yield some additional literature then what research teams often do is they hook up with someone from Europe, that they put on their team as a literature search specialist, to get access to that database. A good example is Medline Embase. We often, in North America, have access to Medline, this is easy. But Embase, this is sort of the European version of medical literature, this is more difficult. In the end, you might find that both databases for your topic, pretty much have the same literature. If you can make that point, then you’re getting off the hook. They will just let you go with Medline. If there is some indication that Embase in Europe is going to yield some additional studies then you better hook up with some one over in Europe that has access to this that can get you those articles, or you have to pay for it. The pay option is always there because access over the Internet is no longer a big deal. There is the notion that we should look international, we should go very broad these days. Not to introduce any regional bias.

(Slide 33) Chad: Cochrane forced—forced may be a little strong—but they were able to demonstrate the importance of being able to identify randomized trials as part of the search terms, the MESH terms. Over a couple of years, with some evidence to show what was going on in terms of the review process, they were able to persuade them to put in a new mesh term that took into account design, making the process much simpler too. That kind of thing is resource driven task that is useful. There have been some discussions about doing the same kinds of things in ERIC and PsycINFO info and so forth and other search strategies that would have real benefit for us in efficiency. If you have done any of this at all, and y’all have. We know, we sort through a lot of unnecessary, useless pieces of information, to get to the stuff we really want, but we still need to do it. So, common databases I’m sure you’re familiar with these. FRANCIS down here is a French database that didn’t use to be nearly as accessible as it is to us now. There is a Spanish database. There’s a Portuguese database that also has some access. There’s the Chinese ERIC. I had someone from China tell me at one point that there were over 1000 journals, of rehabilitation journals, published in the Chinese language. I don’t know if that’s true or not, let’s assume it’s a lot less. If it’s 100, that’s a lot of material that we don’t have access to, if we’re not able to find that work. Is that right?

Cindy: I think in China there are so many provinces, cities, and associations…

(Slide 34-35) Chad: Yeah, that’s true. OK. In the process, constructing search terms in the strategy of how you search is important and worth paying attention to. We’re not going to spend any time on it here but, we deal with keywords that get to the task and try to limit and truncate where we can to try to make the process more efficient. It is the quality of the retrieval of studies that will determine the quality of the review in general. It’s not a one-shot deal. That’s probably the best take away from this. The process of retrieval is a multiplicative additive kind of arrangement. It’s important that we keep track of what we do. What terms are used? What the sequence, or Boolean language, if you please, is for replication purposes. It also helps when I have to go back and do this again because you can’t do a search at the start of the study, without six years later doing a search again to find out what went on during the six years since you first started.

Ok, I think with that we’re almost at a break time.

1