Barriers to Innovation

Annexes to Chapter 5: Preliminary Results

Priority Medicines for Europe and the World
"A Public Health Approach to Innovation"

Annexes to 5.4
Preliminary Results

15 September 2004

Annex 5.4A: Drug to Drug Comparisons

Annex 5.4B: Balancing Risks and Benefit: A preliminary attempt at quantification

Annex 5.4A: Drug to Drug Comparisons

We found 187 drug to drug comparisons in Clinical Evidence that dealt with interventions for diseases on the preliminary “priority” list. Twenty four (12.8%) of these were multiple comparisons (e.g., drugs A,B,C, wherein A v. B, A v C, B v. C). Of the total comparisons, forty eight (48/187= 25.6%) showed significant differences ( at the 95% confidence level) in clinical efficacy between one intervention and another (i.e., between drug A v. B or all combinations of several drugs compared two at a time). A full 75% of these 187 comparisons revealed no significant difference in clinical efficacy between all of the drugs tested. A table showing the comparisons that showed NO statistically significant differences is found in Table 5.4A1 and a table showing the remaining significant comparisons is in Table 5.4A2.

Clinical Evidence itself performed a related analysis and found that a “quick scan” of the 1851 treatments in volume 10 revealed that 270 treatments (15%) were beneficial and 1050 (56%) were either unlikely to be beneficial, likely to be ineffective or of unknown effectiveness.

Annex Table 5.4.A1 Comparative Trials- NO Difference between Interventions

Table 5.4A2: Comparative Trials

Annex 5.4B: Balancing Risks and Benefit: A preliminary attempt at quantification

There is limited available data on side effect profiles of most pharmacological interventions in the Cochrane Database. The data, we believe, is not sufficient to determine which side effects are clinically important. For policy analyses, we have quantified the relative risk of side effects with the limited data available to use. We find that certain interventions, even though they are clinically efficacious have side effect profiles that suggest an unfavourable benefit/risk balance.

There are two key limitations, however, to this "balancing" analysis. First, many clinical trials found in the Cochrane database do not even report adverse events so our dataset is weak. Second, measuring the clinical effectiveness of a “harm” in a clinical trial is difficult, particularly since the more rare, but important side effects, will not be seen in a trial with few patients and run over a relatively short time period.

Figure 5.4B1 is a schematic diagram of a risk/benefit relationship for a host of interventions and a variety of diseases. Relative risk numbers on the X axis greater than 1 denote benefit (as per our convention). Adverse event relative risks on the Y axis are usually, but not always, less than 1 (as per our convention). Figure 5.4B2 presents actual data for those chronic conditions where both "adverse events" and efficacy were recorded via meta-analysis. The vast majority of these are for interventions for treating mental conditions.

From a public health viewpoint, beneficial interventions with risky side effect profiles should be investigated further. The present Project raises questions about the public health definition of “innovative R&D”. We suggest that “innovation” does not necessarily mean creating new products but might mean taking a renewed look at existing medicines- in part to improve their side effect profiles.

Figure 5.4.B1 Schematic Risk Benefit matrix

Figure 5.4B2

For each meta-analysis that has both sets of data, we have tried to estimate the balance between adverse events and clinical effectiveness. An example of a dataset is shown below in Figure 5.4B3 which shows the balance of risk and benefit for selected interventions used in treating infectious diseases.

Just as we have looked at the lower confidence interval of the relative risk ratio to count the number of times this interval falls in the zone of “no benefit” and the zone of “benefit” (see Chapter 4 of Report), we have looked at the lower confidence intervals of the adverse event relative risk ratios. For all the adverse event data that we have for a given intervention (which is admittedly sparse) we took the average lower confidence interval score as an estimator of adverse events. Since we are looking at adverse events, this interval is almost always less than 1 (as we have defined it) and we are asking, “how far away from 1 (i.e., no harm) is it?.” The further below 1 the number, the more people were affected by the particular adverse event in the clinical trial. We arbitrarily have made three zones of “adverse event” based on the average of the lower confidence interval:

Low Risk: Average lower confidence interval between 0.99 and 0.6

Medium Risk: Average lower confidence interval between 0.59 and 0.3

High Risk: Average lower confidence interval less than 0.29

Similarly, for outcomes designed to measure benefit, we took the average lower confidence interval of the relative risk ratio as a measure of a beneficial event measure. Since we are looking at purported beneficial events, this lower confidence interval should be greater than 1 for many analyses and we are also asking, “how far away from 1 (no benefit) is it?” The higher the number, the more people are positively benefited in the clinical trial. We arbitrarily have made three zones of “benefit” based on the average of the lower confidence interval:

Marginal Benefit: Average lower confidence interval between 1.01 and 1.2

Medium Benefit: Average lower confidence interval between 1.21 and 1.39

High Benefit: Average lower confidence interval greater than 1.4

We have used data generated in this way to summarize the average adverse and beneficial event measures for a variety of acute and chronic conditions, as presented below in raw data form in Tables 5.4B1 and Table 5.4B2. The benefit/ risk “index” is simply the difference of these two averages. For those interventions with "high risk" as we have defined it, small differences (column 3 in the Tables) mean that these interventions that might be marginally beneficial, but also carry the weight of potentially risky side effects.

Figure 5.4B3 Adverse Event and Benefit Measures for Acute Conditions