Incorporating Patient-Preference Evidence Into Regulatory Decision Making

Incorporating Patient-Preference Evidence into Regulatory Decision Making

Authors:Martin P. Ho, MS;F. Reed Johnson, PhD; Juan Marcos Gonzalez, PhD; Herbert P. Lerner, MD;Carolyn Y. Neuland, PhD;Joyce M. Whang, PhD;Michelle McMurry-Heath, MD, PhD; A. BrettHauber, PhD; Telba Z. Irony, PhD

Author Affiliations:Center for Devices and Radiological Health, US Food and Drug Administration (Irony, Ho, Lerner, Neuland, Whang); Triangle Health Preference Research (Johnson); RTI Health Solutions (Hauber, Gonzalez).

Corresponding Author: Telba Z. Irony, PhD, Chief, General and Surgical Devices Branch, Division of Biostatistics, Office of Surveillance and Biometrics, Center for Devices and Radiological Health, US Food and Drug Administration, 10903 New Hampshire Ave, Building 66, Room 2232, Silver Spring, MD 20993-0002 (Email:; Phone: 301-796-6014; Fax: 301-847-8123).

Coauthors’ e-mail addresses:

Martin P. Ho /
F. Reed Johnson /
Juan Marcos Gonzalez /
Herbert P. Lerner /
Carolyn Y. Neuland /
Joyce M. Whang /
Michelle McMurry-Heath /
A.Brett Hauber /

Contents

Appendix Table 1. Features, Outcomes, and Levels in the Study Design

Appendix B. Experimental Design

Appendix C. Response-Rate Calculations

Appendix D. Risk Tutorial

Appendix E. Tests of Internal Validity

Appendix F. Characteristics of Included Respondents, Excluded Respondents, and General Obese Population

Appendix G. Choice-Model Log-Odds Parameter Estimates

References Cited in Appendices

Appendix A.Attributesand Levels in the Study Design

Device Attributes / Survey Definition / Levels
Average weight loss / Doctors cannot say exactly how much weight people will lose, or how long people will maintain the weight they lose with a weight-loss device. However, doctors know what has happened to people who have gotten a particular weight-loss device in the past.The typical or average result can help people understand what they could expect from that device themselves. /

• 5%
• 10%
• 20%
• 30%

Type of surgery / Endoscopic surgery – Endoscopic surgery uses a long, flexible tube with a camera and light attached to it to place a device in the body. The tube is put into the stomach through the mouth and throat. For example, a stomach balloon is a device that is put into the stomach using this method. The balloon is filled with liquid or air, which makes less space for food.
Laparoscopic surgery – Laparoscopic surgery is an operation that uses small cuts in the belly.These openings are used for a camera with a light and tools for operating on the stomach or placing a device in the body to place the device in the body. For example, an adjustable lap band (laparoscopic band) is a device that is placed around the stomach through a small cut in the belly.It squeezes the stomach to make it smaller.
Open surgery – Open surgery is an operation that uses large cuts to open the belly to place a device in the body. /

• Endoscopic surgery
• Laparoscopic surgery
• Open surgery

Chance of dying from getting a weight-loss device / Each human figure in the box below represents one person who gets a weight-loss device. There are 100 human figures representing 100 people in the box.

2%(2 out of 100)
The human figures in color represent the number of people who die within a year after getting a weight-loss device. The human figures in gray represent the number of people who do not die within a year after getting a weight-loss device. / Each respondent was randomized with a 50% probability to either one of the following two sets of levels:

• 0%
• 1% (10 out of 1,000)
• 3% (30 out of 1,000)
• 5% (50 out of 1,000)
• 10% (100 out of 1,000)

Or

• 0%
• 1% (10 out of 1,000)
• 3% (30 out of 1,000)
• 8% (80 out of 1,000)
• 15% (150 out of 1,000)

Recommended dietary restrictions / Eat ¼ cup of food at a time – This amount of food is about half the size of an apple. After eating, people have to wait at least 1 hour before eating again.
Wait 4 hours between meals – People cannot eat food for 4 hours after finishing a meal.
Can’t eat sweets or foods that are hard to digest – People cannot eat sweets (such as ice cream or milk shakes) or foods that are difficult to digest (such as pizza, French fries, or steak). Also, people cannot eat food with a lot of fiber. /

• Eat ¼ cup of food at a time
• Wait 4 hours between eating
• Can’t eat sweets or foods that are hard to digest

Average weight-loss duration / How long the weight loss lasts depends on the particular device and people’s own behavior. When people lose weight after getting a weight-loss device, they need to change their lifestyle and accept diet restrictions to avoid regaining weight. In addition to diet restrictions, these changes include increasing physical activity and having weekly meetings with a support group. /

• 6 months
• 1 year
• 5 years

Average side-effect duration / After getting a weight-loss device, some people have side effects. These side effects include difficulty swallowing, nausea, vomiting, and pain in or around the stomach. With some devices, side effects last about 1 month, on average. With other devices, the average duration of the side effects is much longer. People treat these side effects using over-the-counter medicines, or doctors prescribe medicines to help with these problems.
Even after taking over-the-counter or prescription medicine for these side effects, some people still do feel bad enough about once a week that they have trouble doing everyday work or social activities. /

• None
• 1 month
• 1 year
• 5 years

Chance of side effects requiring hospitalization / Within a year after getting a weight-loss device, some people will need to be hospitalized to treat serious side effects.
Treating these serious side effects can often require:

• Visits to the hospital with no operation – After getting some weight-loss devices, some people have problems such as serious infections or dehydration from severe vomiting that require care in a hospital such as intravenous (IV) antibiotics or fluids. After being treated, people with these problems have to stay in the hospital for one or two days.
• Visits to the hospital for an operation – After getting some weight-loss devices, some people have problems such as serious bleeding inside the body that are severe enough that they require an emergency operation. After the operation, people with this kind of problem will have to stay in the hospital for several days.

/

• None
• 5% chance of going to hospital with no surgery
• 20% chance of going to hospital with no surgery
• 5% chance of going to hospital for surgery

Average reduction in dose of prescription drugs for comorbidity at the lower weight
or
Chance of getting comorbidity / Possible obesity-related comorbidities included hypertension, high cholesterol, and diabetes.If diagnosed with a single comorbidity, the comorbidity outcome was defined as average reduction in need for prescription drugs to treat that comorbidity.If diagnosed with more than one comorbidity, the comorbidity outcome was defined as average reduction in need for prescription drugs for the comorbidity of most concern to the respondent. If no comorbidity diagnosis, the comorbidity outcome was defined as average reduction in risk of comorbidity of most concern to the respondent.
Effects of weight loss on diabetesResearch studies show that people who are overweight are about 7 times more likely to develop diabetes than people with normal weight. Diabetes causes people’s level of blood sugar to be higher than normal.People with diabetes have a much higher chance of having health problems such as kidney failure, blindness, heart attacks, and amputations. Taking medicines and following diet limitations every day for the rest of their life can lower their blood sugar and decrease the chance of having these serious health problems. If people can lose weight and maintain the weight loss after getting a weight-loss device, they can decrease the chance of getting diabetes. For people who already have diabetes, maintaining their weight loss can let them take less medicine or stop taking medicine.
Effects of weight loss on high blood pressure Research studies show that people who are overweight are about 6 times more likely to get high blood pressure than people with normal weight. Although high blood pressure usually has no symptoms, over time people with high blood pressure have a higher chance of having health problems such as kidney damage, strokes, and heart attacks. Taking medicines and following diet limitations every day for the rest of their life can lower their blood pressure and decrease the chance of having these serious health problems.If people can lose weight and maintain the weight loss after getting a weight-loss device, they can decrease the chance of having high blood pressure. For people who already have high blood pressure, maintaining their weight loss can let them take less medicine or stop taking medicine.
Effects of weight loss on high cholesterol Research studies show that people who are overweight are about 2 times more likely to develop high cholesterol than people with normal weight. Although high cholesterol usually has no symptoms, over time, people with high cholesterol have a higher chance of having health problems such as strokes and heart attacks. Taking medicines and following diet limitations every day for the rest of their life can lower their cholesterol and decrease the chance of having these serious health problems. If people are able to maintain the weight they lose after getting a weight-loss device, they can decrease the chance of having high cholesterol. For people who already have high cholesterol, maintaining their weight loss can let them take less medicine or stop taking medicine. / For primary diagnosed comorbidity:

• Eliminate need for prescription drugs to treat comorbidity
• Half the current dose to treat comorbidity
• No change

If no diagnosed comorbidity:

• Eliminates risk
• 50% lower risk
• No change

1

Appendix B.Experimental Design

A statistically rigorous choice-format conjoint-analysis survey requires an experimental design with known statistical properties. Specifically, the choice-format conjoint-analysis survey instrument requires assembling a series of trade-off questions that allow precise estimation of the preference weights of interest. The design consists of combinations of the attribute levels that describe a set of hypothetical device profiles. The experimental design used in this study defines a series of trade-off questions designed to generate as much preference information as possible while limiting subject burden.

During the pretesting of the survey, respondents were skeptical of devices that resulted in substantial weight loss without a corresponding substantial reduction in comorbidity risk or treatment requirements, particularly when the alternative device shown in the same question resulted in modest weight loss with large improvement in the comorbidity attribute. Whether or not respondents’ perceptions are clinically accurate, in the experimental design we must limit the confusion related to completely independent variation in attribute levels. For this reason, the experimental design was restricted to include only plausible combinations of weight loss and comorbidity benefits. The restrictions still allowed some degree of free variation between the amount of weight loss and the severity or likelihood of comorbidities.

Most choice-format conjoint applications currently use a D-optimal design to reduce the number of paired comparisons to the smallest number necessary for efficient estimation of preference weights.2-5 Efficient designs can be produced using an iterative computer algorithm.6 We used a commonly used D-optimal algorithm in SAS software (SAS Institute, Inc., Cary, North Carolina) to search for a near-optimal experimental design.

In addition to evaluating the design D-efficiency, we ran two diagnostic tests to evaluate the frequency of attribute levels and balance their use in the design and specific pairwise correlations between the attribute levels.

Frequency of Attribute Levels

The experimental design in this study does not have a disparity greater than 19 between the frequencies of any attribute, meaning that no one level of an attribute is presented in the experimental design more than 19 more times than any other level of the same attribute. Ideally, these disparities would be 0 for all attributes. It is likely that the restrictions imposed between the levels of weight loss and the severity or likelihood of comorbidities required some imbalance in attribute levels.

Table B-1.Attributes and Levels Used in the Trade-Off Questions

Attribute Labels / Levels and Level Labels / Variable
Average amount of weight loss / 30% [Expressed in pounds; calculated based on reported weight] / WTLS1
20% [Expressed in pounds; calculated based on reported weight] / WTLS2
10% [Expressed in pounds; calculated based on reported weight] / WTLS3
5% [Expressed in pounds; calculated based on reported weight] / WTLS4
Type of operation / Endoscopic surgery / OPTYP1
Laparoscopic surgery / OPTYP2
Open surgery / OPTYP3
Chance of dying from getting a weight-loss device / Range 1 / Range 2
0% / 0% / MORT1
1% (10 out of 1,000) / 1% (10 out of 1,000) / MORT2
3% (30 out of 1,000) / 3% (30 out of 1,000) / MORT3
5% (50 out of 1,000) / 8 % (80 out of 1,000) / MORT4
10% (100 out of 1,000) / 15% (150 out of 1,000) / MORT5
Average reduction in dose of prescription drugs for [comorbidity] at the lower weight
or
Chance of getting [comorbidity] / For primary comorbidity / If no comorbidity
Eliminate need for prescription drugs to treat comorbidity / Eliminates risk / COMOR1
Half the current dose to treat comorbidity / 50% lower risk / COMOR2
No change / No change / COMOR3
On average, how long the weight loss lasts / 5 years / WLTIM1
1 year / WLTIM2
6 months / WLTIM3
On average, how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month.) / None / SETIM1
1 month / SETIM2
1 year / SETIM3
5 years / SETIM4
Chance of side effects requiring hospitalization / None / HOSP1
5% chance of going to hospital with no surgery / HOSP2
20% chance of going to hospital with no surgery / HOSP3
5% chance of going to hospital for surgery / HOSP4
Recommended dietary restrictions / Eat ¼ cup of food at a time / DIET1
Wait 4 hours between eating / DIET2
Can’t eat sweets or foods that are hard to digest / DIET3

Additional design instructions:

• Respondents were randomly assigned to one of 15 versions or blocks.
• The order of the questions each respondent saw within a version was randomized.
• Assignment of range 1 and range 2 under “Chance of dying from getting a weight-loss device” was randomized and maintained across respondents.
• For the attribute “Average reduction in dose of prescription drugs for [comorbidity] at the lower weight OR Chance of getting [comorbidity],” the following was done:

–If question B11 was equal to Don’t know or None of the above, the survey presented the “chance of getting [comorbidity]” attribute label. Populated the [comorbidity] field with the response to B11a. Used the “If no comorbidity” set of level labels.

–If question B11 was equal to only one of Diabetes, High Blood Pressure, or High Cholesterol, the survey presented the “Average reduction in dose” attribute label. Populated the [comorbidity] field with the response to B11. Used the “For primary comorbidity” set of level labels.

–If question B11 was equal to two or more of Diabetes, High Blood pressure, or High Cholesterol, the survey presented the “Average reduction in dose” attribute label. Populated the [comorbidity] field with the response to B11a. Used the “For primary comorbidity” set of attribute labels.

–If questions B11 and B11a were both equal to Don’t know, then the survey presented the attribute label “Chance of getting [comorbidity].” Populated the [comorbidity] field with Diabetes. Used the “If no comorbidity” level labels.

• Respondents answered one version, or 8 questions, in total.

Appendix C.Response-Rate Calculations

All members of KnowledgePanel® have a known probability of selection. As a result, it is possible to calculate a proper response rate that takes into account all sources of nonresponse.Table C-1 contains the components of the response-rate calculations. A detailed description of how to compute response metrics for online panels can be found in Callegaro and DiSogra.1

Table C-1.Response Rate Summary Metrics

Metric Category / Value
A.Number of Assigned Panelists / 1057
B.Study-Specific Average Panel Recruitment Rate (RECR) / 15.6%
C. Study-Specific Average Household Profile Rate (PROR) / 67.4%
D. Study-Specific Average Household Retention Rate (RETR) / 31.4%
E. Number of Total Study Completes / 710
F. Study Completion Rate (COMPR) / 67.2%
G. Number of Study Break-offs / 55
H. Study Breakoff Rate (BOR) / 7.2%
I. Number of Qualified Completes / 568
J. Study Qualification Rate (QUALR) / 80.0%
K. Cumulative Response Rate 1 (CUMRR1) / 6.8%
L. Cumulative Response Rate 2 (CUMRR2) / 2.1%

Formulas Used for Calculations

The following formulas were taken from Callegaro and DiSogra1and used to calculate the response summary metrics reported above.Respondent-level cohort recruitment, profile, and retention rates were calculated for each study respondent and averaged across all study respondents to yield the study-specific rates reported in Table C-1.

Respondent-level Panel Recruitment Rate (RECR) =

Respondent-level Profile Rate (PROR) =

Respondent-level Retention Rate (RETR) =

Study Completion Rate (COMPR) =