Use of a Modified Delphi Process in Developing

USE OF A MODIFIED DELPHI PROCESS IN DEVELOPING

TECHNICAL SPECIFICATIONS AND TESTING PROCEDURES

Technical Report FRN-1

NASA Research Grant NGR 05-058-003

March, 1974

Prepared by

Frederick H. Reardon

Department of Mechanical Engineering

CaliforniaStateUniversity, Sacramento

INTRODUCTION

In many areas of modern technology, the development of operationalstandards for complex equipment and the accompanying procedures for evaluation requires that a consensus be reached among a number of technical personswith design, development, or operating experience. Traditional approaches tosuch standards and procedures, involving committees and subcommittees, may notbe effective, particularly if the technical community is dispersed geographicallyor if there are a small number of persons who dominate the larger group. Theproject described in this report developed out of the awareness of the need fora new approach to achieving technical consensus on standards and procedures.

The development of high-performance liquid-propellant rocket engines hasbeen plagued almost from the beginning with a problem known as combustion instability. The occurrence of combustion instability during engine operationnearly always led to impaired operation, if not the destruction, of the engine. In order to coordinate research and development efforts related to this problem,the U.S. government organized a "working group," which was composed of engineersand scientists from the Army, Navy, NASA, and the Air Force, together withexperts from industry, the universities, and research institutes. This JANNAF(Joint Army-Navy-NASA-Air Force) Liquid Propellant Combustion Instability Working Group began in 1964 to meet annually to exchange technical information and

to develop an integrated, comprehensive program to solve the combustion instability problem.

Early efforts were hampered by the extremely complex nature of the problemand by the large number of different engines under development in the 1960's,many of which had very tight budgets and/or time schedules. Gradually, however,the complexities began to be untangled and it became clear that it was possibleto develop an engine that did not suffer from the problem of combustion instability. The theoretical and practical knowledge accumulated by the members ofthe working group was gathered into a manual on instrumentation techniques anda general reference book, to which some 90 authors contributed (Ref. 1). Thesedocuments contributed greatly to the orderly development of liquid propellantrocket engines.

However, in the late 1960's, another concern arose. A clear specification of what constitutes combustion instability that wouldbe acceptable to the entire liquid propellant technical community was needed, aswell as a procedure for evaluating engine stability, both for guiding the initialdevelopment of the engine and components and for verifying the stability of thecompleted engine. The existing situation was such that the stability portion ofthe engine specification varied substantially from one program to another. Moreover, inadequate testing procedures were often used that delayed development andled to severe operational problems after the completion of the development program.

A committee of the JANNAF Liquid Propellant Combustion Instability WorkingGroup was formed in 1969 to provide the basis for specifications that would defineand verify combustion stability in real engines. In their work, the committeesent two questionnaires to Working Group members, one at the beginning of theeffort and one after the rough draft of their report had been written. Theyreceived only a few responses. The combustion stability specification andverification procedure developed by this committee were criticized at theannual working group meeting in October 1971 on several grounds:

(a) the definition of stability was not acceptable to all members;

(b) the test sequences as presented would be likely to require anexcessively large testing program; and

(c) the specifications did not apply to all types of rocket engines.

Since time was limited at the annual meeting, the working group directedthe committee to review the document during the next year and solicit additionalcomments. They were instructed to include these comments in the report, butno format for doing this was specified. Such an action is typical of the"traditional" approach to group or committee functioning. There was no assurance that a large response to the committee's request for comments would beforthcoming, or that including all comments received would result in a satisfactory report. Therefore, an alternative approach was proposed, based largelyon the Delphi technique of sequential questioning interspersed with informationfeedback. This alternative approach was carried out under the sponsorship ofNASA (Research Grant NGR-05-058-003 ). The NASA Technical Officer was Dr. Richard J. Priem, Chemical PropulsionDivision, LewisResearchCenter, Cleveland, Ohio.

It is clear that the variety in experience and interest among liquid propellant rocket scientists and engineers can be advantageous, if properlymanaged, in preparing such documents as specifications and verification procedures. The advantage lies in the wealth of ideas resulting from differentviewpoints. However, it is necessary to get interaction among participants sothat important ideas are not lost and judgments are not distorted by lack ofinformation. A small committee cannot without consultation produce a representative report. Review copies distributed to the technical community usuallyinspire minimal response. Round-table discussions among all interested personsare not feasible because of the expenses (time and money) involved in assemblinga group that is widely dispersed geographically. Insufficient time is availablefor detailed technical discussion at the short and busy annual meeting of theWorking Group.

THE PROCESS

The Modified Delphi Process

When a group of competent persons of differing experiences desires tomake effective decisions in an area of common interest for which well-definedguidelines are not available, it is better to proceed by the method of groupconsensus rather than by majority vote. In this method, the knowledge andexperience of all group members are forged into a group decision to whichall members can give at least partial, if not total, assent. The Delphitechnique (Ref. 2,3) of obtaining a group consensus does not require thegroup to meet, thus avoiding not only the travel expenses, but also the effectsof psychological pressure often found in group discussions. Each participantresponds individually and anonymously to a series of questionnaires. The composite group responses to each questionnaire are returned to the participantbefore he completes the next questionnaire. If his answer to any questiondeviates greatly from the general group consensus, he is asked to justify hisanswer. These justifications become part of the information fed back to allparticipants.

As originally developed for forecasting purposes, the questions are posedin such a way as to produce numerical answers (e.g. "in what year will . . . ?"). The responses are ordered numerically, and the group consensus is taken as therange given by the middle 50% of the participants. The 25% who give loweranswers and the 25% who give higher answers are then asked to explain.

In the Modified Delphi technique developed at CaliforniaStateUniversity,Sacramento, qualitative rather than quantitative answers are treated. The mostfrequently mentioned answers to each question are returned to the participantsalong with the percentage of respondents giving those answers. For example, agroup of about 30 students was asked, "What will be the most significant development of the next 30 years?" Of the dozen different answers given, the mostfrequently mentioned (by about 29) was "world-wide, international cooperation. "The students were then given the result and asked, "Do you agree with this prediction? Why?" About 62% responded "yes," with five different reasons given. The 38% who responded "no" gave three different reasons. This result, includingthe arguments for and against the prediction, was returned to the students. Theywere asked to examine the reasons given for each answer and then to state whetheror not they agreed with the prediction. The result was that 76% agreed.

In general, experience at CSU, Sacramento, with voluntary student participation in this Modified Delphi technique has shown that the students becomeprogressively more involved. Ideas are exchanged on a more objective basis,since each participant has time to think the arguments through and the discussionis not dominated by a few highly articulate persons.

Stability Specifications and Verification Procedures

Seven questionnaires were developed and mailed to scientists and engineersin government, industry, and academic institutions. The original mailing listincluded over 50 persons who were known to have had an interest in the problemof combustion instability at one time. Of this original group, 28 participatedactively; several others responded to the initial questionnaire with lettersindicating that their interests had changed or that they did not feel competentto participate.

Since the discussion at the Working Group meeting had shown that thecommittee-developed combustion stability specifications and verification procedures were not totally unacceptable, it was decided to concentrate only onthose parts which had been criticized, viz., the definition of stability, thetest sequence, and the applicability to different engine types. Accordingly,the first step taken was to get consensus on a practical, or "working" definitionof combustion stability. Since combustion instability is generally characterized

by fluctuations of the pressure in the rocket combustion chamber, a workingdefinition of stability had to include the allowable limits of amplitude anddamping time of any disturbance that would potentially be unstable. This taskrequired 5 cycles of questioning to reach the 90% level of agreement on thebasic definition of stability and the amplitude limit for acceptable engineoperation. A somewhat smaller, but still sizable level of 82% agreement wasreached in regard to damping time.

The second task undertaken was to specify the accepted approach to verifying stability of a rocket engine. This task was begun with the second questionnaire and completed, with 85 agreement, with the sixth questionnaire. Statements defining the type of hardware to be used in testing, the minimum numberof tests at each test condition, and the number of units to be tested weredeveloped.

The final task was to prepare a typical test program, with options andalternatives for application to different kinds of engine systems and missions. This task occupied the last four questionnaires. It was clear at the outsetthat all possible eventualities could not be treated in specifying such a testprogram. Rather, the result of this effort was a guideline, to be modified andadapted to the particular requirements of each engine development program (Ref. 4).

Example: Definition of "Dynamic Stability"

The process can best be discussed by means of an example. The initialquestion of the project dealt with the definition of "dynamic stability," whichis the goal of every rocket engine development program. In spite of the greatamount of work in progress for several years to achieve dynamic stability in thevarious engines under development, no effort had been put into getting agreementon a satisfactory working definition of dynamic stability.

It seemed desirable to start with the definition as written by the JANNAFcommittee:

A. The term "dynamic stability" denotes the property of anengine system to return to normal operation within a prescribed time period after the combustion process is perturbed,without having been shut down or other corrective actiontaken.

The participants were asked, "Is this an adequate definition?" Of those responding, 68% said that the above definition was adequate. They were alsoasked, "What additions should be made to strengthen the definition?" and'What deletions should be made to make the definition more workable?" Therewere a number of responses to the latter two questions. These responses couldbe classified into two groups. Each group of suggestions was used to formulatean alternative definition, as follows:

B.The term "dynamic stability" denotes the property of an enginesystem to return to normal operation within a prescribedperiod of time, consistent with engine constraints, afterthe combustion process is perturbed by any kind of disturbancefrom equilibrium that does not itself damage the engine, without having been shut down or other corrective action taken.

C. The term "dynamic stability" denotes the property of an enginesystem in which transients resulting from a change of any kindin the system operation return to a sufficiently low level ina sufficiently short time to preclude any undesirable effectson the system.

It can be seen that Definition B introduces the practical considerationthat the required damping time of a disturbance should be related to the otherengine operating characteristics as well as its mission. Moreover, this alternative definition includes the idea that there may be disturbances from which adynamically stable engine cannot be expected to recover, namely, disturbancesthat cause damage to the engine. Definition C was an attempt to simplify thewording, while eliminating the arbitrariness Implied in the word "prescribed"by use of the criterion that there should be no undesirable effects caused bytransients in the engine operation.

Participants were asked to choose which of the three definitions was themost satisfactory as a general definition and to explain the reasons for theirchoice. 35% preferred Definition A because they thought it was clear, concise,easiest to interpret technically, and was specific to combustion instability. 41% preferred Definition B, because it was more general, included a betterdefinition of the perturbation, and was related to engine constraints. 24% preferred Definition C because it was more precise, made system requirementsparamount, and avoided the use of a prescribed time period, which these respondents believed should not be part of a general definition but was a way ofdemonstrating dynamic stability.

These results were reported to the participants, together with an attemptat a consensus, which was described as a definition that incorporated all majorideas so that all could accept it, at least as a "second best" definition. Thisconsensus definition was

D.The term "dynamic stability" denotes the property of anengine system to return to normal operation within asufficiently short time to preclude any undesirableeffects on the system, without having been shut down or other

corrective action taken, after the combustion processis perturbed by any kind of disturbance that does itself not damage the engine.

Participants were asked whether Definition D was acceptable. If theythought it was not acceptable, they were asked to write a definition thatthey believed would be acceptable to all those concerned with the problem ofcombustion instability. Since 94% of those responding indicated acceptance,the process was halted at this point. Of course, there were additionalquestions directed at a definition of "normal operation" and at a guidelinefor "a sufficiently short time."

RESULTS AND CONCLUSIONS

After reaching substantial agreement on the test program, a draft of"Guidelines for Combustion Stability Specifications and Verification Procedures" was written and sent to the participants in the questionnaire process. Comments, suggestions, and corrections were received from over half of theparticipants and were incorporated into the final draft, which was approvedby the JANNAF Working Group on Combustion at its annual meeting in August,1973. The "Guidelines" have been published as an official JANNAF document,CPIA Publication No. 247.

In general, the participants felt that the Modified Delphi Techniquewas a highly successful approach to getting the best judgment of the rocketengine community applied to a common problem. One participant noted thateach questionnaire constituted a relatively small time and effort investmentand hence didn't "scare one off." Also, the process was seen as providinghelpful feedback and cross fertilization. Several others admitted to beingsurprised at the success of the method and impressed by the clarity and cohesiveness of the "Guidelines."

The original Delphi method, because all data from the participants is innumerical form, involves little influence on the part of the coordinator. Infact, the entire analysis of the responses to a questionnaire, and much of theformulation of the following questionnaire, can be done by a computer withouthuman intervention. This is not true for the Modified Delphi Technique. Thecoordinator serves actively as interpreter and editor, and therefore exerts alarge influence on the results. This may be an advantage or disadvantage,according to the particular application. For successful application of the

Modified Delphi Technique the coordinator should be knowledgeable and experienced

in the field of interest, but must be sure that divergent viewpoints are notdistorted by personal biases.

One final observation is that conducting the questionnaire process by mail,as was done in this case, stretches the project out over a period of severalmonths, although not requiring much more involvement time than the more traditional meeting approach. Where the total elapsed time is not a critical factor,the involvement and acceptance produced by the Modified Delphi Technique makeit the preferred approach.

REFERENCES

1. Harrje, D.T., and Reardon, F.H., (Eds.), Liquid Propellant RocketCombustion Instability NASA SP-194, U.S. Government Printing Office,Washington, D.C., 1972.

2. Helmer, 0., "Analysis of the Future: The Delphi Method," in Technological Forecasting for Government and Industry, ed. by J.R. Bright,Prentice-Hall, 1968.

3. Dalkey, N.C., and Rourke, D.L., Experimental Assessment of Delphi Procedures with Group Judgments, Rand Corporation Report R-612-ARPA, February1971.

4. Reardon, F.H., Guidelines for Combustion Stability Specifications andVerification Procedures, Chemical Propulsion Information Agency, PublicationNo. 247, 1973.