Designing, Conducting and Analysing Surveys and Questionnaires (ASQ)

FOLIO

Facilitated Online Learning as an InteractiveOpportunity

Designing, Conducting and Analysing Surveys and Questionnaires (ASQ)

Choosing the question type (part 2)

Closed Questions, Open-Ended Questions & Likert Scales

Introduction

This is the second of our briefings on choosing question types. Below we explore the use of closed questions, open-ended questions and Likert scales.

Closed Questions

Closed questions limit the respondents to a range of fixed responses on particular points to gather quantitative information about students' opinions (1). For example:

1. 'Do you find the mediated literature search service to be useful?'

(select one answer): 'yes' 'no' 'don't know'

2. 'Having completed the critical appraisal course, I feel that I know more about the topic'

(select one answer):

Strongly disagree

Disagree

No opinion

Agree

Strongly agree

Closed questions are useful because they enable questionnaire responses to be analysed easily (for e.g. the results of closed questions can be analysed simply by adding up the numbers of respondents that replied 'Yes' or 'No' to a question). From the point of view of respondents, they are also very quick and easy to answer (1). However, because responses are limited to a set of alternatives, closed questions have the potential to be biased. (2) In addition, respondents may feel the need to provide an explanation about their answer, but are unable to due to the restrictive nature of the question type.

Open-Ended Questions

Open-ended questions are qualitative and allow the exploration of many themes as respondents can write about any issues they wish in response to the question. It is argued that open-ended questions are the most potentially valuable type of survey question as it is usually the only part of a questionnaire where respondents are free to address the issues that concern them, rather than the issues that concern those conducting the survey (3). For example, if you are evaluating user satisfaction of your service, the open-ended questions often tell you what aspects of service are of most concern to your users. Whereas you might find that if you are asking respondents to rate aspects of service on a scale, they might rate four or five items as low. With an open-ended version of the question, respondents may only have strong opinions about one aspect of the service - this then tells you where to start in improving the service. (3) In answering open-ended questions, respondents may well raise issues that you have not anticipated, or even thought of (4).

However open-ended questions can also create some problems precisely because of the freedom with which the respondents answer. For example, some respondents may answer ambiguously or in a contradictory fashion. Some may have misunderstood the question and answer something unrelated, or someone's answer could get misunderstood by the person analysing the questionnaires. (5) Also you may find that those respondents without strong opinions on a particular question may write very little or nothing at all. Answers to open-ended questions are both time consuming for respondents to complete and for you to analyse when you are collating your data (2). If questionnaires responses are handwritten, you may find that some are illegible and therefore will be wasted (4).

Tips on open-ended questions

Open-ended questions allow for thoughts, feelings and experiences to be recorded (4).
It is a good idea to use open-ended questions when it is not possible to compile a comprehensive list of alternative options for respondents to choose from (6).
Open-ended questions should begin with words such as "why" and "how" or phrases such as "What do you think about..." (5)
Make sure you leave enough space for people to fill in their responses to questions, however also be wary of leaving too much blank space as respondents may find this daunting and be discouraged from responding (4).
Open-ended questions can be used as a follow-up to closed questions in order for respondents to qualify their answers (2). For example you may ask respondents to rate an aspect of the library and information service on a scale of 1-5 and then ask an open-ended question for their reasons why they have rated the service at this point on the scale.
Be wary of including too many open-ended questions on one questionnaire as you may find that this negatively affects the response rate (7).
If when you design your questionnaire you find that you only have open-ended questions, consider instead asking them via a semi-structured interview. If time and/or resources permit this, reconsider your questions and try to restructure some of them into closed questions. (4)

Examples of open-ended questions

[On an information skills training course evaluation] Why did you enrol on this course?
[On a library and information service user survey] Are there any services not currently available that you would like the library to provide?
[At the end of a questionnaire] Are there any additional comments you would like to make?

Likert Scales

The Likert scale was developed by Rensis Likert in 1932. Questions using Likert scales usually present a statement and respondents are asked to express agreement or disagreement on a scale, generally consisting of five points as below:

Strongly Agree Agree Neutral Disagree Strongly Disagree

1 2 3 4 5

The number assigned to each response becomes the value for that response. (8) This scale can also be reversed so that 1 = Strongly Disagree and 5 = Strongly Agree (9). This type of scale may also be replaced by a combination of written and numerical such as:

Strongly Agree 1...... 2...... 3...... 4...... 5 Strongly Disagree

However, bear in mind that there is evidence that more reliable and valid data is yielded if all the points on the scale are labelled. It also helps to ensure constancy of meaning, as different respondents may attach different meanings to unlabelled points. (10) Also note that the numbers do not have to appear on the questionnaire that the respondents fill in, it is acceptable for the scale to consist solely of strongly agree, agree etc.

As shown above, Likert scales give an idea of how strongly respondents feel about something, and therefore are potentially more useful than a "yes/no" type question (11). As well as "agree v disagree" type dimensions, Likert scales can also be used for:

Satisfied v Dissatisfied
Important v Unimportant
Always v Never (10)
Successful v Unsuccessful

Questions using Likert Scales are relatively simple to construct and they are easy for respondents to complete (8), so you may find the response rate to these type of questions is higher than open-ended questions. When analysing the responses from Likert scales, the data can be treated as quantative data and is therefore easier to analyse statistically (11).

Note that Likert scales are sometimes referred to as summated scales or summated rating.

Some issues with Likert Scales

There is some debate over whether or not to include a neutral point on the scale - some researchers avoid respondents "sitting on the fence" by giving an even number of choices (i.e. 4 or 6) (9), this is know as a "forced choice" method (12).

The reliability of the Likert scale tends to increase with the number of items used. However as the number of items increases, so does the time taken to complete the question and this may demotivate your respondents. There is no hard and fast rule to determine the final number of items in a scale and this will reflect the nature and complexity of the question (9), but it is generally accepted that between 5 and 7 points on a scale is optimal (10).

Likert scales may be subject to distortion from several causes. Respondents may:

Avoid using extreme response categories, i.e. points 1 and 5 on the scale (known as central tendency bias).
Agree with statements as they are presented (known as acquiescence response bias) - i.e. if the statement is positive (e.g. "I find the library and information service useful") respondents are more likely to answer in the affirmative.
Try to present themselves/their opinions in a more favourable light (known as social desirability bias). (12)

Examples of Likert Scales

[On an information skills training course evaluation]

How difficult did you find this training session? (1=very easy, 5= very difficult):

(Please circle your choice) 1 2 3 4 5

[On a library and information service user survey]

Please indicate on the scale your opinion of the following statement:

The inter-library loans service provided by the library is important to me.

Strongly Agree Agree Neutral Disagree Strongly Disagree

Examples of library user surveys:

Evidence Base. Library user survey
LondonSchool of Economics. Information skills training evaluation form
MiddlesexUniversity. Information Skills Training Evaluation Form
University of ManchesterBusinessSchool. Database training evaluation form
Essex County Council. 'Speakup' Library Services Questionnaire

References

1. Soroka, S.N. (2002) Number of Responses and the Most Important Problem [online] [Accessed May 2006]

2. Galloway, K. (1997) Advantages and Disadvantages of Closed and Open-Ended Questions [online] [Accessed May 2006]

3. Fitzgerald, J. (1996) The Best Questions to Ask [online] [Accessed May 2006]

4. Boynton, P.M. (2002) CHAIN Conference, Sunningdale - February 2002 [online] [Accessed May 2006]

5. Cano, V. (2000) Questionnaire Design [online] [Accessed May 2006]

6. Leung WC. How to design a questionnaire. StudentBMJ 2001; 9: 187-9. [online] [Accessed May 2006]

7. Van Bennekom, F. (2006) CRM applications, solutions, advice and tips [online] [Accessed May 2006]

8. Page-Bucci, H. (2003) The value of Likert scales in measuring attitudes of online learners [online] [Accessed May 2006]

9.University of Salford (2006) Attitude Scales [online] [Accessed May 2006]

10.University of Surrey (2005) Questionnaire Design 2: Attitudes, Question order, Filtering etc. [online] [Accessed May 2006]

11. Holah, M. (2006) Rating Scales [online] [Accessed May 2006]

12. Wikipedia (2006) Likert scale [online] [Accessed May 2006]