Formats for Problems; Multiple Choice

Chapter 7

Formats for Problems; Multiple Choice

Problems for written tests can be posed in many different formats. The choice that is made depends on the purpose of the test e.g. a simple quiz to be given at the start of a new subject, or an end-of-the-unit test. Moreover, the choice depends on the level of the question. A question assessing basic skills in general will have a different format than a question that requires mathematical reasoning or choosing your own mathematical tools.

Why do you need different formats for problems in a balanced assessment program? Some reasons are:

1. It is simply not possible to operationalize all the goals of mathematics education if we would restrain ourselves by using for example only multiple choice or single answer questions;
2. Some formats for problems will favor one group of students whereas another format appeals to another group;
3. A teacher wants to ensure that in the course of one school year students have experienced a variety of formats for problems so they can show their competencies in different ways. For example, at one time they might need to demonstrate that they are able to reproduce the properties of a rhombus; whereas, at another time you may want them to investigate the least possible number of popular votes required to become president of the United States!

First we will give an overview of the different formats for questions. These will then be elaborated upon in this and the next chapters.

Note: The words “problem” and “question” and “task” are often mixed up. A problem can consist of more than one question. A task is referred to as something you are doing for homework or as a group work assignment. A task may also consist of more than one problem.

An Overview

1. Multiple choice questions

Often multiple choice questions are called “closed questions” since students can only choose one answer and cannot give reasons or calculations to show their answer is correct. There cannot be a partly correct answer.

1. Single answer or short answer questions.

Single answer questions have only one correct answer. They are sometimes called “closed open”. For short answer questions students have to show their calculations. Their answer may be partly correct, for example, because they did not round off properly or forgot to mention appropriate units.

1. Other open questions

The correct answer could still be just a number or a formula, but the activities necessary to find the answer are more complicated or involve higher-order strategies compared to single answer questions. However, the distinction between different types of open questions is an arbitrary one.

1. Extended-response open problems

Students will have to give mathematical reasons to support their answer in a more elaborated way. Often the process is more important than the product. Many times it will be necessary to make a mathematical model of the situation first, and afterwards interpret and justify the results according to the original realistic situation.

1. Multiple question items (super items)

These are tasks that enable the students to get involved with a context by asking a series of questions of increasing complexity and level of competence. The first question is often referred to as an ‘entry-question’, a simple level I question to explore the situation. The next questions are more difficult, preferably as well, on Levels 1, 2 and 3.

1. Investigations, essays

This type of question is mainly used to assess higher order levels of competency. Students are asked to investigate, to draw conclusions or give proof, to reason mathematically about their findings or to use and criticize mathematical models.

1. Two-stage task

A two-stage task may appear in different forms. The previous type of doing your own investigation, may be divided in two parts. The first part consists of questions (open or closed) about the problem, on different levels of competency. The results are assessed by the teacher and the student gets feed back. This way the teacher makes sure all students know what is expected from them and have showed whether they master the basic skills needed to perform the investigation. Moreover, it is possible to prevent students from taking wrong (side) paths that do not lead to a correct answer or that do not assess what was meant to be assessed.

Then the next part is done without a time limit and this part should complement the mathematical competencies that were missed in the first stage.

1. Own productions

Students produce and solve their own problems. By doing this, it is possible for them to demonstrate a much higher level of thinking than the teacher expected. Moreover, by using this type of task, the teacher sometimes finds misconceptions that do not show when students answer the teacher’s question instead of their own.

The different formats will now be discussed in further detail. The order in which the different formats for questions, problems and tasks are shown does not mean this is an order from “bad” formats to “good” formats. It largely depends on the purpose of the test in which they appear and the mathematical skills a teacher wants to assess by using them. And, of course, sometimes there is not even a choice to be made. The test is compulsory.

Multiple choice questions

Not knowing how a student found the answer is characteristic for multiple choice problems. The only advantage of using multiple choice questions is the speed with which you can score them. The only reason to provide students with practice answering multiple choice questions may be that you find them in compulsory tests like standardized tests.

Students do need practice in answering multiple choice questions. Even reading them properly and finding out what exactly is asked can cause difficulties.

If you really want to know what kind of information you are missing about your students abilities, while using the multiple choice format, you might do a small experiment. Ask the students not only to write down the correct answer to the problem but also show all their calculations and their reasoning. You almost certainly will find, as we did:

- students who did all the calculations right but in the end took the wrong letter;

- students who got the right letter, but with a totally wrong explanation as the following example shows.

This example also shows that over-reliance on multiple choice questions creates the illusion that students know more than they really do. Use of simple probability predicts that just by guessing a lot of students will demonstrate they “know” the correct answer. Moreover, the exposure of possible answers guides students towards possible solution strategies whereas if the problem was presented without solutions students may not seem so clever!

You will also find students who just guessed, because they really think that is the proper way you solve a multiple choice problem, or students who solved the problem right but in quite a different way than was expected, for example by filling in all the answers in the equation instead of solving the equation.

Multiple choice questions create the illusion that students’ incorrect answers mean they understand little or nothing about the content addressed. When, in fact, students may know quite a bit but have been distracted or just made a small computational error. Calculating 3 x 3 = 6 instead of 9 is a popular one. But all incorrect answers are counted the same, a teacher gets no information whatsoever about the thought processes involved.

Some students solve the problem in a different way than was expected because they use their daily life experience. Sometimes the designers of the problem obviously do not share this experience.

In solving the next problem, some students used their shopping experience.

Using common sense instead of difficult calculations is an attitude we like to see in students. It often proves to be an effective method as the next example shows.

Sometimes the easiest way to find the correct letter, is to look at the answers instead of the problem. That is not the way the problem should be solved but as stated before, by using this format for a question, students are encourage to find all kind of ways to answer it in a simple way that the designers did not really mean to.In that case it is not certain whether you assessed students’ “test-wiseness” or their understanding of mathematics.

A teacher gave us the next problem and he told us:

“Teaching how to tackle multiple choice problems takes a lot of my time because I want my students to do really well on the test and it takes a specific way of thinking to do those problems well. So I tell them to look at the answers and try to work ‘backwards’. If you do so, always start with D, because they want you to try all of them. So be smart, most of the time it is C or D! Yes, of course this is ‘teaching to the test’, but it wasn’t me who chose these tests!”

Sometimes the question itself is not correct. The next problem is mathematically wrong! But even so, students are expected to find the right answer.

Good multiple choice problems are difficult to design. There are occasions where multiple choice problems are a good format to use, as the next question shows. Posing this one as an ‘open’ question, may result in many halfway correct answers, or answers that are difficult to understand. For this problem, we used the, sometimes cryptic student answers for the ‘wrong’ alternatives.

Summing up our findings in this chapter:

1. Multiple choice problems do not give you insight in the way students find their answers
2. Multiple choice questions are found in standardized tests, so we cannot avoid them.
3. In answering multiple choice questions it proves useful to use ‘common sense’.
4. Multiple choice questions can be answered correctly by using wrong reasons without it being noticed.
5. Good multiple choice questions are hard to design.
6. Multiple choice questions may be answered incorrectly due to minor computational errors.