Formats for Problems; Multiple Choice
Problems for written tests can be posed in many different formats. The choice that is made depends on the purpose of the test e.g. a simple quiz to be given at the start of a new subject, or an end-of-the-unit test. Moreover, the choice depends on the level of the question. A question assessing basic skills in general will have a different format than a question that requires mathematical reasoning or choosing your own mathematical tools.
Why do you need different formats for problems in a balanced assessment program? Some reasons are:
- It is simply not possible to operationalize all the goals of mathematics education if we would restrain ourselves by using for example only multiple choice or single answer questions;
- Some formats for problems will favor one group of students whereas another format appeals to another group;
- A teacher wants to ensure that in the course of one school year students have experienced a variety of formats for problems so they can show their competencies in different ways. For example, at one time they might need to demonstrate that they are able to reproduce the properties of a rhombus; whereas, at another time you may want them to investigate the least possible number of popular votes required to become president of the United States!
First we will give an overview of the different formats for questions. These will then be elaborated upon in this and the next chapters.
Note: The words “problem” and “question” and “task” are often mixed up. A problem can consist of more than one question. A task is referred to as something you are doing for homework or as a group work assignment. A task may also consist of more than one problem.
- Multiple choice questions
Often multiple choice questions are called “closed questions” since students can only choose one answer and cannot give reasons or calculations to show their answer is correct. There cannot be a partly correct answer.
- Single answer or short answer questions.
Single answer questions have only one correct answer. They are sometimes called “closed open”. For short answer questions students have to show their calculations. Their answer may be partly correct, for example, because they did not round off properly or forgot to mention appropriate units.
- Other open questions
The correct answer could still be just a number or a formula, but the activities necessary to find the answer are more complicated or involve higher-order strategies compared to single answer questions. However, the distinction between different types of open questions is an arbitrary one.
- Extended-response open problems
Students will have to give mathematical reasons to support their answer in a more elaborated way. Often the process is more important than the product. Many times it will be necessary to make a mathematical model of the situation first, and afterwards interpret and justify the results according to the original realistic situation.
- Multiple question items (super items)
These are tasks that enable the students to get involved with a context by asking a series of questions of increasing complexity and level of competence. The first question is often referred to as an ‘entry-question’, a simple level I question to explore the situation. The next questions are more difficult, preferably as well, on Levels 1, 2 and 3.
- Investigations, essays
This type of question is mainly used to assess higher order levels of competency. Students are asked to investigate, to draw conclusions or give proof, to reason mathematically about their findings or to use and criticize mathematical models.
- Two-stage task
A two-stage task may appear in different forms. The previous type of doing your own investigation, may be divided in two parts. The first part consists of questions (open or closed) about the problem, on different levels of competency. The results are assessed by the teacher and the student gets feed back. This way the teacher makes sure all students know what is expected from them and have showed whether they master the basic skills needed to perform the investigation. Moreover, it is possible to prevent students from taking wrong (side) paths that do not lead to a correct answer or that do not assess what was meant to be assessed.
Then the next part is done without a time limit and this part should complement the mathematical competencies that were missed in the first stage.
- Own productions
Students produce and solve their own problems. By doing this, it is possible for them to demonstrate a much higher level of thinking than the teacher expected. Moreover, by using this type of task, the teacher sometimes finds misconceptions that do not show when students answer the teacher’s question instead of their own.
The different formats will now be discussed in further detail. The order in which the different formats for questions, problems and tasks are shown does not mean this is an order from “bad” formats to “good” formats. It largely depends on the purpose of the test in which they appear and the mathematical skills a teacher wants to assess by using them. And, of course, sometimes there is not even a choice to be made. The test is compulsory.
Multiple choice questions
Not knowing how a student found the answer is characteristic for multiple choice problems. The only advantage of using multiple choice questions is the speed with which you can score them. The only reason to provide students with practice answering multiple choice questions may be that you find them in compulsory tests like standardized tests.
Students do need practice in answering multiple choice questions. Even reading them properly and finding out what exactly is asked can cause difficulties.
If you really want to know what kind of information you are missing about your students abilities, while using the multiple choice format, you might do a small experiment. Ask the students not only to write down the correct answer to the problem but also show all their calculations and their reasoning. You almost certainly will find, as we did:
- students who did all the calculations right but in the end took the wrong letter;
- students who got the right letter, but with a totally wrong explanation as the following example shows.1.
The radius of a cylinder is 1 centimeter. The height is 2 centimeters. A rectangle can be formed into this cylinder. Find the length of the rectangle.
The length of the rectangle is:
A 2 cm B 4 cm C 2 cm D 22 cm
Age: 13, 14
Content: space and shapes, 3D
Nathan: Answer A: diameter of the cylinder is 2 x radius and because a cylinder is round you need to add , so the answer is 2.
John: Answer A. I did this: area is radius x radius x , 1 x 1 = 2, so the answer is 2.
Babette: Answer A: You find area by calculating radius x radius x , which makes 1, and then x 2 for the height, 2.
Complete nonsense, indeed. But they got the right answer!
This example also shows that over-reliance on multiple choice questions creates the illusion that students know more than they really do. Use of simple probability predicts that just by guessing a lot of students will demonstrate they “know” the correct answer. Moreover, the exposure of possible answers guides students towards possible solution strategies whereas if the problem was presented without solutions students may not seem so clever!
You will also find students who just guessed, because they really think that is the proper way you solve a multiple choice problem, or students who solved the problem right but in quite a different way than was expected, for example by filling in all the answers in the equation instead of solving the equation.
Multiple choice questions create the illusion that students’ incorrect answers mean they understand little or nothing about the content addressed. When, in fact, students may know quite a bit but have been distracted or just made a small computational error. Calculating 3 x 3 = 6 instead of 9 is a popular one. But all incorrect answers are counted the same, a teacher gets no information whatsoever about the thought processes involved.
Some students solve the problem in a different way than was expected because they use their daily life experience. Sometimes the designers of the problem obviously do not share this experience.
In solving the next problem, some students used their shopping experience.2.
Which bottle of cleanser should a person buy to get the best quality for the least amount of money?
A regular size, 1 qt 25¢
B family size, ½ gal. 38¢
C giant economy size, 1 gal. 60¢
D all cost the same per quart
Age: 13, 14
Context is relevant, situation: daily life
Hedy: The answer is C, because the largest size is always the best buy.
That is a perfect reason. Any housekeeper will confirm that. But this is the reasoning the designers of the problem expected the student to do:
There are 4 quarts in a gallon or 2 quarts in a half-gallon, so the family size bottle at 38¢ costs 38¢ : 2 = 19¢ per quart. The giant economy size bottle costs 60¢ : 4 = 15¢ per quart. Therefore the giant economy size bottle is the buy that gets a person the greatest amount for the least money.
A lot of work for the same result!
Using common sense instead of difficult calculations is an attitude we like to see in students. It often proves to be an effective method as the next example shows.3.
The difference between one-half of a number and one-fifth of it is 561. The number is:
Age: 13, 14
Barry wrote: The answers A and E are far too small and D is too big. Half of the number 2805 (B) is not a whole number so the correct answer is C.
Sometimes the easiest way to find the correct letter, is to look at the answers instead of the problem. That is not the way the problem should be solved but as stated before, by using this format for a question, students are encourage to find all kind of ways to answer it in a simple way that the designers did not really mean to.In that case it is not certain whether you assessed students’ “test-wiseness” or their understanding of mathematics.
A teacher gave us the next problem and he told us:
“Teaching how to tackle multiple choice problems takes a lot of my time because I want my students to do really well on the test and it takes a specific way of thinking to do those problems well. So I tell them to look at the answers and try to work ‘backwards’. If you do so, always start with D, because they want you to try all of them. So be smart, most of the time it is C or D! Yes, of course this is ‘teaching to the test’, but it wasn’t me who chose these tests!”4.
What percent of 36 is 27?
A 36% B 50% C 66¾% D 75%
Merel: 27 is over one-half of 36, so A and B are not the right answers. So it is C or D.
C is a difficult one, so I choose D.
75% equals ¾. One fourth of 36 is 9 and 3 x 9 is 27. Got it!
Sometimes the question itself is not correct. The next problem is mathematically wrong! But even so, students are expected to find the right answer.5.
If a – 1 = 3 and a + 1 = 7, then a2 – 10a =
A - 24
Good multiple choice problems are difficult to design. There are occasions where multiple choice problems are a good format to use, as the next question shows. Posing this one as an ‘open’ question, may result in many halfway correct answers, or answers that are difficult to understand. For this problem, we used the, sometimes cryptic student answers for the ‘wrong’ alternatives.6.
The tv chart for the weather forecast shows:
What does a probability of 30% that it will be raining tomorrow mean?
A 30% of 12 hours is about 3½ hours, so we will have 3½ hours of rain tomorrow.
B 30% is less than ½, so we will have rain tomorrow for less than half the day.
C 30% is less than 50% so more likely than not we will have a dry day tomorrow.
D You cannot tell because the weather forecast is often wrong.
Age: 13, 14
Content: statistics, information processing and chance
Context is relevant, situation: daily life
Summing up our findings in this chapter:
- Multiple choice problems do not give you insight in the way students find their answers
- Multiple choice questions are found in standardized tests, so we cannot avoid them.
- In answering multiple choice questions it proves useful to use ‘common sense’.
- Multiple choice questions can be answered correctly by using wrong reasons without it being noticed.
- Good multiple choice questions are hard to design.
- Multiple choice questions may be answered incorrectly due to minor computational errors.