NOTE: Dr. Olson’s comments are in RED.
Forum 7: How difficult should a test be?Frary, in his article, argues that a test, to be useful, should be at least moderately difficult. Restate his argument, briefly, in your own words. Then take a position with respect to classroom assessments. Does his argument apply? If your answer is yes, explain why. If your answer is no, then does any of his argument apply? Explain your position.
Reply by Kathy Courtemanche on February 18, 2010 at 7:43pm
Basically, Frary (1995) asserts that easy questions should be eliminated from tests because almost everyone will get those questions correct anyway. He believes that moderately difficult tests will emphasize the differences BETWEEN students which is useful for ranking them.
Frary (1995) stated he was not discussing tests for measuring the degree of achievement as in criterion-referenced tests, but then went on to say, "What we are concerned about are tests covering fairly diverse topics... Almost any multiple-choice midterm or final examination would be in this category..." Why wouldn't a midterm or final be criterion-referenced? He's confusing me.
I am thankful he is not my professor. I believe his method would be inappropriate and detrimental for classroom assessments. My classroom assessments are to gain criterion-referenced data regarding a student's mastery of the material. I'm not interested in ranking students or elucidating their differences. My goal as an educator is to have every student learn the material we have covered, and be able to demonstrate their mastery. Only presenting the most difficult questions would cause students to become discouraged and possibly give up.
Why wouldn’t you be interested in differentiating among students’ levels of performance in your classes? Isn’t it important to know who has “just” mastered the course competencies and who has achieved beyond what is minimally acceptable? Or who just missed “mastering” the content and who really missed mastering the content? As I mentioned in class, scores kids get on tests do not need to be the grades they get on those tests. In other words, if a kid gets a 75 on a test that 75 could be converted to an “A,” a “B,” a “C,” and so on. Do you really believe that scores on tests truly reflect kid’s level of achievement? After all, the difficulty of a test is under your control. You can make a test as easy or as difficult as you want. If two kids, from two different classrooms with different teachers, both attain a “B” on, say, an English test, does that mean they both attained the same level of achievement?
Reply by Charity Nightingale Horton on February 19, 2010 at 8:21pm
I agree Kathy.....I think his views are very inappropriate as well!
Reply by Sharon Creasy on February 21, 2010 at 5:14pm
If the passing mark is moved to the middle, why would this be so bad? It aligns more accurately with grading of EOG's. I think this dilemma is one of the reasons some teachers use total points. I agree that a assessment can begin with a few simpler warm up questions but I do think a test can be rigorous without being impossible. I always liked the bonus question on a test - if you knew it, great - if not - oh well.
I hope to convince you, later, that bonus questions are not appropriate when assessing achievement of learning targets. Teachers do this because they want kids who have not mastered the LTs to pass.
Reply by Brenda Little on February 21, 2010 at 8:17pm
Kathy: I think it goes back to why your are assessing. Sometimes we may need ranking data and other times we are assessing for mastery. I think the students need to understand what is being assessed and often teachers are not good at explaining the goals of each assessment.
Why can’t a test of score of, say 75, represent a minimal level of mastery? And another score of, say, 85 represent achievement well above mastery.
------
Reply by Brenda Little on February 18, 2010 at 9:19pm
Frary (1995) makes a case for test difficulty....specifically test items being moderately difficult. He takes you inside the box of assessment only to give you a different (or new) perspective on developing tests in which scores are determined by adding up points for each right answer (i.e.; multiple-choice, short-answer, & problem-solving tests). Frary poses a great question: why develop tests items that you know 90% of the examinees will answer correctly? What real information will you gain about each student’s ability to achieve the curricula aims that have been identified for this unit of study using this method?
Frary goes on to say that to get to the point of a more accurate class rankings, teachers must develop tests with more difficult questions. The result of developing more difficult tests will result in a wider spread of scores which in turn reflects a fairer test. Frary states that the most desirable tests have a few easy questions to encourage students and some really hard questions to help students realize their own capabilities and limitations.
I think its how tests are presented by teachers and perceived by learners – a test is an instrument that shows the teacher what I don’t know. Educators need to move away from this negative connotation. We need to present tests as a teaching strategy that documents where students are on the learning continuum. Once this is established, teachers will know what material needs to be covered in order to develop the full scope of the curricula aim. By developing tests that are challenging (not intimidating); the teacher will gain more knowledge about what the student understands and where additional learning needs to occur. Teachers must keep in mind, however, that test difficulty does not equate to tests that use tricky words, unfair choices, and/or grading boundaries. Tests are not the enemy of learning…unfair interpretations of the tests set the true boundaries.
Brenda, I really like your response. You have read Frary correctly. As you kind of point out, the problem is with teachers not understanding test difficulty. (See my responses to Kathy, above.)
Reply by Charity Nightingale Horton on February 19, 2010 at 8:23pm
I like the "enemy of learning" phrase! It is unfortunate that tests could be, or are, viewed like this.
Reply by Sharon Creasy on February 21, 2010 at 5:22pm
I agree with you Brenda about the test "perspective". I think it extends to all assessments. If assessments were used more reflectively in teaching instead of pass out the grade and move on, the perspective might change. Really talking about the positive and negative outcomes would empower the assessment instead of its only legacy being the "grade result". We all know that we learn the most from our mistakes but only when we take the time to look at what went wrong and how we can change it. This is the missing step in perceiving assessments as learning opportunities.
Reply by Tammy Essic on February 21, 2010 at 9:19pm
Sharon, you are so right about learning the most from our mistakes. Unfortunately, so many teachers (and I've been guilty before!) show the students their score without ever taking the time to discuss the mistakes.
As some of the authors of the articles you have read for this class have pointed out, giving constructive feedback following assessments is of critical importance. How else are kids to learn from their mistakes?
------
Reply by Sharon Creasy on February 18, 2010 at 10:28pm
Frary (1995) posits that passing levels at the 70% mark are much too high for high school or college multiple-choice assessments covering broad, detailed content. A teacher has control over a test’s level of difficulty so how can it accurately reflect the student’s comprehensive knowledge of the content being assessed. An appropriate score demonstrating a broad understanding should be in the 50% to 60% range. Frary details that a more challenging test is a fairer test when the spread of grades is greater. He argues that it is a waste of effort, time and resources to pose questions that can be answered correctly 90% of the time other than a couple of moral boosting items.
My college age daughter has had to adjust to a very different passage rate in her undergraduate science classes – organic chemistry for instance. She has had tests where the average grade among 200 students is a 47. A positive outcome of this grading structure is that she has become more focused on doing her personal best and less focused on a number that relates to an alpha outcome. I am not saying that she does not care about her grades - it is just that never knowing what the high grade will be means just doing your best.
I’m not so sure that not knowing your grade is the best approach. In my opinion it would be better to provide students with some guidelines like: 51-60 = C, 61-75 = B, >75 = A. The guidelines, of course would depend upon the test difficulty. Also, it is always possible to
Yes, Frary’s argument applies to the classroom. Time is precious for all of us including students. Spending time on questions that are not assessing understanding or learning by the student is wasteful and a failure to achieve rigor. Next, moving the classroom assessment range closer to standardized testing ranges might create some correlation and understanding by stakeholders of difficulty and below sixty cut scores. Judging question correct response percentage in creating challenging tests would be an ongoing task for teachers but with the use of multiple-choice software and tools, doable.
Reply by Sara Gilliam Crater on February 20, 2010 at 2:44pm
Yes your daughter's personal best should be what she is worried about but if the average is 47, how many does that leave passing the course?
How do we know a pass isn’t 30%? In which case Sharon’s daughter is doing very well, indeed!
Reply by Sharon Creasy on February 21, 2010 at 5:25pm
The performance of the class determines each grading range. It is such content heavy material they cannot know it all as well as the assessments are intended to be challenging.
------
Reply by sharonandersonjones on February 19, 2010 at 12:24am
Most teachers say a grade of 70% or better is a passing grade. This grade is based on making a list of course objectives and writing test questions to formulate a test. Teacher made tests can be unreliable and does not guarantee a test will not be too hard or too easy to be valid.
Teacher tests get better with experience making test items. As tests improve the criteria improves and the percentage of passing scores can drop from 70% to 60% for most students. Scores are derived by adding up points for right answers. A good strategy is to add a few easy questions at the beginning to encourage student achievement.
Again, the difficult of a test is under teacher control. Also, most teachers (and most professors) write terrible test items. Writing items to assess high level skill is very difficult. Also, remember the influence of CIV.
------
Reply by Cynthia Barber on February 19, 2010 at 5:08pm
In his article Frary asks the question why give students tests filled with easy questions. He asserts that everyone will get them correct and the assessment will therefore not provide useful information about students. Frary advocates that when devising an assessment the teacher should aim to include more moderate to difficult items in order to gain more useful information. His definition of useful information seemingly only includes the development of class rankings, determining the differences between students level of knowledge as displayed on an assessment.
When I give an assessment I am not concerned with ranking my students, especially in elementary school. I am more concerned with whether or not the students have learned the material which I have been required to teach. In developing assessments my goal is to make them fair. I include some easier questions not because I want to give an easy test, but because I want my students to feel some sense of confidence when taking the test. I have seen it over and over again. If a student encounters several difficult items at the beginning of the test, they can become discouraged and give up on the rest of the test. I am not advocating that you lump all “easy” questions at the beginning either, but there must be some balance when writing the tests.
I believe we must be honest with our students and not give them a false impression that a test is going to be easy. We have to educate them in ways to handle difficult test questions by teaching them how to take a test, so they will not be filled with anxiety at the mere mention of the word.
Reply by Charity Nightingale Horton on February 19, 2010 at 8:25pm
Being fair is the key!
Reply by Kristi Gaddis on February 21, 2010 at 11:20am
Cynthia,
I agree with your statement that tests should be balanced. I think all levels of difficulty should be present. You want to assess what the student knows without creating a sense of dread and anxiety. I think formative assessments can be short and have all levels of difficulty present. I agree!
Reply by Brenda Little on February 21, 2010 at 8:36pm
Cynthia: I agree with the fact that we must be honest with the students. I don't think teachers spend enough time talking about tests and how the outcomes of the test will be used....which leads to anxiety for the students. They never truly understand the purpose of the assessment.
------
Reply by Tammy Essic on February 19, 2010 at 5:17pm
In his article, Frary (1995) suggests that traditional grading practices that typically maintain a 70% pass rate are arbitrary and not as effective for ranking students as tests that are more difficult. According to Frary, an average score range of 50% to 60% is optimal for effective testing because more ranking information can be gained. Frary also suggests that test items that can be answered correctly by more than 90% of students should be omitted in most cases because these questions are wasteful and provide little information that is useful.
I would not fully apply his argument to classroom assessments. I agree that studens should be challenged and teachers should have high expectations for students, both during instruction and assessment; however, a classroom teacher should be less concerned with ranking his/her students and more concerned with assessing the depth of understanding that each individual student has. Assessing that depth of understanding should include a range of question types and level of difficulties. An additional concern with applying his argument is the negative impact it would have on most students over time. If students are only able to answer half of the questions correctly on a regular basis, motivation will decrease and doubt in abilities will increase.
I think that Frary’s comment about “ranking” students was just a poor choice of words. He is not concerned with ranking students for the purpose of assigning grades. He wants teachers to be able to differentiate among students’ levels of achievement. Test difficulties and grading practices are separate concerns. (See several of my comments given earlier.) There is no reason why difficult tests (i.e., tests with a lot of ceiling) would not work well in the classroom. In some cases, difficult tests might give some of the (much) higher performing kids a chance to shine.
Reply by Charity Nightingale Horton on February 19, 2010 at 8:26pm
I agree....students need to feel successful some of the time in order to gain self-esteem and motivation for the next time!
Reply by Kathy Courtemanche on February 20, 2010 at 12:20pm
I also agree. I believe Frary's ideas do not work well for classroom assessment.
And we think we have a high drop out rate now!
Well, I suppose it would be difficult for students (and parents) to feel comfortable with a test grade of, say, 40, even if it was pointed out that the test was very difficult. This would take educating students and their parents. It might help if test scores were converted to letter grades (something we’ll talk about later in the course.)
Reply by Kristi Gaddis on February 21, 2010 at 11:21am
Yes, I can see how this could increase drop out rates as well!
Reply by Sharon Creasy on February 21, 2010 at 5:34pm
Maybe I am wrong, but I did not interpret Frary's use of "ranking" as ranking the students in an order but of just identifying where they are in their "level of achievement" on the scale of correctness.
Sharon, you got it exactly right. See my comment to Tammy Essic, earlier.
------
Reply by Charity Nightingale Horton on February 19, 2010 at 8:20pm
Frary’s Stance:
Basically, Frary (1995) does not support traditional grading practices which involve accepting test scores of 70% as “passable” levels of achievement. He figures that maximum testing effectiveness occurs when the average score is in the 50%-60% range, and further supports his belief with some fancy mathematical magic.
This article confused me. Or perhaps FRARY is confusing! Is he suggesting it is beneficial to change the grading scale so that 50% and 60% are passing grade, or is he suggesting that teachers make tests harder so that students are achieving within a 50-60% range for his presumed optimal learning range?
Again, as I indicated several times above, grading is a separate issue. What if a test was constructed such that 50% or 60% percent of the items covered all the content that is sufficient to indicate a minimally passable score; and additional items were included to capture the performances of kids who went well beyond what was minimally acceptable?