Scorer- Automatic Test Scoring System

SCORER- AUTOMATIC TEST SCORING SYSTEM

REFERENCE CARD

The SCORER system uses the seven-response NAU General Purpose Answer Sheet. These forms must be filled in using a number 2 pencil. Be sure all erasures are complete. The instructor must complete at least one of these sheets (called Point Sheet 1) when using SCORER. The fields on this sheet (Point Sheet 1) must be completed as follows:

COMPLETING TEST KEY (ANSWER SHEET)

LAST NAME, FI, MI- enter the title which you wish to have printed on your SCORER reports.

DEPT NAME- two or three character department code (e.g., CIS).

COURSE NUMBER-three digit course identifier.

VERSION NO- enter the section number for the course. If more than one section is being scored in one batch, this number should be left blank on Point Sheet 1. The system will then pick it up from the individual student forms.

SOCIAL SECURITY NUMBER- enter 9 ‘1’s for Point Sheet 1 (e.g., 111111111).

INST NO- Leave blank.

SEAT NO- Leave blank.

EXAM NO- ALWAYS leave blank, unless you are scoring separate versions of the same exam. If separate versions are being scored, fill in the number of this version (e.g., 1 for the first version, 2 for the second, etc.).

QUESTIONS 1-200- fill in the correct answer to each question. (Note: If your questions do not all have the same point value, you will be using more than one point sheet. Refer to the SCORER User Documentation before filling in any answers).

COMPLETING STUDENT FORMS

Have your students fill out their forms as follows:

LAST NAME, FI, MI- Student’s last name, first initial, and middle initial.

DEPT NO, COURSE NUMBER- Leave blank. The system will pick up this information from Point Sheet 1.

-1-

SECT NO- if only one section is being scored in this batch, leave this field blank. The system will pick it up from Point Sheet 1. If more than one section is being scored at a time, students should fill in the appropriate section number.

SOCIAL SECURITY NUMBER- student’s ID number

INST NO- Leave blank.

SEAT NO- Leave blank, unless the class has specific seating arrangements.

VERSION NO- ALWAYS leave blank unless more than one version of the exam has been distributed to the class. If multiple exam versions are given out, have the students fill in the number of their version.

QUESTIONS 1-200- Students should fill in their choice(s) for each question.

POSSIBLE POINT SHEETS

Point Sheet 1- always required.

Point Sheet 2- (optional) required for tests with at least two different question point values.

Point Sheet 3- (optional) required for tests with at least three different question point values.

Point Sheet 4- (optional) required for tests with at least four different question point values.

Point Sheet 5- (optional) required for tests with at least five different question point values.

CONTROL SHEETS

Control Sheet 6- (optional) used to indicate which subscale each question belongs to.

Control Sheet 7- (optional) used to indicate question types, omitted questions and questions for which all students should be given credit.

Control Sheet 8- (optional) used for penalty scoring.

-2-

SCORER User Documentation Revision 3.1

January, 1999

CONTENTS

Introduction

SCORER User Documentation Revision 3.1

January, 1999

INTRODUCTION

SCORER is the computerized test-grading system available to the NAU academic community through Information Technology Services.

SCORER was authored in the COBOL language by Mark W. Michrina at the Indiana University in Pennsylvania (IUP), Indiana, PA, in 1982. The basic SCORER system has remained unchanged since then and has been adapted to several different machines.

The Academic Computing (ACAD) group of Information Technology Services maintains SCORER, provides education for academic personnel wishing to use the system, helps to resolve any processing problems and/or conflicts and, when requested and appropriate, captures SCORER data for use by academic personnel. The Operations group provides the primary handling and processing of the tests to be scored.

Despite its simplicity of use SCORER is powerful and diversified. It allows for multiple versions of a particular exam. Common statistics are calculated for each exam version and for the overall exam. They are also calculated for each different class session taking the exam. Questions may be of the single- or multiple-response type. The multiple-response questions can be an exact-match answer or a subset answer. Score adjusting, subscale processing, question omission, and question credit for all students are available. Many question processing options may be weighted. Penalty scoring is available.

Some reports are generated automatically; others must be requested. Reports are available for a class, sections within a class, versions within a class, and version within sections of a class. Report sort options are available.

As you can see, SCORER can take you from the simple to the very complex—your choice.

SCORER User Documentation Revision 3.1

January, 1999

SECTION 1

THE FORM (GPAS)

NAU’s General Purpose Answer Sheet (GPAS) is the form on which students mark their responses to test questions and on which the instructor marks the correct answers—and other data needed in grading a test. This is done by using a pencil with No. 2 or softer lead to darken specific circles on the GPAS. These circles are commonly called “bubbles.” GPAS filled out by the instructor with information used for test scoring purposes are referred to as either POINT or CONTROL SHEETS. The GPAS provides the instructor with up to 200 possible questions, each with up to 7 possible answers. The area on the GPAS containing the information that identifies each sheet is divided into fields. There is a field for the NAME, SOCIAL SECURITY NUMBER, DEPT NAME, COURSE NUMBER, INST. NO., SEAT NUMBER, and VERSION- NO. Collectively, these fields are referred to as the Heading Information of the GPAS.

SCORER allows for multiple versions of a particular exam. An instructor may have two or more versions of an exam scored in one batch without separating the answer sheets by indicating in the VERS NO. field in the Heading Information on all POINT and CONTROL SHEETS which exam version each of the SHEETS belongs to (version number and EXAM NO. are used interchangeably in this documentation and have the same meaning). Different exam versions consist of the same exam questions shuffled around into different orders to help reduce the possibility of students cheating. A word of caution—if the instructor scores more than one exam version at a time, all students must mark the correct version number on the GPAS or SCORER cannot function properly. Each exam version must have the same number of omitted questions. The instructor will need to fill out a separate set of POINT and CONTROL SHEETS for each exam version.

1-1

SCORER User Documentation Revision 3.1

January, 1999

SECTION 2

POINT SHEETS

POINT SHEETS are used to indicate to SCORER the point value of a correct answer to a question or, in the case of a weighted question, the point value of a particular response to a question.

A POINT SHEET is indicated by placing one of the following five values in the SOCIAL SECURITY field in the Heading Information.

111111111

222222222

333333333

444444444

555555555

Up to 5 different point values can be used per exam. For example, on a 20-question test where the first 10 questions are worth 2 points each and the last 10 questions are worth 4 points each, the instructor would fill out POINT SHEET 111111111 with the first 10 answers and POINT SHEET 222222222 with the last 10. The instructor then indicates on the Parameter Sheet that POINT SHEET 111111111 has a value of 2, while POINT SHEET 222222222 has a value of 4.

Of all POINT SHEETS, only sheet 111111111 is required. If a test is created in which all questions have the same point value, POINT SHEET 111111111 must be filled out with the correct answers and the Parameter Sheet must indicate the point value.

The instructor may override any student information entered in the following fields by filling the fields with the desired information on POINT SHEET 111111111:

DEPT NAMECOURSE NUMBERSECT. NO.INST. NO.

On POINT SHEET 111111111, SCORER reads all Header Information fields. On POINT SHEETS 222222222 through 555555555, only the SOCIAL SECURITY and the EXAM NO. fields are read; all other Heading Information is ignored.

2-1

SCORER User Documentation Revision 3.1

January, 1999

SECTION 3

CONTROL SHEETS

CONTROL SHEETS are used to tell SCORER how to score the test. A CONTROL SHEET is a GPAS that has a special value in the SOCIAL SECURITY field of the Heading Information. These values range from 666666666 to 888888888. The simplest test has no CONTROL SHEETS, while the most complex has three CONTROL SHEETS. CONTROL SHEETS are used to indicate in which subscale(s) a particular question is to be scored, which type of multiple-answer question is being scored, whether a question is to be omitted from the test scoring process, whether all students are to be given credit for a question, or how many choices there are for each question that is involved in penalty scoring.

SCORER reads only the SOCIAL SECURITY field and the EXAM NO. field of the Heading Information on all CONTROL SHEETS. Instructors may fill in the other fields for their own use.

Sections 4, 5, and 6 will discuss the three types of CONTROL SHEETS and their uses.

3-1

SCORER User Documentation Revision 3.1

January, 1999

SECTION 4

SUBSCALES

Subscales can be used to tally student responses to different types of questions. For example, an instructor can determine from using subscales if students perform better on factual or conceptual questions, true/false or multiple choice questions, questions based on text or on the lecture, questions based on older material or newer material, etc. Subscales can also be used to measure the validity of a test by comparing the results of the first half of the test with the last half, the even-numbered questions with the odd-numbered ones, etc.

Subscales are indicated by using a CONTROL SHEET which has 666666666 in the SOCIAL SECURITY field of the Heading Information. There are eight different subscales available for each question which correspond to the letters A through G and the blank (no bubbles darkened). For a question to be tallied in a particular subscale, the subscale for that question must be marked on CONTROL SHEET 666666666 at that question number. For example, if question number one is to belong to the two different subscales A and F, then for question one on CONTROL SHEET 666666666 bubbles A and F should be marked.

Reports are available for subscale breakdowns by section, exam, version, and by the individual student’s performance. To get SCORER to print the reports the instructor must indicate on the Parameter Sheet that they are desired.

4-1

SCORER User Documentation Revision 3.1

January, 1999

SECTION 5

MULTIPLE ANSWER QUESTIONS

SCORER provides for scoring multiple-answer questions. This offers a variety of options in creating a test to suit all of an instructor’s needs. To score multiple-answer questions, a CONTROL SHEET is filled out containing 777777777 in the SOCIAL SECURITY field of the Heading Information. A question marked on this sheet indicates one of the following three things:

1. This is a multiple-answer question:

1. with a Single Bubble Response(A)
2. with Multiple Bubble Response-Exact Match(B)
3. with Multiple Bubble Response-Subset(C)
4. with Multiple Bubble Response-Adjusted Score(D)

2. This question is to be omitted(F)

3. All students are given credit for this question(G)

If a 777777777 sheet is included with the test to be scored and a question is not marked on the 777777777 sheet then for that question, only one answer may be bubbled on the POINT SHEET, i.e., if a 777777777 sheet is included, not all questions need to be of the MULTIPLE ANSWER type.

SINGLE BUBBLE RESPONSE QUESTIONS

ANSWERS OF THE SAME POINT VALUE

The first type of multiple-answer question limits the student to a single bubble response. If there is more than one correct answer to a question with each answer consisting of only one bubble and each answer being worth the same points, the instructor would mark all of the correct answers on a single POINT SHEET and darken in the A bubble on the CONTROL SHEET 777777777. In this manner, a student marking any ONE of the correct answers is awarded the specified amount of points.

5-1

SCORER User Documentation Revision 3.1

January, 1999

WEIGHTED ANSWERS

If each answer to a multiple-answer question with a single bubble response is to be awarded a different point value, the question is said to be weighted. To fill out the CONTROL SHEETS for this type of question, the instructor must darken the bubbles on the POINT SHEETS (111111111 through 555555555) which correspond to the weighted responses and CONTROL SHEET 777777777 must have bubble A darkened for that question. Bubble A indicates that this question is of a single bubble response type. For example, if question five is a weighted question with response A worth five points, response C worth 3 points, and response F worth 1 point, the POINT SHEETS and the CONTROL SHEET will be filled out as follows: on POINT SHEET 111111111, bubble F would be darkened for question five; on POINT SHEET 222222222, bubble C would be darkened for question five; on POINT SHEET 333333333, bubble A would be darkened for question five, and on CONTROL SHEET 777777777, bubble A would be darkened for question five. (The point value to be assigned to POINT SHEETS 111111111 through 333333333 is handled on the Parameter Sheet). This combination of four sheets indicates that for question five: if the student marks bubble A, 5 points are awarded; if the student marks bubble C, 3 points are awarded; if the student marks bubble F, 1 point is awarded. Any other response by the student, or multiple responses, yields zero (0) points. See the following example.

1. The National Debt in 1981 was:

1. Close to $1 TrillionB. Zero

C. 30% of the GNPD. Five dollars

E. Six DollarsF. Very Large

Here the National Debt was indeed approximately $1 trillion. Therefore, answer A is awarded 5 points. 1 Trillion dollars was also 30% of the GNP, so answer C is awarded 3 points. The National Debt was also very large, so answer F is awarded 1 point. Keep in mind that a student may mark one and only one response on this type of question.

5-2

SCORER User Documentation Revision 3.1

January, 1999

MULTIPLE BUBBLE RESPONSES-EXACT MATCH

The second type of multiple-answer questions is a question for which each possible answer requires a combination of two or more bubbles. These answers can also be weighted by placing the different answers on different POINT SHEETS. To indicate this type of question, the B bubble is marked on CONTROL SHEET 777777777 for this question. For example, if B & C is worth 3 points, and C & D is worth 1 point, the POINT SHEETS and the CONTROL SHEET would be filled out as follows: bubbles B & C would be darkened on POINT SHEET 111111111; bubbles C & D would be darkened on POINT SHEET 222222222; and, bubble B would be darkened on CONTROL SHEET 7777777777. See the following examples.

1. Which two of the following items make up the best breakfast combination?

1. MacaroniB. Bacon

C. EggsD. Toast

Here, B & C are the desired answers, so they are worth 3 points. However, if the student marked C & D, 1 point would be awarded because this breakfast is acceptable, but is not as good as B & C.

1. Match the author in Column 1 with the novel in Column 2.

(1)(2)

01) DickensAB) Three Musketeers

02) TwainAC) Grapes of Wrath

03) HawthorneAD) Madam Bovary

04) FieldingAE) Huckleberry Finn

05) DumasAF) Tess of the D’ubervilles

06) SteinbeckAG) Golden Bowl

07) HardingBC) Farewell to Arms

08) HemingwayBD) Oliver Twist

09) BalzacBE) Tom Jones

10) JamesBF) Scarlet Letter

This is an example of a set of matching questions that show when two letters would be necessary for a correct answer. This type of question also shows that weighting is not necessary

5-3

SCORER User Documentation Revision 3.1

January, 1999

for a question of this type. Using two letter responses, a maximum of 21 choices in a single matching list may be created. Using three letter responses, a maximum of 35 choices in a single matching list may be created.

MULTIPLE BUBBLE RESPONSES-SUBSET ANSWER

The third type of question is one where the student must answer the question by darkening in a subset of the bubbles that the instructor has darkened. Any bubble that the student darkens that the instructor hasn’t results in zero (0) points for the student for that item. The instructor may mark a group of bubbles on each POINT SHEET. To indicate a question of this type, bubble C is darkened on CONTROL SHEET 777777777. For example, students will receive 5 points if they darken in bubble A, B, or C, or any combination of A, B, & C, or the students will receive 1 point if they darken in bubble D, E, F or G, or any combination of D, E, F & G, the POINT SHEETS and the CONTROL SHEET 777777777 would be filled out as follows: bubbles A, B, & C must be darkened on POINT SHEET 111111111 and D, E, F & G must be darkened on POINT SHEET 222222222 and C must be darkened on CONTROL SHEET 777777777. See the following example.