Torment II Version 118D User Manual Error! Unknown Switch Argument
Torment II Version 118d – User Manual Error! Unknown switch argument.
Torment II
A Program for Processing Multi-Choice Exams.
User Manual
Software Version 1.18e
Document Version 1.1
Dick Martin,
I.T.S., University of Otago
20 February 2001
Revised 14 June 2002
Torment II Version 118d – User Manual Error! Unknown switch argument.
Contents
History ...... 3
Introduction ...... 4
Processing Options ...... 5
The Report File ...... 7
A Simple Example ...... 12
A More Complex Example ...... 16
Things to Be Aware of ...... 20
Appendix A — Parameter File Specifications . . . 21
History
TORMENT was originally written, mostly in Pascal, for the VAX VMS operating system. Because of the likely demise of this operating system on our campus in the near future the program has been entirely re-written and is now in a form which enables it to run with minimal changes on many different types of desktop computer running such operating systems as MS-DOS, MS-Windows, Linux and MacOS. This new version is now called TORMENT II.
TORMENT was used to process responses from multiple-choice exams and the data was usually keyed in manually. TORMENT II performs this same basic task, but the responses are now entered by the exam candidates themselves onto electronically readable forms. These forms are scanned by an OMR (Optical Mark Reader) scanner which saves the data file to disk. Some additional facilities were incorporated into TORMENT II to deal with issues introduced with OMR scanning but otherwise it does the same task as the earlier program — processing of marks from multiple choice exams and reporting those marks, any errors or inconsistencies encountered, and an analysis of the marks to highlight both well framed questions answered as expected by candidates or questions which were answered with difficulty.
Several ancilliary programs have also been written which allow manipulation of data file formats etc. Normally these programs, and indeed TORMENT II itself, are run by I.T.S. staff rather than the examiner, but examples will be given below of different data file formats to indicate the range of possible conversions.
Introduction
TORMENT II uses three input files:
Data containing the Student ID and responses for each question, the responses for each examinee occupy one line. The first line of responses contains a special ID of 0000000 identifying the correct answers. This file is frequently derived from one produced by an OMR scanner if the multiple choice responses are electronically scanned. It is then pre-processed by an ancilliary program to strip extraneous values and put it into the appropriate form for TORMENT II. The scanning of the multiple choice response forms is normally done by I.T.S. staff.
Names containing a list of names and Student ID's. One line of data for each examinee. This file is normally obtained from Student Records.
Parameter containing some basic information, such as the names of the data and names files, the number of questions to be marked, and a list of options to be applied to the questions such as mark to be subtracted for wrong answers. This file is normally prepared by the I.T.S. staff member running the program, from information supplied by the examiner.
The Data file contains only the Student ID, as recorded on the multiple choice response form. The Names file supplies the relationship between these Student ID's and their real names. The Parameter file contains a list of keywords and related fields which define the various options for this exam and thus defines the marking schema. The permitted options are discussed in the next section.
These three files are processed and a report file generated. Normally an examiner would see only this report file.
Figure 1. File Relationships
Processing Options
The various processing options are set via the Parameter file. The syntax for this is presented in Appendix A, but this would normally not need to be known by the examiner. The examiner generally only needs to indicate his requirements in English and these directions will be translated into appropriate parameter statements by whoever runs the TORMENT II program. The detail provided here is intended to indicate the range of possibilities.
Each option in the Parameter file is specified by a keyword followed by one or more arguments, specifying the question number or range of question numbers and/or some other text or numeric item.
For example, the option for specifying whether marks are to be subtracted for an incorrect answer is specified as:
MARKSOFF Qy-Qz n.nn
where Qy-Qz specifies the question number or range (e.g. Q4 or Q17-Q23) and n.nn specifies a numeric value, with up to 2 decimal places, to be subtracted from the examinees score if the answer is incorrect (e.g 0.25).
So a typical entry in a parameter file might look like:
MARKSOFF Q4-Q18 0.2
The examiner's request, which resulted in this parameter statement, may have been expressed as "candidates should have 0.2 marks subtracted from their score for each incorrect answer".
And the description in this section of the User Guide would specify this as
MARKSOFF question_range value_to_subtract(0.00)
Some keywords expect a text option, such as DATAFILE which expects the name of the datafile to given after the keyword, as in:
DATAFILE TOUR214-feb-SS
and this would be presented below as
DATAFILE text
Parameter Descriptions
COMMENT text
A comment line
Example: COMMENT TOUR214 Summer School 2001
DATAFILE text
NAMEFILE text
REPORTFILE text
The names of the datafile, names file and report files respectively
Example: DATAFILE TOUR214-Feb-SS
QUESTIONS value(000)
The number of questions to be marked for this exam. May be smaller than the actual number answered but not greater than.
Example: QUESTIONS 80
VALID question_range number_of_valid_options
The number of valid options for the given range of questions. If options a to d were valid for questons 17 to 25 then this would be expressed as:
VALID Q17-Q25 4
UNMARKED question_range
species a range of questions to be unmarked in this exam. This might be because of a decision made after the exam was set (or sat) to exclude this question or questions.
Example: UNMARKED Q17
SCALING maximum_mark_to_scale_up_to(000)
The maximum possible mark after scaling, a whole number. The example expresses the marks as a percentage.
Example: SCALING 100
MARKSOFF question_range value_to_subtract(0.00)
Specifies the marks to be removed from an examinees score for an incorrect answer.
Example: MARKSOFF Q12 0.25
WEIGHTING question_range value_to_multiply_by(0.00)
The amount the scores for specified questions should be multiplied by.
Example: WEIGHTING Q18-Q23 2.5
CORRECT question_range
A question (or questions) to be scored as 'correct' irrespective of the examinees response. This (as with UNMARKED) may result from problems with the content of an exam question which were not able to be corrected prior it being sat.
Example: CORRECT Q4
MANSWER question_number alternative_response alternative_response
Specifies several alternative correct responses for the given question. This (as with UNMARKED and CORRECT) may result from problems with the content of an exam question which were not able to be corrected prior it being sat.
Example: MANSWER Q12 a d
The Report File
The report file has a name of the form xxxxxrep.txt and is originally a text file. TAB characters are used extensively in tables etc. and correct alignment of columns in tables (as in examples below) is dependent on your local TAB width setting.
The report file is arranged in sections which closely echo the original sections of the original TORMENT reports. In general the differences are minor, if you are familiar with the original TORMENT you will be comfortable with the reports from TORMENT II.
Report Sections
Parameter File Listing:
The report initially echoes back the parameter file contents verbatim.
Example:
TORMENT II - DEVELOPMENT VERSION, NOT FOR PRODUCTION USE.
EXAMS -- RUN TIME Thu Nov 23 15:35:33 2000
PARAMETER FILE -- [Orogeny:Pangaea II:EXAMS Perl Code:WORK FOLDER:M206-NOV.par]
COMMENT M206-NOV 2000
DATAFILE M206-NOV
NAMEFILE M206-NOV
REPORTFILE M206-NOV
QUESTIONS 40
SCALING 100
VALID Q1-Q40 4
EXIT
Warning Messages:
TORMENT II looks for certain types of error in the data as it reads the data file and these will be reported after the parameter file listing, if found.
Errors reported include
Duplicate ID's.
Invalid data (i.e. a choice outside the valid range of options for that question).
Invalid characters (a character which is not a letter).
Errors of this kind are likely to result from incorrect marking of forms, incomplete mark erasure or crossing out of mistakes and alert the examiner to the need to examine the responses indicated to confirm the intent of the candidate. The ID number is always reported along with the detected error. Note that if an invalid character is found it will be either a '*' or a '?' indicating that the candidate has crossed out or imperfectly erased a mark additional to the intended mark (see the example below) and the scanner has flagged this by inserting one of these characters.
Examples:
WARNING ****** duplicate ID 4008762 *******
WARNING ****** invalid ID 67z3420 *******
WARNING ****** invalid ID 795 866 *******
WARNING ****** invalid data associated with ID 795 866 ********
************************* 795 766CBDCCBDCCADBCBzACBCACBCCACACBADBACACCCAD
WARNING ****** invalid data associated with ID 672174 ********
************************* 672174 CDDCBBACDBDCCABACCCBBDCC*CACDACBACCAACBB
WARNING ****** invalid character - scanner misread? ID = 672174 ****
Listing of Name File:
This is a formatted verbatim listing of the names file containing the Student ID's and names of candidates.
Example:
Listing of Name File:
1245669 Nash, Walter
124566 Holyoke, Keith
3455778 Shipley, Jenny
5789566 Kirk, Norman
Listing of Answer File:
This is a formatted verbatim listing of the data file containing the Student ID's and exam responses. Note that ID 0000000 contains the correct answers.
Listing of Answer File:
Student Answers
0000000 AAABBCCABABCCDDDADACBAADDDBBDC
1245669 AAAABCCABDBCCDDDADACBCCDDDBBDC
124566 AAABBCCABABCCDDDAABABCCDDDBBDC
3455778 AAABBCCABABCCDDDADACBAADDDBADC
5789566 AAAABABCCDBCCDDDADADDAADDDABDC
Correct Answers:
A formatted listing of the correct answers for each question, i.e. the responses for candidate 0000000.
Example:
Correct Answers:
1 A
2 A
3 A
4 B
5 B
6 C
7 C
8 A
9 B
Marks Sorted by ID:
The final marks for each candidate, in order of ID. Additionally the number of correct, incorrect and unanswered responses are given. Invalid responses result from multiple responses to a single question where this is not permitted. Sometimes these result from imperfect erasure or crossing out of mistakes.
A candidate who is not in the names file will have their name shown as ***************** in the body of the table and will have their ID listed after Missing Students (the next section) with the annotation 'not named'.
Example:
MARKS sorted by ID:
Student ID Mark Scaled Correct Incorr Unansw
Holyoke, Keith 124566 25 83.3 25 5 0
Nash, Walter 1245669 26 86.7 26 4 0
Shipley, Jenny 3455778 28 93.3 28 2 0
Kirk, Norman 5789566 21 70 21 9 0
Marks Sorted by ID:
The final marks for each candidate, in alphabetical order of candidate name.
Missing Students:
A listing of all students found in the names file for whom no records existed in the data file. The may be followed by an additional listing of student ID's not present in the names file (see above).
Example:
Missing Students:
1122456KELLY, NED ***** absent *****
223390 KID, BILLY THE ***** absent *****
Mark Summary:
A brief gross statistical summary giving the average mark for the class, the standard deviation, number of candidates in the class and number present, and the number of questions in the exam.
Example:
Mark Summary:
*** A V E R A G E *** = 25.3
*** S T D D E V *** = 3.3
Maximum Possible Mark = 30
Total in class = 4
Number present = 4
No. of questions = 30
Mark Analysis:
Is a frequency table in groups of 5 of the gross marks of the class.
Example:
Mark Analysis:
Mark Frequencies
%Range No. of
Students
100 0
95-99 0
90-94 0
85-89 0
80-84 0
75-79 0
70-74 1
65-69 1
60-64 4
55-59 3
50-54 9
45-49 3
40-44 9
35-39 2
30-34 2
25-29 0
20-24 0
15-19 0
10-14 0
5-9 0
0-4 0
Mark Histogram:
Is the visual equivalent of the above frequency table. Both the table and the histogram are provided so that the overall mark distribution can be seen, to identify problems with the exam mark distribution, aid scaling etc. Difficulties such as a strongly bimodal distribution of marks can be clearly seen.
Example:
Mark Histogram Distribution of Percentage Marks
P E R C E N T A G E O F C L A S S
%Range .1 . . . 5 . . . .10 . . . .15 . . . .20 . . . .25 . . . .30 . . . . .35 . . . .40
100 0
95-99 0
90-94 0
85-89 0
80-84 0
75-79 0
70-74 1 ******
65-69 1 ******
60-64 4 ************************
55-59 3 ******************
50-54 9 ****************************************************
45-49 3 ******************
40-44 9 ****************************************************
35-39 2 ************
30-34 2 ************
25-29 0
20-24 0
15-19 0
10-14 0
5-9 0
0-4 0
ANALYSIS TABLE:
Is a table intended to further assist interpretation of the mark distributions, in this case focussing upon individual questions. It can be used to identify, in particular, questions which are 'too hard' or 'too easy'. In the example given below Q4 may have a problem - it has a negative Quality score indicating that more intelligent people may have misinterpreted it, and 80% of candidates answered incorrectly. It had a high Difficulty score. The examiner may choose to rewrite this question for the following year's exam, maybe to remove an ambiguity etc. Q50, by contrast, has a Quality score near zero, and was answered correctly by nearly all candidates. Maybe this question was too easy.
The table has one line per question. The percentage of candidates answering each question option is listed towards the left of the table. The correct answer is marked with '**'. Two columns labelled 'Quality' and 'Difficulty' follow, which are discussed in further detail below the example. Finally, on the right of the table, is printed a histogram of the percentage of candidates answering each question correctly.