Chi-Fen Emily Chen (Spring 1999)
Analysis of the Test of Spoken English
Outline
I. Introduction:
- TSE: oral proficiency language test for nonnative speakers of English, administered by Educational Testing Service (ETS) through the TOEFL program
- Purpose: to measure the ability of nonnative speakers of English to communicate orally in English.
- Use: to select international teaching assistants, and to certify all other individuals’ oral English proficiency in a professional field
- Theoretical construct: communicative language ability
II. The TSE test methods:
- Testing environment: a semi-direct audiotaped test, typically administered in language laboratories with a microphone and a tape-recorder.
- Test rubric:
(a) TSE consists of 12 questions, each of which requires examinees to perform a particular language function, such as narrating, persuading, and giving an opinion.
(b) The instruction is given by the interviewer on the test tape and printed in the test book as well.
(c) The time allotted for each response ranges from 30 to 90 seconds, and some questions allow the planning time for the response from 30 to 60 seconds. The total testing time is approximately 20 minutes.
(d) Each TSE answer tape is rated independently by two trained TSE raters and they assign a score level using descriptors of communicative effectiveness related to language function, appropriateness of response to audience/situation, coherence and use of cohesive devices, and linguistic accuracy (see Appendix II). The score scale is from 20 to 60, divided into 5 levels. The scores of the two raters are averaged and reported in 5-point increments (i.e. 20, 25, 30, 35, 40, 45, 50, 55, 60).
- Test input:
(a) The test questions are written in the test book and recorded at normal speed on the test tape too. Non-language visual stimuli, such as maps, pictures, graphs, and charts, are used as part of input. The questions specify clearly the audience, the social context, the role the examinee has to play, and the language function the examinee is going to perform in each task.
(b) The test questions are of general nature and the language used is context-embedded everyday English, without any specialized technical vocabulary.
- Expected response:
(a) All of the responses are “constructed response,” which have to be recorded on the answer tape. They are relatively short passages of speech (from 30 to 90 seconds).
(b) The responses are expected to demonstrate how the examinee gives appropriate oral response to different audience and situations while performing various language functions.
(c) Restrictions: time constraint, restrictions on organizational, propositional, and illocutionary characteristics, psychological problems from the use of audiotaped response, and lack of nonverbal communication in the response.
- Relationship between input and response: It is a nonreciprocal input and response, which precludes live interaction between the interviewer and the examinee. The questions are pre-recorded and the examinee is left to respond.
III. The test content of the TSE Practice Questions: (see Appendix I)
IV. Construct validity of the TSE test:
- The test content matches four competencies (functional, sociolinguistic, discourse, and linguistic competence).
- The test methods affect the examinee’s performance, especially in terms of discourse competence and strategic competence.
V. Conclusion:
- advantages
- disadvantage
- suggestions
Paper
I. Introduction
Probably the most well-known oral English proficiency test for nonnative speakers of English is the Test of Spoken English (TSE). It is a member of the Test of English as a Foreign Language (TOEFL) family of tests developed by Educational Testing Service (ETS). The TSE test, a standardized instrument, is designed to evaluate the speaking ability of nonnative speakers of English, at or beyond the post-secondary level, to communicate orally in English. It does not measure listening, reading, or writing skills in English. The examinee’s performance on the TSE test is considered to indicate his/her oral language ability in English to communicate in academic or professional environments. The TSE scores are used by many North American institutions of higher education to select international teaching assistants, and to certify all other individuals’ oral English proficiency in a professional field, such as health professionals (physicians, nurses, pharmacists, and veterinarians). The scores are not regarded as predictors of academic or professional success, but only as indicators of nonnative speakers’ oral communicative language ability in English (ETS 1999b).
The TSE test was revised in 1992 in an effort to make it more ‘communicative” in both its test content and rating scale criteria, focusing on overall communicative language ability rather than just on linguistic accuracy (i.e. accuracy in terms of pronunciation, grammar, and vocabulary). The revision project was completed in 1995, which was developed “on the premise that language is a dynamic vehicle for communication, driven by underlying competencies, which interact in various ways for effective communication to take place” (Chen & Pooler 1999). Therefore, the theoretical construct underlying the revised TSE test is communicative language ability, which, for the purposes of the test, includes strategic competence and four language competencies: functional competence, sociolinguistic competence, discourse competence, and linguistic competence (Chen & Pooler 1999). It corresponds to Bachman’s (1990) model of communicative language ability quite well.
In this paper, I would like to evaluate the TSE test’s construct validity from two aspects: test methods and test content. Specifically, I will analyze the test methods that TSE uses to measure oral language ability based on Bachman’s (1990) framework of test method facets. Then, I will discuss the content of the TSE practice questions (ETS 1999a; see Appendix 1) and evaluate them to see whether the TSE test content matches its claimed theory, communicative language ability.
II. The TSE test methods
According to Bachman’s (1990) framework of test method facets, there are five categories: (1) testing environment, (2) test rubric, (3) nature of the input the test taker receives, (4) nature of the expected response to the input, and (5) relationship between input and response. I will use his model to analyze the test methods that TSE uses and discuss how the test methods affect the examinee’s oral language performance.
(1) Testing environment:
The TSE test is a semi-direct test delivered via audio-recording equipment and a test book, and it is typically administered in language laboratories. An interviewer on the test tape leads the examinee through the test; the examinee responds into a microphone, and responses are recorded on a separate answer tape. The examinee, therefore, will be tested in a tiny cubicle and has to interact with a tape-recorder and a microphone, not with a person. This may cause some affective problems for test takers if they are not familiar with the equipment, and then may have negative influence on their language performance in the test tasks. The TSE test designers have taken this factor into consideration. In order to help the examinee become more accustomed to speaking on tape and to allow for adjustment of the audio equipment as needed, they provide a “warm-up” task before the test begins: the interviewer on the test tape asks some simple questions about the examinee and then the examinee can practice speaking on tape and his response in this part will not be scored.
(2) Test rubric: It includes test organization, instruction, time allocation, and scoring.
The TSE test is composed of twelve questions, each of which requires the examinee to perform a particular language function such as narrating, persuading, and giving and supporting an opinion (ETS 1999a). There is no difference in the degree of difficulty or importance for each question in the TSE test, but only a difference in the functions that the examinee has to perform.
The instruction given by the interviewer, including the testing procedures, the tasks to be completed, and the allotted response time and planning time, is printed in the test book and recorded on the tape too. That is, all the instruction of the TSE test is presented both aurally and visually in English. The advantage is that the examinee can either listen or read the instruction of the test and this way can ensure the examinee to understand exactly what he is going to do in each test task.
The time allotted for each response ranges from 30 to 90 seconds, and some questions give the planning time for the response from 30 to 60 seconds (The total testing time is approximately 20 minutes). The examinee actually responds to each question under big pressure of time constraint. This may also have a negative influence on the examinee’s oral performance, for he has to always think quickly and make good use of the short period of time to express himself.
As for scoring, each TSE answer tape is rated independently by two trained TSE raters; neither knows the scores assigned by the other. Each rater evaluates the response and assigns a score level using descriptors of communicative effectiveness that are delineated in the TSE rating scale (ETS 1999b; see Appendix 2). As of July 1995, the TSE score consists of a single score of communicative language ability, which is reported on the following scale:
60 Communication almost always effective: task performed very competently
50 Communication generally effective: task performed competently
40 Communication somewhat effective: task performed somewhat competently
30 Communication generally not effective: task generally performed poorly
20 No effective communication: no evidence of ability to perform task
The scores of the two raters are averaged and are reported in 5-point increments. If the two ratings are not shown adequate agreement, the tape is rated by a third independent rater. Final scores are based on a resolution of the differences among the three scores (ETS 1999b).
With regard to the rating scale criteria of the TSE test, they in fact include four categories: ability to perform the language function clearly and effectively (functional competence), appropriateness of the response to the designated audience or situation (sociolinguistic competence), coherence of the response and use of cohesive devices (discourse competence), and accuracy in the use of linguistic features (linguistic competence). However, since the score is given holistically based on the examinee’s overall communicative oral language performance, no single area of competence, such as linguistic competence, is over-emphasized. The raters react to the examinee’s speech as a whole: they do not think about the individual components of the speaking ability or count the number of errors that the examinee makes. For example, the examinee can get the score of 60 if he communicates effectively regardless of his minor errors. This scoring method encourages the examinee to focus on the overall effectiveness of his communication rather than merely on the accuracy of his language use. It matches the theory of communicative language ability underlying the test.
(3) Test input: It involves the format and nature of language that is used in test input.
The twelve questions in the TSE test are of general nature and designed to inform the raters about the examinee’s oral communicative language ability in English. All of the questions, like the instruction, are written in English in the test book and recorded at normal speed on the test tape as well. Non-language visual stimuli, such as maps, a series of pictures, graphs, and charts, are used as part of input to elicit the examinee’s speech. The questions specify clearly the audience, the social context, the role the examinee has to play, and the language function the examinee is going to perform in each task. This information is very useful for the examinee to decide how to use his language appropriately in different situations, which helps to elicit the examinee’s oral performance in terms of pragmatic competence.
Since TSE is a test of general oral language ability, the language used in the questions is everyday language, without any specialized technical terms or culture-specific vocabulary. In addition, the language of the test input is context-embedded, for the topics of the questions are generally familiar or known to all examinees, regardless of native language, type of educational training, or field of employment (ETS 1999b). Therefore, the examinee’s oral performance will not be affected due to unfamiliar vocabulary or topics in the input material.
(4) Expected response: It consists of the format, nature of language, and restrictions on the expected response to the input material in a test.
The examinee is required to record his oral responses in English to the TSE questions on the answer tape. All of the expected responses are relatively short passages of speech (from 30 to 90 seconds), but not single sentences. Such responses are the productive use of language responding to the input material, called “constructed response” (Popham 1978, as cited in Bachman 1990). This type of response requires the examinee to organize his speech coherently in a discourse level rather than just producing some individual sentences. The examinee’s discourse competence is expected to be demonstrated in this type of response.
The language required from the examinee is commonly used vocabulary in everyday communication, but since the TSE test is mainly used to select international teaching assistants or to certify health professionals, the response may involve the examinee’s specialized vocabulary in his field. However, the most important part in the expected response is how the examinee gives appropriate response to different audience and situations in his speech while performing various language functions. The examinee’s pragmatic competence, including functional and sociolinguistic competence, is expected to show from his response.
There are some restrictions on the expected response in the TSE test. The most obvious one is that the response is constrained by time. With only 30 to 90 seconds for the response to each question, it actually imposes restrictions on organizational, propositional, and illocutionary characteristics in the examinee’s oral performance. It will be a big challenge for the examinee to take care of every detail in his response within such a short period of time. In addition, the response recorded on the tape may restrict the examinee’s ability to perform normally as he does in real-life situations if he is afraid of speaking into a microphone. Also, the nonverbal communication such as gestures and facial expressions, which is actually indispensable part in interpersonal communication, cannot be seen from this type of response.
(5) Relationship between input and response:
The semi-direct audiotaped format of the TSE test results in a nonreciprocal input and response which precludes live interaction between the interviewer and the examinee. The questions are pre-recorded and the examinee is left to respond; there is no feedback, and the language used does not affect subsequent language use. Since communication is a two-way system, using reciprocal input and response for the oral proficiency test should be more appropriate to measure an individual’s oral communicative language ability, especially if we want to find how he uses strategic competence, which can only be measured in the interactive communication.