Writing tests at Universities

ESF Testing project, Vilnius, January 29-30, 2007

Vita Kalnberzina, Dr.Phil., Assist.Prof., Latvia University

Contents

Writing tests at Universities 1

Contents 1

List of Tables 1

Course Programme 2

'Writing' as a concept 3

What do you understand by the term 'writing'? 3

What does the term 'writing' mean to the student/test-taker? 5

How is written language different from spoken? 5

Is writing informational, interactional or both? 6

Teaching Writing at University Level 7

Product approach 7

Conventions For Term Paper Writing 7

Process approach 8

Help for students at Dartmouth University 8

Instructions to Writing assistants: 8

Moving from Writer-Based to Reader-Based Prose 9

Writing test development 9

Indirect approach 9

Direct approach 10

What do you need to know to develop a task? 10

Writing Tests at University 13

Qualspell 13

What task types can be used to test writing proficiency at the University? 13

Which text/task-types are appropriate for your students? 14

How do you know that you have produced a good task? 15

What does a test-taker need to know to do a task: develop the test specification 15

Marking writing tests 15

Formative Assessment 16

Summative Assessment 16

Holistic marking scales 16

Analytic marking scales 18

Assurance of test quality 20

Validity 20

Reliability of the examination 21

Inter and Intra-marker reliability 21

Bibliography 23

Online sources 23

List of Tables

Table 1 You as a writer 3

Table 2 What is writing? 4

Table 3 Writing from the point of view of the test-taker 5

Table 4 Comparison between written and spoken language 5

Table 5 Interactional versus informational writing 6

Table 6 Comparison between direct and indirect writing tests 10

Table 7 Topics for writing tasks, Compare with Language Domains in CEF 11

Table 8 Writing as a process (Callaghan, Knapp, and Noble 1993, p.193) 14

Table 9 Impressionistic marking scale 16

Table 10 CEF levels of writing production 16

Table 11 Holistic scale (British Council IELTS written production evaluation) 17

Table 12 BC analytical writing evaluation scale 18

Table 13 Validity of the writing tasks 21

Table 14 Reliability of the Year 12 examination in 1999 22

Table 15 Intercorrelations between the different tests in 2001 22

Course Programme

January 29
9:00 – 10:30 / session / Introduction, Writing as a concept
10:30 / coffee break
11:00 – 12:30 / session / Approaches to testing writing
13:30 / Lunch
15:00 – 16:30 / session / Writing tests and tasks
16:30 / tea break
17:00 – 18:30 / session / Marking scale types and marking reliability
30 January
9:00 – 10:30 / session / Development of new writing tasks
10:30 / coffee break
11:00 – 12:30 / session / Development of marking scales
13:30 / Lunch
15:00 – 16:30 / session / Pretesting of the new writing tasks and marking scales
16:30 / tea break
17:00 – 18:30 / session / Feedback on the new writing tasks and marking scales and evaluation of the course

'Writing' as a concept

Objective: Discuss the concept of writing and hence validity of writing tests

What do you understand by the term 'writing'?

Take a couple of minutes to answer the following questions:

Table 1 You as a writer

Question / Answer
What do you write?
How do you write?
What are three main things you want your students to learn about writing?
How would you define 'writing' as a skill?

Read the 5 texts and fill in the table 3 below:

1.  Writing is both tool and art, but it is not a body of knowledge. Those who have taught writing and observed writers at work are convinced that writing is a complex, multifaceted, dynamic skill, affected by cognitive demands, affect (whether the writer feels good on that day, whether he or she likes the topic, or the reader of the writing, or the reasons of writing) and variations in test conditions. Above all writing is a cognitive activity that calls on many components of personal experience, attitude, knowledge of ideas, issues and factual information, but that requires a person to take and shape some parts of this complex web of material into a rhetorical, syntactic and mechanical whole (Hamp-Lyons 1993, p.11).

2.  The writing test tests the test-takers’ ability to write correct and appropriate sentences, use conventions peculiar to written language, think creatively and develop thought excluding irrelevant information, manipulate sentences and paragraphs to use language effectively, write in an appropriate manner with a particular audience in mind, organise and order the written material (Year 12 test specification).

3.  Textual competence includes the knowledge of the conventions for joining utterances together to form a text consisting of two or more sentences which are structured according to the rules of cohesion (marked relationships among sentences) and rhetorical organisation (development of the text) (Bachman 1990)

  1. A successful writer needs to have content knowledge (knowledge of the concepts involved in the subject area), context knowledge (social context in which the text will be read, the reader's expectations, knowledge of the cotexts alongside which this text will be read), language system knowledge (lexis, syntax) and writing process knowledge (the most appropriate way of preparing for a writing task) (Tribble 1995).

Table 2 What is writing?

Year 12 test specification (1999) / Hamp-Lyons (1993) / Tribble/Bachman (1990)
What is writing?
What subskills does writing consist of?
What is it affected by?

What does the term 'writing' mean to the student/test-taker?

Table 3 Writing from the point of view of the test-taker

Question /
Answer
What do your students write?
In what settings do they like/do not like to write?
What factors inhibit/encourage them to write?
What contexts stimulate them to write well?

How is written language different from spoken?

Read the extract below and say what language aspects are lost when we transform spoken language to written language? How can we compensate for the loss? Is this enough?

Contemporary views of the differences between written and spoken language support the idea that

  1. written and spoken language do possess distinctive features and
  2. that texts can be distributed along the continuum from the most typically spoken to the most typically written.

Apart from the loss of the contextual information that is available to the people who participate in a conversation (for example, knowledge of the background and the histories of the individuals concerned) two of the most important aspects of speaking that are normally lost in any transcription are prosodic (rhythm, phrasing, pauses) and paralinguistic features (the way someone is speaking, facial expressions and gestures). To compensate for this loss we use punctuation and other features (Tribble 1995, p.17, 18).

Fill in the table choosing from the box below:

Table 4 Comparison between written and spoken language

spoken / written
vocabulary
Grammar
Text

Incomplete utterances, full word forms, rare words, 'incorrect' grammar, vocabulary of Latin origin, complete sentences, hesitations, reduced word forms, informal vocabulary, many phrasal verbs,

common words, conventional abbreviations, polysyllabic words, high density of content words, sophisticated and intricate grammatical resources, heavily nominalized style, clauses instead of sentences as a minimum unit, carefully planned paragraphs

Is writing informational, interactional or both?

Weir (1995) considers that there are two types of writing operations: interactional and informational.

Fill in the table choosing the operations from the box below

Table 5 Interactional versus informational writing

Type of operation / Operations/action / Product
Interactional in social and service texts:
Informational in academic contexts

Summary checklist of writing operations (Weir 1995)

·  Describing process which might involve: purpose, describing means, results, process, change of state, sequential description, instructions, summary

·  Describing phenomena and ideas: definition, classification, identification, comparison and contrast, exemplification,

·  Arguing: stating a proposition, stating assumptions, induction, deduction, substantiation, concession, summary, generalisation, speculation, comment, evaluation

·  Expressing thanks, requirements, opinions, attitude, confirmation, apology, wants, needs, lacks, ideas, information, complaints, reasons, justifications

·  Eliciting information, directions, service, clarification, help, permission

·  Directing: ordering, instructing, persuading, advising, warning

Teaching Writing at University Level

Product approach

Conventions For Term Paper Writing

http://www.lu.lv/mvf/Studij_bak.htm

Students having registered for the academic BA programme (English Philology) will be eligible to write 3 term papers before writing their Bachelor Paper (BA paper).

Students can choose themes for term papers in the following areas:

1.  Aspects of the English Language (Phonetics, Grammar, Lexis);

2.  British (American, Canadian) studies;

3.  English Language Teaching (ELT) Methodologies;

4.  Linguistics;

5.  Translation and Interpreting;

6.  Literature.

Length Of The Term Paper

The term paper will be between 5000 words and 7500 words.

You are not required to produce a term paper embodying totally original work and ideas that will be a contribution to knowledge, although if you do, you will receive appropriate credit for it.

Organisation And Content Of The Term Paper

Conventionally, there will be:

Title page

Declaration of academic integrity

Abstract in English and Abstract in Latvian

Table of contents

Introduction;

Chapters;

Conclusions;

Theses;

Bibliography;

Appendix (or appendices).

Plagiarism

You plagiarize if you take someone else’s work, ideas, words and use them as if they were your own. You can avoid plagiarism by using correct methods for quoting, paraphrasing and referencing. Quoting means using the exact words of the writer/speaker, whereas paraphrasing means restating the words and ideas from a book, an article, or a lecture in your own words.
Failure to produce references amounts to plagiarism, and your work will be deemed invalid.

Layout And Presentation

Use A4 size white paper on one side.

Word process the text using Times New Roman.

Use twelve point for the main text; fourteen point for headings.

Use margins: top/bottom/right -2.5 cm; left-3.5 cm (for binding);

Use 1.5 line spacing;

Indent quotes longer than three lines by five characters; use single space between the lines;

Process approach

http://www.dartmouth.edu/~writing/services/support_student.shtml

Principles of process pedagogy at Dartmouth University:

  1. writing can be understood as the culmination of several steps in a complicated process, which include prewriting, writing, rewriting, and all their attendant strategies
  2. use of dialogue as a teaching technique.
  3. challenge the traditional, authoritative models of teaching, in which professors (who know everything) talk "at" students (who know very little).
  4. empower students by getting them to talk about their writing at every step of the writing process.
Help for students at Dartmouth University

·  If you are a first-year student, we will help you understand the expectations of college writing.

·  We will help you generate, research, and organize ideas.

·  We will talk to you about your structure, grammar, and style.

·  If you are a student for whom English is a second language, we will help you work on English grammar and style. We will also help you understand the conventions for writing an essay in English.

·  If you are a senior, we will provide strategies for writing a thesis or culminating experience paper.

·  We will even help you with multi-media compositions, such as Web sites, videos, or PowerPoint presentations.

Instructions to Writing assistants:

·  Re-read the assignment, if the professor has written one. Make careful notes regarding what, precisely, the professor is asking the students to do.

·  Read through all (or several) of the papers once, without marking. This strategy helps you to get a sense of the range of essays you'll be reading.

·  Read through each paper twice: once to get the gist, the second time to respond. This strategy helps you to see where the writer is going before you jump in with your advice.

·  Respond the first time through, but in pencil. That way, you can erase your remarks if something the writer says on page four changes the way you feel about page two.

·  Take notes, on the side. Sometimes writing assistants want to keep a running commentary of their reading responses, but don't wish to share these responses with the writer. Keep a pad and a pen nearby.

Moving from Writer-Based to Reader-Based Prose

http://www.dartmouth.edu/~writing/materials/tutor/pedagogy/process.shtml

At some point in the writing process, writers must turn their attention from the writing process to the written product. They must transform writer-based prose into reader-based prose. To do this, they must be ready to revise, paying close attention to their prose and to the effect it may be having on their audience.

Often it's precisely at this point in the process that a writer will come to you for help. He has a draft completed, his argument seems logical and even persuasive, but he wants a second opinion. You read the essay and have some trouble following his line of reasoning. You point to a paragraph that is particularly confusing, and you ask what the writer is trying to say. He responds, "But it's all right there!" and goes on to summarize a point that he clearly hasn't made. What's happened? The writer's point is so firmly entrenched in his mind that he really believes that it's "in" the essay. You show him that it's not. You've just given him a lesson in the difference between writer-based and reader-based prose.

Writing test development

How do you develop writing tests? How do you test writing?

How often do you test? How do you mark writing?

What problems do you have?

Indirect approach

Hamp-Lyons (1993, p.5):

Indirect tests (recognize correct word order or structure, choose the correct phrase for the gap, underline the phrase that contains error etc.) do not require the test-taker to write continuous prose, there is little room for personal interpretation, a key is provided for the marker, the role of human judgement is less obvious, as scores are not tied to essay samples or rating scales, but to the judgement of the test developers who have developed questions and decided on the correct answers. In this case training of markers is not necessary and scoring can be done clerically.

In indirect discrete element writing tests (TOEFF, MELAB) knowledge of grammar rules and vocabulary items tend to dominate. While these skills may be related to proficient writing, most of us do not accept that they can represent what proficient writers do. Therefore indirect writing assessment is invalid. It has extremely negative educational consequences on the learning/teaching process as it creates the impression that writing is the same as recognizing errors in sentences, knowing grammar rules, and of not writing about ideas and emotions.