February 21, 2006

VERBAL PROTOCOL ISSUES AND PRELIMINARY DECISIONS FOR DISCUSSION

ISSUE #1: What is The Purpose of The Verbal Protocol Analysis?

  • To establish a preliminary set of online reading comprehension skills and strategies that we can use to begin the Year 2 formative experiments.
  • To conduct a study of online reading comprehension strategies.
  • To discover new questions that we might ask and study during Year 2 that go beyond the specific scope of the project.

ISSUE #2: How Many Sessions Will There Be for Each Student

  • Four, 40-minute sessions will be required to provide a complete range of tasks
  • Note: This will require an amendment to our IRB to increase the number of sessions from 3 to 4.

ISSUE #3: Should The Topics and The Tasks Be The Same Or Different in Each Session?

  • The topics should be somewhat common across tasks, in the same general knowledge domain, and ones that will be somewhat familiar to students (not esoteric) so that we reduce the effects due to prior knowledge.
  • The topic should be about a contemporary issue of social service or social action. These might include: What can we do to make the world a better place? homelessness; volunteer work in the community; heroes and heroines who have made an important contribution to the lives of others; or some other socially positive and progressive topic. Having topics such as these will be related to themes in the books that students will be reading in their Language Arts and Reading classes. They will also be of interest to reviewers and educators.
  • The tasks should vary, as should the online technologies, so that we probe the widest possible range of strategies required for success with a variety of online technologies (blogs, email, search engines, web page structures and elements, wikis, etc.)

ISSUE #4: What Will Be the Nature of Each Session?

Session 1: Introduction, assessment of skill and domain knowledge, training in think aloud, and a closed online reading comprehension task at a moderate-level of difficulty where a student is given a problem (question) and asked to locate, critically evaluate (utility, accuracy, and bias), synthesize, and communicate (email, attachment, blog). We will use an uninterrupted think aloud in order to get the “cleanest” think aloud data during this first session.

  • (5 minutes) Pre-interview to confirm their skill level, to assess prior knowledge, and to put them at ease by positioning them as an important informant to our research. (Tasks and specific introductory script to be developed)
  • (2 minutes) Intro to how to do a think aloud– view video
  • (3 minutes) Short, practice think aloud session (Task and specific script to be developed)
  • (20 minutes) Actual, uninterrupted think aloud session using Camtasia and Windows laptop machines.
  • Note: 10 additional minutes are reserved to allow students additional time to complete the task and to allow a few more minutes for pre-think aloud activities.

Session 2:A structurally prompted, think-aloud session where we probe, inviting students to think aloud, at pre-selected locations, if they do not voluntarily share their thinking at these locations. The task will bepartially open, where students will be able to choose the most personally interesting task from several possible questions (problems). We will pilot different alternatives to the probe that include both questioning and a simple, noise prompt.

  • (2 minutes) A welcome and reminder of their important informant status as well as the purpose of our research. A short prior knowledge task. (See Wolfe & Goldman, 2005)
  • (3 minutes) Repeat the practice think-aloud task from session 1 with different content.
  • (15 minutes each) Two, different, tasks each using a different set of communication tools. (email, blog, wiki, word document attachment, etc.)
  • We will do a structural analysis of the tasks in advance to identify common probe locations. This might include locations where students:
  • Come to understand the questions/problem
  • Select a search engine
  • Enter key words into the search engine
  • Read search engine results.
  • Select a search engine result
  • Navigate through a web page to find information.
  • Read information at an “about this site” link
  • Synthesize information as they communicate the solution/answer
  • Use a communication tool
  • Go back and reread a question
  • If we use a question probe, it needs to be designed so that it is always the same for each student and so that it is as “information neutral” as possible, e.g., “Tell me what you are thinking now?” We should not provide any more information than this during Session 2.
  • We should pilot the use of a sound probe. This might take less time and be a more standardized and neutral prompt, avoiding differences in stress and intonation during a verbal probe.

Session 3: Multiple, short, focused and targeted, think aloud sessions that allow us to probe more aggressively during specific tasks that are either more challenging than tasks from previous sessions (such as learning how to learn a new online technology, e.g., posting to wikipedia) or explore areas where obtaining think aloud data is especially challenging to obtain (such as critical evaluation of the information that someone has created at a spoof site or the synthesis of information during either the accumulation process or the problem solving process). These would be closed and assigned tasks.

  • (2 minutes) A welcome and reminder of their important informant status as well as the purpose of our research. A short prior knowledge task. (See Wolfe & Goldman, 2005)
  • (3 minutes) Repeat the practice think aloud task from session 1 with different content.
  • (8 minutes each). Four different tasks that establish a context and purpose and then ask students to solve the information problem.
  • Examples:
  • You are looking for information about students who are homeless. You want to what you might do to help them. You have found this site and it looks interesting. Now you want to figure out who created this information and think carefully about how they may be shaping the information they provide. What do you do?
  • You are doing a report on unusual animals and you have found this site: We want you to figure out if the information is accurate? Can you think aloud as you do this?
  • You are looking for an Egyptologist and your search engine has produced this set of results. Can you find this Egyptologist and think aloud as you do this?
  • We want you to find the answer to a question that is important to you. Can you do this and think aloud as you do so?
  • The nature of these tasks might be planned a bit in advance, since we do know some of the more challenging areas in which to obtain think aloud data (problem identification, critical evaluation, synthesis, learning how to learn) but we should also keep these open enough that we can learn from the first two sessions which areas we need to probe more aggressively in.
  • If we wanted to explore more than four areas, we could develop a dozen or more of these focused tasks and randomly assign them to students, such that each student receives 4 tasks to complete.
  • We should think about using this session to also establish the types of conditional knowledge that students use as well as declarative and procedural knowledge.

Session 4: A stimulated think aloud, using a Camtasia recording of online reading comprehension using an open ended, self-defined problem. The think aloud would take place only during the viewing of the Camtasia recording, when the experimenters would ask students to tell us what they were thinking and doing at each decision point in the recording, during the reading of a screen and each activity that took place there and especially as students moved from one screen to another.

  • (2 minutes) A welcome and reminder of their important informant status as well as the purpose of our research. A short prior knowledge task. (See Wolfe & Goldman, 2005)
  • (15 minutes) Students define at least 1 problem/question that they are interested in answering and then read online to answer it. If they complete the activity early, we ask them to develop another question and read online to answer that one as well, until 15 minutes expire.
  • (3 minutes) Saving the Camtasia video and preparing it for playback.
  • (20 minutes) View the Camtasia video of online reading comprehension and, using a structured probe framework, ask students what they were thinking and doing at each decision point in the recording. This would include the reading of each screen and each activity that took place on that screen as well as when students move from one screen to another.

ISSSUE 5: Who Should The Audience Be For The Communication Aspects of Each Task?

  • We should think about varying the audience on different tasks to ensure the widest possible range of register and style.

ISSSUE 6: How Should The Experimenters Be Assigned to Students?

  • We wondered if having the same experimenter work with the same student would be either helpful or harmful? On the one hand, having the same experimenter would establish greater solidarity and familiarity so the student might feel more comfortable and willing to share think-aloud observations. The experimenter might also develop a better understanding of the student that could then be brought to bear on both the session and the analysis. On the other hand, having a different experimenter for each session would not permit the assumption, by the student, that he/she had already shared a particular strategy and need not repeat it. It might also allow for different experimenters to think afresh about what a student was telling them and think in new and different directions during a session.

ISSUE 7: How Should The Analysis Be Conducted?

  • This has not yet been thoroughly explored. Various suggestions have been made from simply listing each articulated strategy and the context in which students report using it, to an analysis of situations and the strategies that students, who succeed on a micro or macro task, report. Also mentioned was a visual analysis of Julie’s data to confirm the generalizability of strategies across tasks and populations as well as a “hot-cold” strategy to segregate out more successful from less successful strategies. We might also wish to use a declarative, procedural, conditional knowledge type of framework established by Paris and others.

Timelines

  • Piloting – End of February (only a few days in Connecticut before state testing) 27-28 Maybe use Meghan and Josh and other accessible students.
  • Permissions distributed - March 14- 15
  • Task 1: End of March as much as possible and maybe a bit in the first few days of April
  • Try to do all students with the first task before AERA
  • Then, complete Tasks 2-4 from mid-May to June 15th
  • We need to plan the human resource issue now for this timeline since it extends a bit into the summer.