Proposal for a Design Study

Janette Cullinan

Yasuhisa Kato

David Slater

ED 333a

July 8, 1999

Proposal for a Design Study

Situative Perspective

Overview:

The focus of our design study will be a CD-ROM based lecture series that prepares students to pass the CPA examination. This instructional tool contains 31 lectures that are 4 hours in length (124 hours for the total course). Typically for every hours of lecture there is one hour of homework. The lectures are interspersed with actual questions from past CPA Exams that pertain directly to the material being covered. The recommended study period is four months, typically involving eight hours of lecture and eight hours of homework per week.

This courseware is considered a “self-study” product: the student purchases the CD-ROMs and associated course components (such as lesson books) and works independently through the material. A suggested study schedule is provided with the CD-ROMs along with a phone number and email address for “academic support.” The students can contact academic support as often as they like and ask questions on any CPA Exam related topic, however, the support is not proactive: the support desk will only respond to student inquiries. The support desk is manned by CPA’s who are very familiar with the course content and have taken the CPA Exam several times. Often, these Academic Support Specialists are part-time instructors for the traditional “live” version of the same course.

This same company also runs a traditional “live” course. This course has one instructor who uses a computer and LCD projector to show the exact same course material from the CD-ROM. The instructor does not actually “teach” the material in the traditional sense, rather she lets the CD-ROM presentation play and intermittently takes questions from the class or interjects additional material.

The Problem:

Of the students who begin a course of study with the “self-study” CD-ROM based lecture series, only 50% actually end up sitting for the CPA Exam. We theorize that the low percentage of students actually taking the exam may be due to one or more of the following four factors: low self-confidence, lack of motivation, lack of rapport with a content expert, and lack of discipline to stick to a study schedule.

We theorize that a major contribution to the average 95% test taking rate for the traditional “live” course is attributed to the environment of the classroom that promotes the aforementioned four conditions.

Proposed Solution:

We speculate that the addition of two new elements into the learning environment will alleviate the depressed test taking rates: (1) proactive mentoring from a qualified CPA Exam expert and (2) collaboration among students.

Mentoring will greatly increase the confidence levels of the students by reassuring them that they are doing what is necessary to pass the exam. Motivation will be achieved through encouraging words expressed by the mentor. An established rapport with the instructor will facilitate the student clarifying confusing topics that might otherwise cause the studying process to stall. Discipline to complete the course will be instilled by the need to check in with the mentor periodically and report study status.

Collaboration among students will also encourage students to complete their studies and sit for the exam. Since this particular study process is time consuming, a support network with which to share feelings of being overwhelmed will help boost confidence levels. Motivation will be assisted through peer support and fellow students who have more knowledge about a particular area could potentially clarify confusing topics. Finally, discipline to stick to a study schedule would be facilitated by a desire to remain in sync with the peer group.

Design Study:

Finding the Root Cause

We would begin by clarifying the factors that influence whether students take the exam. For the purposes of this inquiry, we will consider students in three groups: those who purchase the CD-ROM but do not take the exam, those who purchase the CD-ROM and take the exam but do not pass, and those who purchase the CD-ROM and pass the exam. For students who choose to take the exam, the current “pass” rate is roughly the same for CD-ROM students and classroom students. Therefore, we are most interested in the differences between students who take the exam and students who do not.

We expect to find that students who took the exam were either (1) simply more self-confident, motivated, comfortable obtaining clarification, or disciplined at the outset or (2) had better coping strategies to deal with the inherent short-comings of the system in regards to these factors. We do intend to include other considerations in our analysis to make sure our design addresses the root cause of the problem.

We will study one term of CD-ROM students (approximately four months long with 1,000 students). We would use three main sources to better understand the root cause of the low test-taking rate. First, we suggest pre-testing all students with sample CPA exams in order to ensure skill level prior to the course is not a strong factor in test-taking rates. Second, we propose to observe the students who purchase the CD-ROM by tracking how often they use it, how long they use it at a sitting, and how many of the modules they complete. Since direct observation would completely alter the normally solitary environment, we would recommend using tracking software that could provide this data without impacting the user. Naturally, just knowing that your usage is being tracked might change student behavior, but not as much as introducing a direct observer.

Third, we will conduct interviews at the end of the term with students, Customer Service Representatives, and Academic Support Specialists. Customer Service Representatives take phone calls from students with product complaints and returns (along with other administrative concerns). Academic Support specialists receive questions from students on the course content. For all three groups, we would explain the problem being considered and ask what seems to be working well in the current system and what needs improvement, along with their ideas for how to best improve the experience and foster higher test-taking rates. Those general, open-ended questions may elicit factors we have not considered. The compiled information (format below) should provide information on how the system operates today (+) and what participants in the system see as possible improvements (D).

Interview Subject / + / D
Students who do not take the exam
Students who take the exam, but fail
Students who pass the exam
Customer Service Representatives
Academic Support Specialists

When interviewing students, we will also ask for specific information about the constraints and affordances in their experience in regards to confidence level, motivation, ability to obtain clarification, discipline, and other factors. The chart in the next page is a rough approximation of how we expect to look at the data, including some sample potential answers that will help us understand the system at work.

1

Students who do not take the exam / Students who take the exam, but fail / Students who take the exam, and pass
Factor / Sample Potential Answers / Constraints & Affordances
Relevant Facts / Number of lectures completed
Pre-test score
Educational background
Professional background
Stated Reason / Reason for taking/not taking exam / §  Perceived readiness
§  Did not complete the program
Self Confidence / Perception of readiness / §  Fully prepared
§  Unprepared
§  Not yet prepared
Perception of own test-taking abilities / §  Strong
§  Medium
§  Weak
Motivation / Motivation for taking the exam / §  Career advancement
§  Required for work
§  Increased compensation
Commitment to others to take exam / §  Everyone knew I was studying to take exam
§  Spouse was counting on me to pass
Ability to Obtain Clarification / Comfort level using academic support services / §  Wasn’t comfortable calling a total stranger with a “dumb” question
§  Never noticed there was a number to call
§  Couldn’t call during open hours
Personal resources / §  CPA friend who answered questions
§  Study group
Discipline / Adherence to schedule / §  Followed schedule provided
§  Tried to do all at the end and did not complete
Self-monitoring systems / §  Reported progress to spouse weekly
§  Checked off progress on wall calendar
Other Contributing Factors / Financial incentives and disincentives / §  Employer reimburses cost of course if student passes
§  Increased compensation
Computer proficiency / §  Computer literate (comfortable)
§  Never used CD-ROM before
Personal/Family Environment / §  Children/Single parent
§  Uninterrupted time to study
§  Stressful personal situations
Work Considerations / §  Extensive travel
§  Time off to study
School Considerations / §  Competing coursework (time constraints)
§  Relevant coursework

Designing the Solution

Based on the analysis above, we would design the collaboration and mentoring practices and systems to best support students in completing the program and taking the exam. If the data supports it, we would seek to support increased confidence, motivation, ability to obtain clarification, and discipline through the mentoring and collaboration systems, paying particular attention to the suggestions made by the students and other participants (Customer Service Representatives and Academic Support Specialists).

Piloting the Solution

We propose using the same methods of evaluation to follow another cohort of students through one term, pre-testing, CD-ROM usage tracking, and student interviews. This would, of course, include additional information to gather feedback on specific facets of the design. Ideally, we would have four comparison groups (1) those who had the benefit of both the mentoring and collaboration systems, (2) those who had the benefit of just the mentoring system, (3) those who had the benefit of just the collaboration system, and (4) those who participate in the program without the improvements. Using the same evaluation techniques will afford us a very direct comparison and the ability to measure improvement. An increased test-taking rate would be the ultimate measure of improvement. This would also be an opportunity to make additional design enhancements after observing the system in action.

1