New York University Student Bar Association

Bill No. 03-04

A Resolution Regarding the Law School’s ExamSoft Policy

History

The SBA first became interested in computerized exam taking (CET) in February 2002 as the result of student inquiries as to why the Law School does not allow CET. Seeking to determine the rationale behind this policy, the SBA learned from the administration that, at one time, students were allowed to use laptops for in-class exams. Under that system, a program was loaded onto the student’s laptop, which disabled all functions except word processing. Because of problems with the software’s technical support, NYU Law eventually returned to the traditional practice of handwritten exams.

Having looked at schools which use some form of CET, the SBA at that time determined that the technical hurdles the Law School had faced in the past could, most likely, be overcome. If it was determined that overwhelming support for CET existed within the student body, the SBA resolved to strongly suggest that the administration move towards implementing a CET program. However, the SBA, in initial discussion about the issue, was itself clearly split as to whether implementation of CET would be beneficial to the student body. Because of this situation the SBA decided to survey the student body to try to find "the center of gravity" - where most people were leaning on the issue. The SBA intended the results of this survey to enable it to come to an internal consensus about the type of recommendation it would then make to the administration.

The survey was distributed on February 20, 2002 and consisted of seven questions designed to determine class year, laptop ownership, degree of preference for different CET choices, and overall preference between the current system and implementation of CET. On the reverse of the survey was a statement of purpose and excerpts of the longer supporting and opposing arguments separately distributed. The results of the survey indicated that the split among the members of the SBA on this issue reflected those held by the student body at large. On the basis of these results, the SBA determined that it was not possible at that time to reach a consensus that would have it push in favor of a change in the status quo of written exams with the option to use mechanical typewriters. However, the SBA believed that the results clearly indicate that CET was an important issue to the study body and therefore recommended that the Administration look carefully at the needs and expectations of incoming students, in order to draw its own, independent, conclusions on this issue.

Specifically, the SBA suggested that the Administration take all necessary steps to further understand the nature of student’s desires for CET. Suggestions of how to accomplish this included:

  1. Conducting a more detailed poll of students that provides more explicit detail about students’ interest and trepidation about CET and allows open commentary for them to voice their concerns and desires;
  2. Holding a community discussion chaired jointly by the students, faculty, and administration to discuss the results of any such survey;
  3. Facilitating additional research into the logistical ease or difficulty of developing a CET plan along with more direct consideration of the problems and benefits experienced at other institutions;
  4. Determining, with continued vigilance, if matriculating students are interested in such a policy and if they would like to see the nature of exam taking changed. This could be accomplished by a paired survey issued with the student’s admission packet (upon acceptance of NYU’s offer) and a follow up survey taken after the student has his/her first series of exams during the 1L year; and
  5. Monitoring developments at other top 5 schools to determine which schools are adopting CET and why.

The SBA does not know which, if any, of these steps were taken.

In September 2002, the Administration expressed to the SBA its interest in CET, specifically, in a program called ExamSoft, and indicated that the Law School had already contracted with the owners of ExamSoft in order to be able to test it with the incoming 1L class. In October 2002, the SBA again met with the Administration regarding ExamSoft and learned that the Administration was aware that the software only worked with Windows-based laptop computers. Regarding the test of this software on incoming 1Ls, the SBA was unsure how it would proceed and how success would be defined. Throughout Spring 2003, the SBA engaged in several discussions with the Administration regarding ExamSoft's limitations regarding Macintosh and about loaner computers, however, none of these meeting resulted in any clear resolution. In fact, the SBA did not learn the exact terms of the Administration’s proposed ExamSoft policy until April 30, 2003, a mere two days before the beginning of Spring Examination Period, when a group email containing the new ExamSoft policy was sent to all rising 2Ls and 3Ls.

At the beginning of Fall Semester 2003, the SBA saw that the ExamSoft policy was at the forefront of student concerns and accordingly devoted much of its time in meetings with the Administration to this issue. Particularly, the SBA was concerned that:

  1. There had been no report by the Administration regarding the “pilot test” on 1L students last year. The SBA became even more concerned when it learned that there had been no simulation of in-class exams; last year, all in-class exams were handwritten and only on take-home exams was there the possibility of using ExamSoft.
  2. There was a serious lack of communication between the Administration and students regarding the new ExamSoft policy.
  3. There were significant technical issues raised by students who had used ExamSoft that hadn’t been addressed by the Administration.

As a result of the SBA bringing up these issues, the Administration agreed to hold an ExamSoft Town Hall Meeting to discuss and hopefully alleviate some of the student concerns. At the Town Hall Meeting, which was held on October 27, 2003, many students took issue with the new ExamSoft policy. As a result of this meeting, the Administration decided to solicit formal input from the SBA regarding the new ExamSoft Policy.

The proposed policy embodied in SBA Bill 03-04, “A Resolution Regarding the Law School’s ExamSoft Policy,” is the formal SBA response to the Administration’s request. With less than one week to work in, the SBA brainstormed nine possible exam taking policies, which included the previous status quo, the new ExamSoft policy, and seven other variations, published them for all students to see, and solicited student comments. The SBA also considered the overriding goals of the Administration – retaining exam integrity – and the professors – control over the examination environment and a preference for typewritten exams.

The SBA believes that its proposed policy more than meets the Administration’s and professors’ goals while alleviating the many concerns of students. As is the case with most policies, this is a compromise, therefore the SBA decided to attach the Explanatory Notes below to explain in further detail each student concern and how these compromises were achieved.

History1

New York University Student Bar Association

Bill No. 03-04

A Resolution Regarding the Law School’s ExamSoft Policy

Explanatory Notes

First Whereas:

The SBA has found that there are three instances where application of the Law School’s current ExamSoft policy results in the perception of inequity amongst students. They are: (1) lack of sufficient notice for returning 2Ls and 3Ls; (2) exam results; and (3) the Law School’s proposed solution.

  1. Lack of sufficient notice for returning 2Ls and 3Ls:

The switch from Blue Books, the standard for decades, to computer-based testing is a seismic shift in exam practice and law school policies. A change of this magnitude, with all of its financial and test-taking implications, requires notice and a phase-in period. However, all returning second and third year students matriculated without notice that the Law School would be switching to computer-based testing. While arguably this may not be enough to form a reliance interest per se, as the law school did not represent to any particular type of exam technology, the Law School did, at least implicitly, represent to a fair exam taking policy. The only agreement among students currently is that the present administration plan will result in test taking results that will be substantially and inappropriately different between the two groups of students because of the technology differences, and that these technology differences are being imposed essentially without choice or notice (or the choice to expend an extra $2,000 to re-establish an even playing field)

Until April 2003, there was no notice that in-class exams would be administered via ExamSoft and require certain equipment. It is this “switching in mid stream without notice” that is the cause of the perceived unfairness. Students relied up to that point on the administration (both Law School and University) and purchased equipment that is non-compliant with the new requirements. In the future, when all students will have had notice prior to matriculation as to the technology requirements, this particular fairness issue will disappear.

  1. Exam Results

Implementation of an exam taking method which allows two different modalities – computer-based and handwritten – results in perceived inequitable exam results. Student concerns include:

1)Conscious bias by professors against handwritten exams

2)Unconscious bias by professors against handwritten exams

3)Discrepancy in word count limitations

4)Discrepancy in editing abilities inherent in word processing:

  1. Faster (or slower) pace for typing
  2. Mass delete, cut and paste abilities
  3. Spell check

5)Advantage for handwritten exams in classes requiring tables, graphs, and other exam-required answers not supported by ExamSoft (e.g. transfers, class of 2006 and L.L.M.s required to use ExamSoft are disadvantaged compared to those allowed to handwrite).

Despite administration efforts to alleviate concern over some of these specific issues through simple statements that exam results will not be inequitable, no studies have been referenced by the administration to support its statements, nor have the results of any beta-tests. Conversely, common sense and in-class statements by professors to the effect that they much prefer typed exams has instilled in students the perception that they will be disadvantaged if they choose to handwrite.

  1. The Administration’s proposed solution

The Administration’s proposed solution is to allow any student to purchase an ExamSoft-compatible laptop. The reaction of the impacted student body is this is a financially inequitable solution because:

1)Until October 21, 2003, the law school via the university computer store were still endorsing the sale of MacIntosh computers and desktop PCs and based on this recommendation students bought non-ExamSoft compatible computers and do not believe they should be forced to purchase a second computer to achieve parity with other exam takers.

2)To remedy the situation, students are forced to either purchase an entirely new computer system, with estimated prices of $1,850 (plus tax) at the NYU computer store, or to purchase a wireless card and other upgrades to make an existing laptop compliant (cost variable).

3)The option of additional loan support, while appreciated, does not address the long-term financial burden on the significant part of the student body that will be pursuing legal careers in public interest practice.

The number of students without an ExamSoft-capable laptop is unknown, as there have been no surveys of the student body by the administration that the SBA is aware of. However, regardless of the results of any such survey, the inequity still exists for those without the compliant equipment.

Second Whereas:

The SBA has received numerous emails from students concerning technological issues with the roll-out of ExamSoft. The SBA has heard from the Administration many times that it is confident that not a single test will be lost as a result of the ExamSoft software itself, however, there is still student concern regarding this issue that has not been answered. For example, one current 2L emailed the SBA the following:

“I was nearing the end of my exam. I suddenly got a notice saying the program could not save my document and would be shutting itself down.
It turned out okay, because I'm relatively computer savvy, and because the professor didn't lock out all my other programs. I was able to copy the full contents of my exam into MS Word and save it there. I immediately emailed a copy of that document (with about 45 minutes left in the exam, but I wanted certain proof that it all happened before, not after the time limit) to the professor's admin assistant. Then I proceeded to complete the exam in MS word, and emailed the completed version, about 10 minutes before the end of the exam, to the admin assistant.
Ultimately, ExamSoft lost my whole exam--they were not able to recover it. They said I had an error they had never seen before. They accepted my MS word version, so I wasn't penalized, but I had to spend about 5 hours after the exam on the phone with their tech support people.
Ultimately, their advice was to never again use that particular computer with ExamSoft.”

Aside from repeating that no exam had ever been lost (as indeed, this student’s email makes clear, the exam wasn’t lost because it was copied into MS Word), the Administration has not effectively addressed and alleviated student concern regarding these issues.

The substantial number of remaining technology issues do not relate to the ExamSoft software itself but rather relate to the Law School’s IT system and its ability to handle 1,800 students taking tests on laptops over a two week period. A sampling of issues not yet addressed to student satisfaction include:

  1. What to do if a laptop freezes during the exam?
  2. What kind of technical support will be available before, during and after the exam?
  3. What should be done if there are Internet connectivity problems?

Third Whereas:

The implementation of ExamSoft is an Information Technology Process Change. IT process change initiatives are common and widespread across both corporate America and American educational establishments. The SBA recognizes that primary responsibility for policy decisions lies with the Administration. However, as the SBA Constitution mandates, the SBA’s role is to “provide an effective medium for the expression of students’ views” and to “maintain an effective liaison between students, the administration, and the faculty of the Law School.” To those ends, the SBA believes it to be its duty to note that some of the basic practices which are common to all IT process change programs were apparently not followed in the ExamSoft implementation. The basic process is as follows:

  1. Project Working Committee formed of impacted stakeholders/users. They write the following documents, which are presented (after approval by the project manager and key stakeholders – i.e. administration) to the developers:
  2. Process map
  3. User requirements
  4. Use cases
  5. Designers build prototypes (non-functioning dummy screens), validated by PWC by working through the use cases and comparing against the user requirements.
  6. After validation by PWC, hard-coding.
  7. Beta Testing in non-live environment
  8. Bug fix
  9. Pilot Testing in a live environment of representative user groups
  10. Bug fix
  11. Phased roll-out.

Other common practices include:

  1. Best Practices research of other similarly situated institutions that have implemented (or attempted to implement) the process change proposed. To the knowledge of the SBA, the Administration did not compile, examine, or make available to the student body any detailed studies of the results of the process change (both IT technical issues and impacts on the substantive issues of exam taking and results) at other law schools where computer based examinations and/or ExamSoft were implemented. The SBA was particularly concerned by reports from transfer students from Harvard and information from the SBA president of Cornell Law School that ExamSoft was attempted and dropped due to poor results. While the SBA is happy to share this information with the Administration at this point in time, the larger issue is that this information was not actively solicited and analyzed by the Administration at the beginning of their process, to our knowledge.
  2. Project Working Committee (PWC) of all impacted stakeholders formed to review proposals, processes, and technologies. When this was suggested by SBA as a result of the survey conducted in February 2002, it was not acted upon by the Administration. The SBA made this suggestion again, with several members volunteering to serve on the focus group; again, the Administration did not act on this and the comment on this proposal was “its not their [the volunteering students’] job.” This response is indicative of the Administration’s misperception about IT process change: it may be, in the end, a policy decision by the Administration but the process of finding the appropriate decision is not a decision solely of “best technology” but of “best process with new technology to meet the needs of the stakeholder.” Finding a new process necessarily involves involving all impacted stakeholders. Discussing ExamSoft with SBA during and around regularly scheduled Dean’s Meetings is not a substitute for a formal PWC: a process-focused stakeholder review and input of the proposed process change and new technology.
  3. Beta testing, which involves parallel implementation of the new process and technology in a faux live environment. Beta tests usually test capabilities of the system over all possible use scenarios. This, to the SBA’s knowledge, was never done.
  4. Pilot testing to demonstrate the effectiveness of the new IT process change in scenarios mirroring or representative of full-implementation. The pilot test in this case was not representative: the pilot test was of 1L take-home open book open hard-drive exams where the option of using non-ExamSoft computers was available. To the extent that this mirrors current proposed practice for all take-homes, it was an appropriate pilot (although to the knowledge of the SBA, no systematic feedback from the pilot test group was gathered, as is normal in IT process changes). The current Administration policy, which requires ExamSoft for in-class closed hard-drive exams, was not tested in the pilot. Nor was the dual-modality (e.g. some students on ExamSoft, some students handwriting) tested to examine differences in grading results.
  5. Clear, constant, focused, and repetitive communication. IT process change programs should constantly, with the guidance of both the administration and the focus group, be distributing pertinent information to all impacted stakeholders as well as soliciting input. While the SBA respects and agrees that the decision is ultimately the Administration’s, the SBA believes that clear information flowing in both directions would have resulted in better implementation of this IT process change. Examples of lack of communication include:
  6. Failure of administration to publicly address the issues discovered in February 2002 SBA survey.
  7. Failure to provide notice and guidance on IT purchases to JD Class of 2005 despite the apparent fact that ExamSoft, and its requirements, were known prior to August, 2002.
  8. Lack of proper public announcement of new exam taking policy: The SBA believes that a single email prior to the beginning of exams is an inadequate method of communication which did not provide an opportunity for response. Preferred methods of communication for IT process change include:
  9. Article or advertisement in school newspaper
  10. Announcements in the Docket.
  11. Flyers
  12. Communication in conjunction and with assistance of focus group (did not exist) and/or with SBA (not informed prior to decision being announced).
  13. Opportunity for notice and comment.
  14. Dissemination in plain language of requirements and FAQ for the changed process (FAQs are frequently discovered and answered by the beta testing and focus group processes, as well as from best practices research, three elements missing from the ExamSoft process change).

The SBA believes that a better process roll-out would have caused the following results: