Group Software Evaluation Model

Group Software Evaluation Model

Cape Breton University

Submitted by:

Kemble Flynn

Charlotte MacKeigan

Michelle Planetta

July 24, 2006

Part A

The method of software evaluation that will be used is user survey where the administration and/or teachers and the students are used as the learners. The best case scenario of who should evaluate software is by the people who use it the most, which in our case, as educators, would be our students. However, the software should initially be selected and previewed by the teacher and/or the administration. Once the teacher and/or administration has had an opportunity to determine if the software is useful, appropriate and cost-effective, then the evaluation would be done by the students. Reiser and Dick (1990) state, “subjective-matter experts (teachers) are not able to identify software that is instructionally effective.” Reiser and Dick (1990) “believe that trying out software with learners is worthwhile as long as the obtained information is useful.” Reiser and Kegelmann (1994) verify this information by stating, “[m]ore valid information results from incorporating students as participants in the evaluation process.”

A User Survey model could be incorporated, having at least six students doing the evaluation (two students who are at a high ability level, two students who are at a medium ability level, and two students who are at a low ability level). This would ensure the full effectiveness and age appropriateness of the software. Another way to ensure that valid information is gained, as noted throughout the readings, is to establish a benchmark by administering a pretest to be used later as a comparison after a posttest, observation, and evaluation.

The software testing could be administered through a variety of scenarios such as:

ü  In the younger grades, it could be done as a center while others are doing guided reading of journal writing.

ü  If there are lunch students, this could be an indoor activity on rainy days.

ü  This could be done after school with parental permission

ü  Incorporated as an activity during math, science, social studies, health or language arts.

Part B

We have developed six phases for our model. This involves the use of rubrics, as well as pre-and post-tests to ensure that the software is appropriate for use in classrooms. Teachers will begin the process to ensure suitability for students’ viewing. From there, students will use the software and determine if they find it helpful and motivating.

Phase One would involve software evaluation. Teachers and administrators, as well as three students each at a low, medium, and high level ability would do a pretest. The following questions would be asked:

§  Is there a high level of technical quality?

§  Are the principles of learning and objectives met?

§  Does the content fit with the curriculum requirements?

§  Is it cost effective?

§  Are there teacher support resources available?

The following rubric, adapted from: http://www.arches.uga.edu/~cali/softwareeval/Rubrics.html could be used as a guideline to assess the software thus far. It would be completed by the teachers and administrators, not the students.

Phase One / Poor (0 – 1 pts.) / Satisfactory (2pts.) / Exemplary (3pts.)
Technical Quality / *Unable or hard to navigate
*Links do not work
*Difficult to install / *Relatively easy to navigate most of the time
*Most links work
*Some difficulty installing / *Easy to navigate
*All links work properly
*Easy to install
Principles of Learning and Objectives / *Incomplete and inaccurate information
*Confusing and unclear
*Information lacks purpose or central theme
*Students are not asked to apply what they have learned / *Some complete and accurate information
*Information has some purpose or theme
*Sometimes students are asked to apply what they have learned / *Complete and accurate information
*Clear and precise information
*Students asked to apply what they have learned by using higher level thinking skills
Content/Curriculum Fit / *Pertains to few or none of the SCO’s and/or GCO’s / *Pertains to most of the SCO’s and/or GCO’s / *Pertains to all of the SCO’s and/or GCO’s
Cost Effective / *Software is expensive
*Software not able to be licensed for multiple users / *Software is a fair price
*Software allows for only limited users (less than 10) within school / *Software is very reasonably priced
*Software is licensed for unlimited users within school
Teacher Support / *Teachers cannot assess student’s progress
*Does not provide extra tips or supplies for teachers / *Teachers can sometimes assess student’s progress
*Provides some extra tips or supplies for teachers / *Teachers can assess student’s progress
*Provides extra tips or supplies for teachers
Results

Phase 2 would be the evaluation of the results from the pretest with teachers and administrators. Under this phase, the feedback is reviewed and it is decided whether or not the students are allowed to test the software. If the feedback is negative, we do not move any further with this specific software. Phase 1 will begin with a new program. If the feedback is positive, we move on to Phase 3. Questions such as the following could be posed:

§  Do you feel the software is appropriate to be tested by other students?

§  Do you feel the software will help in your learning?

Phase 3 is the testing of the software with students. Which students? How many? The students would be identified by the teachers’ that had done the Pretest in Phase 1. The optimal sample size for testing the software would be the whole class (this could be done through centers); however, six students would probably be enough (2 at a low level ability, 2 at a medium level ability and 2 at a high level ability). If more than one teacher went through Phase 1, then more than one class would be participating in Phase 3. Through observation, teachers could take note of any motivation the students have gained through the use of the software. Other questions worth considering are:

§  Is it user friendly?

§  Are there social/diverse considerations?

Phase 4 involves the evaluation of feedback from Phase 3. If these three points are met, and should “the software…. move up here?

the software continues to be promising, we want to proceed to phase 5. If less than three are met, discussions would occur to decide how critical these factors are in the overall scheme of the software. Recommendations would be made at this point if the school was going to implement the software or start over again with a new software title.

If all previous four phases are satisfactory, then Phase 5 would occur. This is the implementation of software into the classroom. At this phase, teachers would have been inserviced or have had time to do Phase 1 of our model where they have the chance to go through the Pretest of the software. Any reference materials and/or handouts would be distributed at this time.

Phase 6 is the posttest of the software. This ensures that the software is being used accordingly and that the students are achieving success.

Do you think the posttest should be phase 5 and the implementation phase 6? I think it would be a waste of money if we implemented it before doing the posttest and then found out it was no good. Good point, the posttest could be put into phase 5. So maybe the following could go under Phase 5:

The posttest would be done on the original students who were involved in Phase 1. Their pretest, testing and posttest results would be the collected data to review and ensure the software is being used accordingly and that students are achieving success.

Part C

My mind is tired. Can you please read through this and ensure that it flows. I simply cut and paste paragraphs that you both contributed.

The purpose of this model is to identify software that promotes student success and achievement of the outcomes. With this purpose in mind it is therefore important to include students in the evaluation process. This can be achieved by enabling them to have a voice in the software evaluation process. That being said, it seems only natural that the very people who will be impacted by software introduced in the classroom will be the ones who evaluate it. This evaluation model addresses many important characteristics to help ensure that the software is appropriate, useful, engaging and supportive in the learning process all of which is determined by the user, our students.

One of our group members allowed us to use her classroom philosophy that shows our group, Summer Dreamers, beliefs about education. “I believe that students should take an active role in their learning. My classroom is a place where the students are partners with myself in learning. My role as the teacher is to help the students develop a sense of self-confidence, self-worth and help develop the students critical thinking abilities. Students learn best when they are active learners being challenged inside and outside the classroom. Hands on activities play a big role in helping my students develop their skills and abilities. My classroom helps each student find their own voice” (http://www3.ns.sympatico.ca/walter.flynn/).

Komoski et. al (1995) discuss the importance of gathering post-use feedback. They outline that that this feedback can serve to be valuable in analyzing appropriate software. Further, they explain that the post-use feedback should serve to identify meeting the outlined objectives with actual student performance.

Komoski et. al (1995) discuss that post-use feedback is also useful for determining the next direction to take in software selection. They state that “[p]ost use feedback can be significant to help to a school’s systematic process of software selection, purchase and use. The accumulation of user feedback, including anecdotal experience on the part of both teachers and students, will naturally serve to improve future needs analyses.” This is shown in our model in Phase 5.

De Laurentiis (1993) discusses that when learning is individualized and effective that students enjoy the learning process and seek to learn more. It is for this reason

De Laurentiis states that “…student control is so important.” He goes on to discuss that giving students an opportunity to control their learning enables them to form important links to their already existing knowledge. De Laurentiis also states that “[e]ducational software should provide access to all of the knowledge and activities it contains under student control.” That being said, this reinforces our view that students need to have a say in the software they use because they as individuals are the ones who are most greatly impacted by the implementation.

Part D

Teacher Checklist

Title of Software: ______

Platform/Version: ____ Mac ____ Window

Media: ____ Diskette ____ CD-ROM ____ DVD

Other requirements: ____ Internet ____ Microphone ____ Other ______

TECHNICAL QUALITY

Please place a checkmark in each applicable section:

Installation and Setup: ____ Difficult ____ Time Consuming ____ Simple

Sound is: ____ High Quality ____ Low Quality

Videos: ____ Run jerkily ____ Run smoothly

Graphics: ____ Age Appropriate ____ High Quality

____ Low Quality ____ Realistic ____ Not Realistic

____ Cluttered

Colours: ____ Black & White ____ Vivid Colours ____ Use of both

____ Didn’t matter

Printing: ____ Available _____ Not Available

Saves student’s work: ____ Available _____ Not Available

Place a checkmark in any of the areas listed below that may have drawn attention away for the effectiveness of this software:

____ Sound

____ Videos

____ Graphics

____ Colours

____ Too much clutter

For each question below please circle Yes or No.

USER-FRIENDLY

Can you easily manipulate the software? Yes No

Is the software interactive? Yes No

Are the graphics meaningful for students? Yes No

Does the program offer students feedback? Yes No

Is the software self-paced? Yes No

COMMENTS: ______

PRINCIPLES OF LEARNING & OBJECTIVES:

What age group is the software intended for? ____ years

Does this software help student achieve the prescribed outcomes? Yes No

Are the students actively engaged in the software? Yes No

Does the software offer different levels of difficulty? Yes No

Does this software meet the instructional needs of Yes No

the students?

If YES, is it easy for students to switch to a higher or lower level Yes No

of difficulty?

Once a level has been mastered, can the student be Yes No

challenged further by the program?

Do the students learn the skills set out by the software? Yes No

Is it possible for students to apply the skills from the software Yes No

to other work?

Does the software encourage independent learning? Yes No

Does the software promote creativity? Yes No

Does the software promote discovery? Yes No

Does the software promote collaboration? Yes No

Does the software promote high-order thinking? Yes No

Does the software promote problem solving? Yes No

Does the software promote memorization? Yes No

Does the software use drills? Yes No

COMMENTS: ______

CONTENT/CURRICULUM FIT:

What grade(s) do you teach? ______

Circle each subject(s) that the use of this software may be useful.

Math Science Language Social Studies Health

Circle the grades level(s) that this software is useful for?

P – 2 3-6 7-9 10-12

Do you feel the software supports the general curriculum Yes No

outcomes in the appropriate subject area?

Does the software use appropriate subject-specific language? Yes No

Does this software promote cross-curricular learning/instruction? Yes No

Does the level of instruction match the age and level of the students? Yes No

Is the vocabulary appropriate for the age and level of the students? Yes No

Doe the software reflect safe practices? Yes No

Does the software use Canadian references and examples? Yes No

COMMENTS: ______

MOTIVATION:

Are students motivated to use this software? Yes No

Does the software stimulate curiosity with the user? Yes No

Is the software challenging for students? Yes No

Are there real-world connections to help motivate the user? Yes No

Does this software promote students to strive for a greater Yes No

level of success?

Would students recommend this software to a friend? Yes No

COMMENTS: ______

SOCIAL/DIVERSE CONSIDERATIONS: