STUDY ISLAND SOFTWARE EVALUATION PLAN

Study Island Software Evaluation Plan

Joy B. Renfro

Introduction

The focal point of this evaluation will be the usage of Study Island software at Calhoun Elementary School. Teachers at Calhoun Elementary have noted significant gains in student achievement since the implementation of Study Island two years ago. However, it is difficult to ascertain whether these gains are due to Study Island usage or to one or more of the other school wide initiatives that have been put into place over the same time period. Therefore, this evaluation will be used to gather data about the emphasis that is placed on Study Island in the classroom, how it is being used by teachers and students, and how the most effective methods for implementing Study Island can be made more prevalent and pervasive in the school. The client for this evaluation is Wesley Roach, principal of Calhoun Elementary School and the primary stakeholders are students, teachers, parents, and administration.

To establish a foundation for this evaluation and to guide its development, a review of relevant literature was conducted. In the majority of today’s schools, technology is taking on an increasingly important role. As stated by Englert, Manalo and Zhao (2004), “ the mediating quality of technology has particular appeal for supporting the participation of very young students or students with learning difficulties in intellectual activities that are beyond their immediate grasp” (p. 5). They continue to describe the benefits of using technology for scaffolding student learning, enabling students to perform at more advanced levels of understanding (Englert, Manalo, & Zhao, 2004). However, in light of the economic constraints that are prevalent in the majority of America’s schools, funding cannot be devoted to technology solely for the sake of technology. According to Jones, Staats, Bowling, Bickel, Cunningham, and Cagle (2004), “many policymakers and community members want evidence that technology supports student learning as they make decisions about technology investments” (p. 177). Therefore, schools must constantly evaluate the programs that are in use or under consideration for adoption to ensure that they will have the desired impacts on student learning.

In the wake of No Child Left Behind policies, a magnitude of software programs that are marketed to schools hoping to boost the test scores of their students. Some of these programs have been carefully evaluated and proven to have positive results on student achievement. One such study of the Merit reading software program conducted by Jones, Staats, Bowling, Bickel, Cunningham, and Cagle (2004) found that “computer based instruction, when used to complement everyday instruction by teachers, can significantly improve the performance of…students’ reading skills as measured by standardized tests” (p. 191). Mendicino, Razzaq, and Heffernan (2009) found that “students learned significantly more with Web-based homework than with paper-and-pencil homework” (p. 342). Their findings are particularly interesting given that Study Island is designed to be used at home as well as at school.

Although the majority of the available research is supportive of the usage of technology and software programs in the elementary classroom, some programs have been found to be ineffective. In their study of thirteen software programs commonly used in Canadian and American schools to support reading and writing skills, Lovell and Phillips (2009) found that none of the programs were directly instructional. They noted several obstacles that hinder technology integration, including usage of the software in an unnatural context as well as inappropriate or partial use of the programs (Lovell & Phillips, 2009). It was also noted by Lovell and Phillips (2009) that “the majority of the programs…do not track student progress, provide feedback, or adapt to suit student needs, thereby limiting their usefulness as educational tools” (p. 211). As Study Island does all of these things, it is hoped that the findings of this evaluation will be more encouraging.

As society becomes increasingly dependent on technology, it is logical that technology will become more widespread in schools as well. However, as Parette, Boeckmann, and Hourcade (2008), the research base for the usage of technology in the elementary classroom is still emerging. Educators cannot adopt or continue to use a program simply because it is technology-based. Student learning must continue to be the heart of any initiative, and evaluation of the software programs that are available should continue.

Purpose

The purpose of this evaluation is to examine the implementation and usage of Study Island software. It will determine the amount of time and emphasis placed on the program within the classroom and how teachers encourage their students to use it. It will also determine students’ and teachers’ attitudes toward Study Island usage and uncover some of the most effective means of integrating it in the classroom. This evaluation will be both formative and summative, in that it will provide data about Study Island usage at this point but will also yield data about how it can be implemented more effectively. As a result of this evaluation, the client will be able to determine how efficiently and effectively Study Island is currently being implemented at Calhoun Elementary School and make decisions about the future of the software’s usage.

Evaluation Questions

The goal of this evaluation study is to determine the extent to which Calhoun Elementary students are using Study Island software and how teachers are incorporating it into their classrooms. Common practices for using Study Island in the classroom will also be examined and evaluated to determine their effectiveness.

  1. What’s the average amount of time a student in each grade level spends on SI while at school in a group?
  2. What’s the average amount of time a student in each grade level spends on SI while at school as an individual?
  3. What’s the average amount of time a student in each grade level spends on SI outside of school hours?
  4. Do teachers feel that there is a relationship between SI usage and academic growth?
  5. What different ways are teachers using SI with their students?
  6. Do some ways SI is being used by teachers with their students yield higher results?

Methods

Participants

The participants in this evaluation study will include classroom teachers from grades three through five at Calhoun Elementary School. These teachers have varying amounts of experience and expertise. The participants in this evaluation study will also include students from grades three through five. These students will represent various levels of academic ability and accomplishment. They will include regular education students, EIP students, special education students, ESOL students, and gifted students.

Design and Procedures

The following activities will be used to address the evaluation objectives and questions:

  1. Evaluation questions 1-3: Surveys will be distributed to homeroom teachers in grades three through five as well as to their students. Questions included on the surveys will gather information about the frequency of Study Island usage at school and at home.
  2. Evaluation question 4: The teacher surveys will include questions about teachers’ impressions of the relationship between Study Island usage and student academic growth. This study will rely only upon qualitative data for this objective in the form of teacher opinions rather than quantitative data such as test scores.
  3. Evaluation question 5: As part of the survey, teachers will be asked to describe the methods that they use to encourage student usage of Study Island. Teachers will be asked to describe their practice of using Study Island in the classroom as well as any techniques that they use to encourage student usage of Study Island at home.
  4. Evaluation question 6: Study Island usage reports are available from the software program and will be provided by the client. These reports detail the amount of time that students have spent on Study Island during a given timeframe as well as the rate of accuracy. The reports reflect grade level and classroom data, but contain no identifying student information. Teacher methods for implementing Study Island in the classroom and for encouraging its usage at home will be compared with the data in the usage reports, helping to identify the most effective means of promoting Study Island usage.

Instruments

The instruments used in this evaluation study have been developed by the evaluator and approved by the client. Validity will be ensured by the professor. Instruments include:

  1. Teacher surveys: These surveys will include both closed ended and open ended questions. Teachers will be asked to report on their methods and frequency of using Study Island in the classroom. They will also be asked questions dealing with their impressions of the effectiveness of the software and ideas for making its usage more prevalent and efficient. These surveys will be created in GoogleDocs and emailed to the teachers. No identifying data, other than demographical information, will be gathered.
  2. Student surveys: These surveys will include only closed ended questions and will be administered to each classroom by the teacher. Students will be asked questions about their Study Island usage and impressions of the software program.
  3. Study Island usage reports: These reports will be provided by the client and will provide comparison data. The report data does not include any student identifying information, but does provide information about the amount of time students spend on Study Island, the number of questions that they answer, and the accuracy of their responses during a given time frame.

Teacher Surveys / Student Surveys / SI Usage Reports
Questions 1-3: SI Usage / X / X
Question 4: SI Usage/Academic Growth / X
Questions 5: SI Usage Methods / X
Question 6: Methods and Results / X / X

Data Analysis

Teacher Surveys

The evaluator will carefully analyze the responses to each survey question to uncover response patterns. Questions that were formatted to reflect the Likert scale will be averaged. The responses will be further analyzed based on demographic information reported by the respondents such as teaching experience, familiarity with technology, and age. Open-ended responses will be grouped by similarities.

Student Surveys

The evaluator will carefully analyze the reponses to each survey question to uncover response patterns. Questions that were formatted to reflect a modified Likert scale (to make the surveys more child-friendly) will be averaged.

Study Island Usage Reports

The evaluator will analyze the data provided in the Study Island usage reports and compare the results with the data gathered from the teacher and student surveys. The evaluator will look for patterns and trends in methods used frequently by teachers and their students’ usage of and success with Study Island.

Conclusion

The results of this evaluation will be compiled in a report that will be made available to the client and supervising professor. The report will be available on or before April 11,2011.

References

Borman, G. D., Benson, J. G., & Overman, L. (2009).A randomized field trial of the fast forword language computer-based training program.Educational Evaluation and Policy Analysis, 31(1), 82-106.

Englert, C. S., Manalo, M., & Zhao, Y. (2004). I can do it better on the computer: the effects of technology-enabled scaffolding on young writers’ composition. Journal of Special Education Technology, 19(1), 5-21.

Jones, J. D., Staats, W. D., Bowling, N., Bickel, R. D., Cunningham, M. L., & Cadle, C. (2004). An evaluation of the merit reading software program in the calhoun county (wv) middle/high school. Journal of Research on Technology in Education, 37(2), 177-195.

Lovell, M., & Phillips, L. (2009). Commercial software programs approved for teaching reading and writing in the primary grades: another sobering reality. Journal of Research on Technology in Education, 42(2).197-216.

Mendicino, M., Razzaq, L., & Heffernan, N. T. (2009).A comparison of traditional homework to computer-supported homework.Journal of Research on Technology in Education, 41(3), 331- 359.

Parette, H. P., Boeckmann, N. M., & Hourcade, J. J. (2008). Use of writing with symbols 2000 software to facilitate emergent literacy development. Early Childhood Education Journal, 36, 161-170.

U.S. Department of Education Institute of Education Sciences (March 2010). WWC intervention report: plato achieve now. Retrieved from

Appendix A

Teacher Survey

  1. What grade level do you teach?

345

  1. How many years have you been teaching?

1-56-1011-1516-20 21 or more

  1. What is the average amount of time students in your classroom spend on Study Island while at school each day?

0-5 minutes5-15 minutes15-30 minutes>30 minutes

  1. How do students in your classroom typically use Study Island?

Individuallypartnersmall groupwhole class

  1. Given the limited number of classroom computers, which students receive priority when using the computers (such as for Study Island)? Check any that apply
  • Students who have completed assignments
  • Students who do not have computer access at home
  • Students who are well behaved
  • Low performing students (such as those receiving tier 2 or tier 3 interventions)
  • I have a rotating schedule
  • No students are given priority – computer usage is first come, first serve
  • Other ______
  1. How would you rate the effectiveness of Study Island? (5 being highest)

12345

  1. How would you rate the amount of time that students are able to spend on Study Island? (5 being highest)

12345

  1. Do you think that Study Island helps students grow academically? (5 strongly agree)

12345

  1. How would you rate your comfort level with using Study Island (to generate reports, make assignments, evaluate student progress, etc.) (5 being highest)

12345

  1. How do you rate your comfort level with using technology in the classroom in general? (5 being highest)

12345

  1. In your opinion, what are the main limitations of Study Island?
  1. What methods do you use to encourage students to use Study Island more frequently?
  1. What would enable you to implement Study Island in your classroom more effectively?

Appendix B

Student Survey

  1. How often do you use Study Island at school?
  • Never
  • A few times each week
  • Everyday
  1. How often do you use Study Island at home?
  • Never
  • A few times each week
  • Everyday
  1. When I use Study Island, I am usually
  • Alone
  • Working with a partner
  • In a small group
  • Working with the whole class
  1. How do you like Study Island?
  • Not at all
  • It’s okay
  • I like it a lot
  1. I wish…
  • I could use Study Island more
  • I could use Study Island the same as I do now
  • I did not have to use Study Island at all

Data Collection Organization Matrix

Questions / Related
Results / Level
Mega
Macro
Micro / Required Data/
Measurable
Indicators / Data
Source / Data
Collection
Tools/
Procedures / Data
Collected
By (Date) / Primary
Responsibilities
What’s the average amount of time a student in each grade level spends on SI while at school in a group? / Students will possess the skills and abilities necessary for success according to the requirements set forth in the Georgia Performance Standards.
Students will be able to read and write fluently, solve appropriate math problems, and possess a sound understanding of suitable scientific and social studies concepts. / Macro
Micro / Data is gathered by evaluator through surveys / Students and teachers / Student and teacher surveys / 3/11/11 / Creating and distributing surveys
What’s the average amount of time a student in each grade level spends on SI while at school as an individual? / Students will possess the skills and abilities necessary for success according to the requirements set forth in the Georgia Performance Standards.
Students will be able to read and write fluently, solve appropriate math problems, and possess a sound understanding of suitable scientific and social studies concepts. / Micro
Macro / Data is gathered by evaluator through surveys / Students and teachers / Student and teacher surveys / 3/11/11 / Creating and distributing surveys
What’s the average amount of time a student in each grade level spends on SI outside of school hours? / Students will possess the skills and abilities necessary for success according to the requirements set forth in the Georgia Performance Standards.
Students will be able to read and write fluently, solve appropriate math problems, and possess a sound understanding of suitable scientific and social studies concepts. / Macro
Micro / Data is gathered by evaluator through surveys / Students / Student surveys / 3/11/11 / Creating and distributing surveys
Do teachers feel that there is a relationship between SI usage and academic growth? / Produce society members who possess the skills and abilities necessary for success, including reading fluency and comprehension, mathematical reasoning, and civic understanding and responsibility.
Students will possess the skills and abilities necessary for success according to the requirements set forth in the Georgia Performance Standards.
Students will be able to read and write fluently, solve appropriate math problems, and possess a sound understanding of suitable scientific and social studies concepts. / Mega
Macro
Micro / Data is gathered by evaluator through surveys / Teachers / Teacher surveys / 3/11/11 / Creating and distributing surveys
Analyzing and communicating findings
What different ways are teachers using SI with their students? / Produce society members who possess the skills and abilities necessary for success, including reading fluency and comprehension, mathematical reasoning, and civic understanding and responsibility.
Students will possess the skills and abilities necessary for success according to the requirements set forth in the Georgia Performance Standards.
Students will be able to read and write fluently, solve appropriate math problems, and possess a sound understanding of suitable scientific and social studies concepts. / Mega
Macro
Micro / Data is gathered by evaluator through surveys / Teachers / Teacher surveys / 3/11/11 / Creating and distributing surveys
Analyzing and communicating findings
Do some ways SI is being used by teachers with their students yield higher results? / Produce society members who possess the skills and abilities necessary for success, including reading fluency and comprehension, mathematical reasoning, and civic understanding and responsibility.
Students will possess the skills and abilities necessary for success according to the requirements set forth in the Georgia Performance Standards.
Students will be able to read and write fluently, solve appropriate math problems, and possess a sound understanding of suitable scientific and social studies concepts. / Mega
Macro
Micro / Data is gathered by evaluator through surveys / Teachers, Study Island software / Teacher surveys and Study Island usage reports / 3/18/11 / Creating and distributing surveys
Obtaining Study Island usage reports and comparing them to results from teacher surveys
Analyzing and communicating findings

1