Evaluation Of Seminars, Workshops And Journal Clubs

UNIVERSITY OF THE VIRGIN ISLANDS RESEARCH INITIATIVE FOR SCIENCE ENHANCEMENT (RISE) PROGRAM: MID YEAR EVALUATION REPORT AUGUST 2001-JANUARY 2002

Background

The RISE program is one of several collaborative training/research programs in the Sciences Division of the University of the Virgin Islands. These programs include: (1) The Minority Access to Research Careers ( MARC) program funded by NIH, a faculty/student research program conducted, in part, in mainland university labs, (2) a NASA scholarship program, and a National Science Foundation (NSF) grant designed to promote excellence in science, mathematics , engineering and technology. These programs are under the umbrella of Emerging Caribbean Scientists program and commonly share the goal of increasing the number of UVI students who are skilled, motivated and confident about their preparation for biomedical and other scientific careers.

Overview of the RISE Program

The goal of the RISE Program is to overcome the geographic and intellectual isolation of the University of the Virgin Islands and increase the number of graduates that are skilled, motivated, and confident about their preparation to enter graduate studies in the sciences. This is accomplished through a three-step process involving students, faculty and The University of the Virgin Islands. Collaboration is structured between UVI faculty in different biomedical disciplines, students, the university, and mainland scientists in approved laboratories.

The student development portion is supported by exposure to a inquiry-based curriculum, research experiences, presentations of research results, participation in seminar series and special workshops, clubs, study groups and travel to mainland universities. Participation in research programs on the mainland provides opportunities to interact with role models and broaden experiences in a student’s chosen field.

The faculty development portion provides faculty the opportunity to initiate collaborations with outstanding research scientists and travel to mainland laboratories with the goal of enhancing their professional careers and productivity.

The institutional development portion provides renovated space, capital equipment, materials, staffing, upper level course redesign to modernize molecular biology training, and the revamping of general chemistry.

Description of RISE Participants and Mentors

1. RISE Participants 2001-2002

There are ten students supported by the RISE program in the 2001-2002 academic year. Eight are female and two are male. Four are sophomores, four are juniors and two are seniors. All participants have a GPA of 2.5 or better, are US citizens and are classified as a minority student. All students selected for the RISE program have demonstrated, by participation in other research based activities and written documentation, an interest in a career in biomedical research and motivation to obtain a Ph.D. In addition to the ten students who are the cohort for the 2001-2002 school year, three additional students were designated RISE students during the summer term, and received support for mainland research projects.

Mentors

Mentors for the RISE Fellows consist of two groups : There are five UVI based mentors at the University of the Virgin Islands and three at mainland universities throughout the United States. The criteria for being a mentor are being an experienced researcher associated with a funded research laboratory.( NSF, NIH, etc.) and a willingness to mentor undergraduate students.

Mentor/Collaborators

University researchers located at mainland universities also serve as collaborators with UVI faculty in the science and math division .

RISE Goal and Specific objectives

Goal

The specific goal of the RISE program is to provide sufficient opportunities, training, mentoring and advising to students coupled with experiences to strengthen faculty productivity that result in 50 percent or more of RISE students enrolling in graduate programs leading to a Ph.D. degree.

Objectives

The objectives listed are designed to be implemented during the projected four years of UVI RISE project. Therefore, and not all objectives are addressed during the first year ( see appendix 1)

1. Students will be provided the support to excel in gatekeeper courses (general chemistry, general biology, calculus, and physics).

2. Retention of RISE freshmen students in biology, marine biology and chemistry programs will exceed 60% in school year 2002-2003

3. Student will have greater interest in and awareness of diversity of biomedical research after attendance at seminars and scientific meetings as measured by survey responses

4. Students will become active scientific researchers as demonstrated by presentations at science seminars, scientific meetings, and reports from mentors. At least 2 RISE students will co-author publications in refereed journals.

5. Students will be knowledgeable in their field as demonstrated by course grades, mentor and faculty evaluations

6. Student confidence will increase as demonstrated by performance in course work, application to graduate programs, scientific presentation, and co-authorship of articles.

7. Faculty in the Division of Science and Math will develop significant collaborative relationships which will lead to productive research careers as measured by progress reports from summer research experience, submitted manuscripts and grant proposals.

8a.Modernized molecular biology instruction as demonstrated by refurbished lab, modular course development and implementation.

8b.Restructuring of the General Chemistry curriculum

Evaluation Plan of the RISE program

Introduction

The goal of the RISE Evaluation Plan is to measure the progress in achieving the stated goals and objectives of UVI RISE by assessing whether stated objectives are being met, activities implemented, and the efficacy of methods, materials, curriculum, and design being utilized to achieve them. The evaluation is a collaborative effort among MBRS-RISE staff, UVI office of Institutional Research, Program Directors of other MBRS projects , and the outside evaluators, Lce Consulting. Evaluation is formative as well and summative in order to provide timely feedback to stakeholders which provides the opportunity to improve the project as it develops. The ongoing evaluation process gathers data during the course of the project. The MBRS-RISE staff collects, tabulates, and forwards to the external evaluators a variety of data gathered on RISE students within the University’s program. The External evaluators analysis and interpret this data in addition to designing instruments to gather data, conducting observations, interviews, focus groups, record reviews, research and providing a written report.

Evaluation Plan

Table 1 ( appendix 1) presents the evaluation questions that framed the evaluation design, methods of data collection, and responsibility allocations. Figure A summarizes the research methods and data collection used to complete the present evaluation.

Overview of Evaluation Activity August 2001- February 2002

RISE evaluation activities consisted of observations of Poster Presentations ( July 30 and September 22), design of student and mentor survey forms (see appendix 2), administration of student and mentor survey forms, faculty/mentor interviews,

FIGURE A

participation/ presentations at the joint MBRS Marc and RISE advisory council meeting, meetings with RISE project Director, conducting focus groups, document reviews, data collection and analysis, and reporting

Poster Presentation:

The observation of Poster presentation (July 30) was to gather baseline data relative to objectives 4 and 6 and to establish rapport with participant stakeholders.

Only three of the six RISE students who had a summer research experience presented on July 30 due to weather condition that prohibited St. Croix participants from attending. The participants were in the areas of Math, Computer Science/Engineering, and Psychology. The three poster presentations were observed, photographed and RISE presenter interviewed. The evaluator, Dr. Creque, gathered data on confidence level, opinions on summer mentoring experience and plans for graduate school. Each participant was observed for approximately 25 minutes.

Confidence level was defined as the ability to:

· -answer all questions posed by visitors and evaluator

· -explain poster exhibit in detail

· -speak knowledgeably about content

· -make eye contact with visitors and observer

· -remain relaxed (smile, speak slowly and distinctly, etc)

Each criterion had a value of 20. Using these criteria , during the period of observation all three students met the five criteria of exhibiting confidence. (minimum scored: 80)

Summer Mentoring Experience:

RISE participants were asked to describe their summer mentoring experience. All responses were positive. Comments were as follows:

Got to see what I was doing. Wonderful to have one-on-one.

I had an opportunity to participate in almost everything. The experience more intense than two weeks in a regular class.

The research encouraged me to go on to graduate school. (I see) what I am moving toward, I can only achieve at the Masters level.

Helped me to focus on what to study in grad school .

Plans for Graduate school

All three students interviewed plan to go to graduate school, however, their discussion and comments center on the Master’s level. Only one student, when asked, responded the continuing on to a Ph.D. was “possible.”

Development and Administration of RISE Student and Mentor Evaluation Forms

RISE Students

Survey forms were designed to collect baseline data on RISE students assessment of their likelihood of entering a Ph.D. program, the quality of counseling and support , level of proficiency in their field and awareness of the biomedical or other research areas after

their summer RISE experience These questions relate to the overall goal of the RISE program and specific objectives, 2,3, 4, and 6.

RISE Mentors

Survey forms for RISE mentors paralleled question in RISE Student Self-report survey instrument. Mentors were asked to assess how likely it was for their mentee to enter Ph.D. program, proficiency in content area and medical research, and their level of awareness of biomedical and other research fields. Mentors were also asked to assess their ability as a mentor.

Direct contact was made with RISE Students and Mentors to explain the survey and respond to concerns. Mentors located on the mainland were telephoned. Only one mentor was not spoken to directly and contact was made through Email.

Results: RISE Program for 2001

There were seven students and five mentors that responded to the program evaluations. Six out of seven of the students said that they intended to pursue a Ph.D. program. Of those expressing a desire to continue, 5 students said that they were planning to wait at least a year before going on, and one student said that they were planning to go in the next 10-12 months. The person who said that they didn't want to go on said that they wanted to go into medicine rather than get a Ph.D. Some of the universities students are considering are Virginia Commonwealth, Howard, Georgia State, Florida State, Georgia Institute of Technology, Stanford, Boston and Ross.

Both students and advisors tended to rate nearly all areas of the quality of academic advising high. Graph 1 compares student and advisor responses. There were only two areas where there appears to be a significant difference between the means of the student and mentor responses (p <.05) These are in Course Selection and in Encouragement to do Research Projects. The mentors rated themselves lower than the students

Graph 2 compares how students and mentors feel about the level of proficiency that they have in certain areas as a result of the RISE Program. Over all, students and mentors agree on the level of proficiency. One additional area is Poster Presentation. The two areas that needed work were layout and content. The mean score was 7.14 out of a possible 10 points for layout and 14 out of a possible 16 for content.

Graph 3 compares the students and mentor's perceived awareness in several areas following participation in the RISE Program.

Awareness (p<.05) are of the Current Issues and Current Scholars in the Field. The two areas in which the differences between means are significant are in the mentors rated both of these areas lower than the students.

Finally, the students evaluated the summer program. There were only three responses to the evaluation. This makes analysis difficult. Overall, however, students expressed that their summer experience helped them to gain an exposure to research and research practices, develop critical thinking skills, and gave them practical experience in research skills. All of the students said that they would recommend the program to another student and would attend another research program in the sciences.

The two areas in which the differences between means are significant (p<.05) are in the awareness are of the Current Issues and Current scholars in the field. The mentors rated both of these areas lower than the students.

RISE Evaluation : August 2001- January 2002 .

The following is an evaluation of RISE program activities for the 2001-2002 fall semester.

Poster Presentation September 22, 2001

Seven MBRS-RISE students made poster presentation at the First Annual Fall Research Symposium held September 22, 2001 based on summer research experience.. The evaluator looked at the quality of the poster presentation and the confidence level of the presenters, using the same criteria developed for July 30th presentation. In addition the quality of the poster was evaluated. A Poster Presentation Evaluation Form was used to determine the quality of poster presentation. Posters were judged on:

· title banner

· text and figures integration

· layout

· writing

· graphics

· content

with a possible maximum score of 50, with 35 being the least possible score to be judged satisfactory.

Results

Scores ranged from a high of 47 to a low of 32. Three student scored between 50-40, two students between 39-35 , and two students below 35. However, it should be noted that both instruments are being refined and will benefit from having more than one observer assess quality of the presentation.

Four of the seven students who were not present at the July 30th presentation were observed for levels of confidence using the same criteria. Confidence levels remained positive with all students meeting the five criteria.( minimum score 80)

Evaluation of Seminars, Workshops and Journal Clubs

The purpose of presentations, workshops, and seminars is to increase awareness of scientific fields of studies, expand knowledge base, provide role models and support research activities. During the Fall Semester there were nine presentations/workshops/seminars provided for RISE participants under the umbrella of the Emerging Caribbean Scientist (ECS) program.