E-Portfolio Evaluation - Interim Report

E-Portfolio Evaluation - Interim Report - March 2007

The following table provides an attempt by the evaluation team to summarise how well the two e-portfolio packages are performing against the criteria previously agreed by the PFWG. We have arrived at qualitative, synoptic judgements drawn from the ongoing experiences of the pilot initiatives within the university, and experiences elsewhere, including:

data from 228 student responses to an initial survey that students are invited to complete at the beginning of the pilot (administered September 2006 and February 2007)
data from 52 student responses to an interim survey (administered December 2007)
experiences and views of staff involved in the pilot activities gathered at various points during the evaluation
insights drawn from the Enhancing Learner Progression Project
observations from a staff and student meeting with a representative from pebblePAD organised by the School of Health (March 2007)

We have used the following simple scoring system for each package against each criteria:

+ overall helps to meet this need

0 neutral - balance of positive and negative features

- overall concerns about meeting this need

? insufficient evidence at this stage

Peter Hughes, Senior Lecturer in Learning Development, TQEG

Carol Higgison, Senior Advisor on E-learning, TQEG

Criteria / pebblePAD / VLE ePortfolio
Score / Comments / Score / Comments
Institutional Needs
Provide a "private electronic space for students to store notes and any outcomes from PDP activities". / + / Clear and self evident. pP was designed to support PDP and this shows. Very much perceived as a 'student-owned' provision. / - / Can be perceived by students as an institutional space. Confusion between content and portfolio areas does not aid usability. Isn't a PDP tool (although can be used for this with considerable staff effort and guidance).
Enable/allow staff to provide guidance and support for the PDP process. / + / Staff can share materials and templates within pP. Version 2 (available from July) will allow form building which will considerably enhance this capacity. / 0/- / Possible, but not straight forward. Sharing an e-portfolio template with whole class is difficult. Support resources can be placed within main Vle site (not portfolio).
Allow or facilitate "student and tutor access [to] documents produced [by the student] for review and/or assessment". / + / A student-owned, one-step process. Students can share items with staff and peers, or can 'publish' items for assessment by tutors. Guidance is needed, but a relatively simple process. / 0 / Can do it, but technically cumbersome and can be a complex process to follow.
Cannot easily share individual items (only the whole portfolio).
Difficult for tutors to manage/administer large numbers of portfolios.
An e-portfolio system that complements and enhances existing PDP activity. / +/? / Package seems to be amenable to most existing activity, but also stimulates new ways of thinking. In-built support and guidance provides an easy way into PDP for students.
Version 2 will be even more customisable. / -/? / Pilot activities have generally been for 'new' activities. Had to be dropped in Dementia Studies as student experience was so negative, so reverted to existing paper-based practice.
That is subject to staff approval rather than resentment. / +/0 / Balance of staff comment from those involved in the pilot is positive. There is though an issue in it being perceived as 'another' package (additional to Bb) to learn how to use. / -/0 / Staff like the idea of the e-portfolio being embedded within the existing Vle platform, but actual experience of using it has generally been negative.
That has the capacity for enhancing the overall student experience. / + / In built support makes it easier to develop reflective capacity.
"I like the way that it takes you step by step through the system" (Student comment)
"The program has good prompts helping structure whatever I'm doing easily." (Student comment) / 0 / Students involved in the pilot have been learning things, but see this as IT skills rather than an overall enhanced learning experience. Positive experiences have been balanced by significant negative experiences, mainly due to usability, the complex nature of the system.
"In the beginning it seemed interesting but after using it started to be boring, complicated." (Student Comment)
An e-portfolio system that is affordable in terms of subscription, training and development needs and ongoing central support. / +/? / Will need to secure budget for institutional subscription. Will need designated staff to support the development and support of staff and materials initially, and also for student support. Pilot experience has shown that after an initial investment in training staff and students to use the package they can be relatively self-sustaining. Gateway management needs to come from centre, but most other aspects can be devolved to tutors. / -/? / Requires subscription to content management of Bb. Likely to require significant investment in development and support. All support needs to be centrally managed.
A system that is flexible enough to accommodate an evolution of the institutional approach to PDP, such that a replacement is not required every time we adjust our practice. / + / System is designed around and supports basic principles of PDP, so should be flexible enough to support any approach to PDP. / - / System is essentially content free - this means that it can (with effort) be built to support a particular approach to PDP, but would need to be completely rebuilt if requirements changed.
A system that we are confident will be subject to ongoing development and improvement by the supplier. / + / pP have already proved very responsive to the needs of Bradford staff (e.g. developing enhanced mobile platform at our request.) As an early adopter we have been able to help shape the development of the package and will continue to do so. Company is UK based and focused. / -/? / Uncertainty over Vle plans regarding eP provision raises questions about how sensible it would be to invest resources into developing materials and processes for the current version.
A system that provides added value, e.g. useful for staff PDP as well as student PDP. / + / Some staff involved in the pilot already reporting use for their own CPD.
Projects at UC St Martins, Wolverhampton and elsewhere are developing this more systematically. / 0/- / Has the potential advantage of being able to link to materials that staff have created within Vle for their teaching (e.g. in a teaching portfolio), but concerns over perceived institutional ownership. Supports one-off presentational portfolios, rather than the process-oriented PDP/CPD portfolios
2. Staff needs
A system that can support and enhance existing activity without too much additional effort (i.e. doesn't demand a complete rethink of what is currently done, although it may well voluntarily stimulate that). / + / This will be especially true of version 2 with its form-building capacity.
Will require development and support at first. / 0/- / Possible, but the lack of infrastructure means this can only be realised through significant additional effort.
Where relevant, a system that can contribute towards the PDP requirements of specific professional bodies. / ?/+ / CS Physiotherapists already adopted pP as their CPD e-P. pP actively developing this aspect of customer base. Form-building will allow relatively easy mapping to professional body skills/competency frameworks within pP. / ?/- / Not really suitable for this purpose.
A system that is flexible enough to cope with the significant variation in PDP activities that will be found across subjects. / + / pP user conferences have shown a wide subject/discipline spread. / 0/? / Has technical capacity, but only with significant investment in developing materials & support.
Attractive, accessible and easy to use. / +/0 / Overall balance of staff comment in the pilot has been positive, although some negative aspects have been highlighted. (See statement on accessibility under student criteria.) / - / Overall balance of staff comment is negative.
Stimulates new ways of thinking and working / + / As staff consider how to get the most out of pP capacity new learning and teaching environments are being developed. / 0 / When used alongside blog & wiki functions (not actually part of e-P) has sparked innovation, but has also acted as a barrier.
Doesn't have a large time cost in terms of training, or in rewriting existing materials to fit the system. / 0 / Will have a cost as it will need to be supported and there will need to be initial training and development. Has the capacity to be relatively self-sustaining though. / _ / Will have larger costs in terms of development of materials, staff development, student development and central support.
3. Student needs
Helps students become better learners, and improve academic performance (get a better degree). / ? (+) / Difficult to provide explicit evidence for this so far, but student comments reveal that pP is helping to develop them as reflective learners, which should be a step toward better academic performance. In one pilot module (15 students) those compiling and submitting a portfolio through pP scored on average 10% higher than those using paper format.
pP goes beyond e-portfolio specification and can be considered an individual learning system. / ?
(-) / Again, difficult to provide explicit evidence so far, but there are suggestions that Vle encourages a more instrumental approach to portfolio building.
Helps to access other opportunities (further study, jobs etc.) / ?
(+) / Anecdotal evidence of pP being used to prepare and submit job applications. / ?
Overall, seems helpful and relevant to personal, academic and professional development. / ?
(+) / Balance of student experience seems positive. / ?
(-) / Balance of student experience seems negative.
Attractive, accessible and easy to use / +/0 / Balance of student comment relating to this is positive, although some features cause annoyance to some. Version 2 provides a greatly enhanced capacity for personalisation which would go a long way to address many of the negative comments.
"It is very useful and attractive"
"Visually it looks excellent." (Student comments)
pebblePAD are clearly aware of accessibility issues and are continually developing the package in response to this - for example version 2 (available July 2007) includes an inbuilt screen reader, and enhanced user personalisation options.
Proactively working with JISC Techdis / 0/- / Balance of student comment is toward the negative, particularly in relation to usability.
An e-portfolio that is student-owned, and that can be taken with them when they leave. / + / Very much student-owned.
Flexible export options, including ability to link directly from web or e-mail to e-portfolio.
People can subscribe as individuals, therefore can be lifelong pebblePADers. / 0/- / Can be zipped and exported, but exists as one-off presentational portfolio, cannot be developed further.
Time spent on e-portfolio activity is an investment rather than a cost / ? / Would require longitudinal study beyond remit of this evaluation. / ? / Would require longitudinal study beyond remit of this evaluation.

Page 1 of 7