Personal Repositories Online: Wiki Environments

PROWE

Personal Repositories Online: Wiki Environments

Open University and University of Leicester

(

EVALUATION REPORT

August 2007

Chris Pegler
Institute of Educational Technology
Open University

Document History

Version / Date / Comments
1.0 / 2007-08-29 / 1st draft CP
1.1 / 2007-11-30 / 2nd draft CP/AG (repackaging)

PROWE

Personal Repositories Online: Wiki Environments

EVALUATION REPORT

August 2007

Table of Contents

Acknowledgements

Executive summary

1.0The Project

2.0The PROWE central research question

3.0Approach to evaluation

3.1Use of tracking information

4.0Brief description of institutional contexts

5.0Project participants

5.1Open University: Project Team

5.2University of Leicester: Project Team

5.3Open University tutor participants

5.4University of Leicester teaching associates and participants

6.0Surveys, establishing the start point

6.1How do tutors organise their resources currently?

6.2Do tutors currently reuse materials?

6.3Prior knowledge of wikis and blogs

6.4Community and professional development

6.5Community policy and ground rules

7.0Focus groups

7.1The Open University meetings

7.2The University of Leicester focus group meeting

8.0Interviews with participants

9.0Participant views on the technologies

9.1OU PROWE blog usability

9.2Uploading into the OU PROWE blog

9.3Help in using the OU PROWE blog

9.4Navigating the OU PROWE blog

9.5RSS feeds and finding content

9.6Offline fallbacks?

9.7UoL PROWE blog/wiki (Plone v 2.1.3) and the University of Leicester users

9.8Preparing content offline

10.0Theme one: Informal repository use

10.1When it’s ‘informal’ will there be high quality?

10.2Provenance and personal profiles

10.3Informal tagging systems – the attraction of taxonomies

10.4Non-formal tagging: Folksonomies

11.0Theme two: Personal repository use

11.1Different types of publishing space

11.2Versions and back-ups

11.3Personal profiles

12.0Theme three: Wikis and blogs as repositories

12.1‘Wiki-published’

12.2Affordances of blogs and wikis

13.0Theme four: CPD and sharing resources

13.1How sharing currently happens

13.2Discipline differences

13.3Researchers use of personal and informal repositories

14.0Sustainability – what happens next?

14.1The Open University

14.2The University of Leicester

14.3What level of moderation is required?

References

APPENDIX: Evaluation plan

APPENDIX: Survey questions

Open University survey questions

University of Leicester survey questions

APPENDIX: Focus Group questions

APPENDIX: Interview Questions

Open University version

University of Leicester version

APPENDIX Accessibility assessment

Acknowledgements

I would like to thank all members of the PROWE team across theOpen University and University of Leicesteras well as the tutors who took part in interviews, focus group meetings and the surveys. I owe a particular debt to the two OU Annes (Anne Gambles and Anne Hewling) for their patience and support at key points, Susan Eales for her quiet encouragement and Gilly Salmon for her great ideas and sustained interest. Roger did all the ‘leg work’ for me at Leicester and he – with Richard Mobbs and Tony Churchill – are key to my appreciation ofthe very interesting work that UoL undertakes.

JISC should also be thanked for funding the project and the various other very interesting projects within the repository cluster. These provided relevant work for this project to reflect upon and learn from. Particular influences have been the CD-LOR, RepoMMan, TrustDR, and Rights and Rewards projects.

Executive summary

This report evaluates the experiences acrosstwo demonstrator communities of practice operating in the course of the PROWE project. The demonstrators wereusingopen source systems, Elgg (version 0.4) and PmWiki (version 2.1.11) at the Open University, and Plone (version 2.1.3) at the University of Leicester.

The evaluations concentrate on and are restricted to typical and actual use of the demonstrators within the PROWE project. The comments and observations within this report are aimed at the PROWE experience and contextand not an analysis of the underlying systems, or comments on their applicabilityin other contexts.

The PROWE project looked at the use and potential of personal and informal repositories used by teaching staff at The Open University (OU) and The University of Leicester (UoL) from mid-2005 to mid-2007. Although originally envisaged as a project where the same system and types of users would be studied across both sites, there have been separate systems used and considerable variety across the sites and within UoL in terms of the types of personnel using and controlling the systems and their motivation and uses.

The systems used were Elgg (version 0.4) and PmWiki (version 2.1.11) (OU) and Plone (version2.1.3), Simple Blog (version 1.2.1)and Zwiki (version 0.58) (UoL) (a good technical description is given in Mobbs, 2007). The two institutional virtual learning environments (VLEs) Blackboard (UoL) and Moodle (OU) were also drawn on by users, as was the FirstClass conferencing system which currently offers OU tutors a route to sharing resources online within and across course communities.

The evaluation approach for the project was modified to take account of this variability (see Approach to evaluation). The most notable findings were:

The contractual arrangement, or what is understood to be the contractual arrangement, for tutors is likely to affect whether and how they share content (e.g. contributors or consumers). Some UoL part-time lecturers may be devising the curriculum for modules which they, and only they, teach. There is little incentive for them to expose this material to others. OU lecturers are very likely to be one of several tutors teaching the same subject. They will however have had little say in the design of most of the course material. They may see themselves as not having the right to share that material more broadly – for example with other institutions where they teach similar courses. These are two of the many IPR concerns that sharing raises in these contexts.
Several users (both sites) reported that they were now making use of online file storage, such as Google Docs or storage available though free email accounts hosted outside the institution (e.g. GMail). This is a recent development and may explain the relative lack of interest in using either of the two PROWE systems for storage of ‘private’ documents.
Tutors and other academic and research staff interviewed were unlikely to, and in some cases uneasy with, the idea of storing private and public content within the same repository. They were more likely to keep the two separate. For instance keeping private content (not for wider sharing or not yet ready for public viewing) on USB pen drives, on portable hard disks, on a laptop, or online on a completely separate system. This content would be shared by email and so was kept distinct from repository content. This mirrors findings on sharing work-in-progress by email already reported in the CD-LOR Report on Personal Resource Management Strategies.
There was a tendency for tutors at either site to automatically see the PROWE systems as places to ‘publish’. The idea of putting material onto either system that only they could access, or see, was almost counterintuitive. So although this facility was there it was hardly used at all and then sometimes on an experimental, or accidental basis.
The interest in sharing with small groups of friends or project teams, either within or beyond the institution was seen as a significant advantage of personal and informal repositories (Macdonald and Hewling, in press). Having local control over how these groups were constituted and what they could access was considered very important across both sites.
Users at OU and UoL had reservations about the usability, particularly the file upload ease, for their respective systems (Elgg (v0.4) and Plone (v2.1.3)). This may have impacted on the type of use made. In one case a single person within the unit did the uploading of documents for all staff in order to overcome this problem.

While there was considerable interest in blogging and wikis,some users – particularly the UoL users – were more likely to see their usefulness in terms of teaching (i.e. communicating with or as a student), or for research, than for staff development. The spontaneous sharing of ideas and resources with other teachers is still a new and relatively untried idea on both sites although PROWE is credited with having taken this practice forward through a number of initiatives.
There were some concerns that blogging was ‘vanity publishing’ and could be too ‘Café chat’ in style and too variable a quality to be useful.
Having a functional community was felt to be important if credence were to be given to resources and advice offered in blogs and wikis. There is a high risk that if someone were to abuse trust (for example taking and reusing material without attribution or posting inaccurate or inappropriate material), that others would stop posting material and stop using the repository altogether. This is perhaps one reason why some level of moderation was felt to be necessary – not in organising or controlling discussions and approving content, but in having someone who could intervene and advise if necessary. So informal use is still perceived as needing some level of control, even if never used.
There is atension between wishing to be in a community, or sharing a repository, with like-minded and similarly experienced people (e.g. tutors on the same course) and interest in ‘leaky boundaries’ between repositories and communities. Having leaky boundaries would allow unexpected serendipitous connections to be made.
There was a tendency to see the usefulness of repository materials as being directed at new and relatively inexperienced tutors. The participants in this project, particularly those who were most experienced, did not usually see themselves as being the recipients, but rather the providers of shareable material.
There were challenges to the project on both sites on how to create and sustain informal communities. At UoL the repository itself was more formal and there is some indication that the blog and wiki tools integrated into Blackboard will prove more popular with part-time and other teaching staff than the Plone system which is perceived not to have this level of official ‘sanction’ as a teaching platform.The project raised questions about how informal use can be facilitated within either embedded institutional systems such as Plone at UoL (which are aligned with formal institutional activities, e.g. Marketing), or projects set up for the purpose of studying informal use over a set time period and within a small project. Informal use, by definition, cannot be managed too directly.
The group blogging and other informal community sharing activities within the OU PROWE blog were seen as its most appealing and interesting aspect. The juxtaposition of informal online communities and repositories is one which could encourage building relationships which support users in knowing more about the provenance of materials submitted.

Where colleagues were co-located, and/or frequently meet, the emphasis is more towards formal use of any online repository. Where there is no clear community with a shared purpose using the systems then there is little incentive to put time into learning to use them, especially if you are employed on a part-time basis and the use is informal rather than linked to clearer more specific or strategic objectives. This may be particularly the case for those academics who see experimentation with technology as something which they should not be expected to do.

Technically competent tutors did not seem to have a problem with switching between different sorts of repository and some mentioned that to maximise their coverage they might use more than one.

There was a preference by some users for an informal folksonomy approach to tagging content.

There is less emphasis than anticipated on sharing resources across courses and institutions than had originally been envisaged.

The personal repository is necessarily a much broader set of resources than the ‘published’ repository. It contains things that were never intended for publication.

The project also raised questions about how informal use can be facilitated within either embedded institutional systems such as Plone at UoL (which are aligned with formal institutional activities, e.g. Marketing), or projects set up for the purpose of studying informal use over a set time period and within a small project.

1.0The Project

PROWE (Personal Repositories Online: Wiki Environments) was funded as part of the JISC Digital Repositories Programme 2005-7 and ran from June 2005 to August 2007. It was one of two projects funded within this programme to specifically look at personal and/or informal repository use. The other project was the Secure personal institutional and inter-institutional repository environment (SPIRE) project ( based in OxfordUniversity’s Technology Assisted Lifelong Learning (TALL) team. It is worth noting that the original name for PROWE was ‘PROBE’, referring to ‘bliki’ (blog/wiki) rather than ‘wiki’ environments. This acronym was dropped in favour of PROWE, but the commitment to look at blogs and blog-like solutions as well as wikis continued.

The PROWE project set out to look at personal and informal repository use by part-time tutors working at a distance from the institution and within a distance learning framework. This group was identified as one which was likely to have a challenging set of requirements, which could include:

relative lack of time to give to keeping up to date and learning to use new technologies;
poor or uneven access to resources, possibly dependant on location;
more than one employment role (e.g. tutoring different courses under separate contracts) between which they might need to transfer materials and resources;
more than one employer and access therefore to more than one set of institutional resources
complex – possibly individual – staff development requirements
poorer opportunities to participate in informal sharing of content than full-time and/or campus based teachers

The project’s broad aim and its ‘central research question’ (taken from the Project Plan of Sept 2005) were to ‘investigate the use of informal repositories within wiki and blogs by part-time tutors on distance education programmes in the partner institutions for sharing and storing resources in the context of their own professional development needs’. (See also 2.0 The PROWE centralresearch question).

The original project objectives were identified as:

Identify the most sustainable technology solution to support the establishment and the development of a proactive community of practice for the partners’ distance education tutors.[1]
Undertake a user needs analysis and feasible metadata model to underpin the development of an electronic collaborative working environment for the part-time tutor community which is both cost effective and non resource intensive.[2]
Evaluate existing collaborative communication tools leading to the recommendation of a clearly preferred, Web Service based and fully accessible communication tool to support proactive online collaboration for other communities of practice.[3]
Evaluate the use of informal and personal repositories by staff across both institutions using data capture methods to record the use and explanations of the users for comparative purposes and deeper analysis of use in practice[4].
Ensure the full interoperability of the collaborative communication tool with other toolkit resources, making full use of RSS feeds to provide additional ‘continuous professional development’ updates[5].
Deliver a shared document repository of a range of case studies in teaching and learning.[6]

For reasons detailed elsewhere in this report this plan was not fully realised. The project became broader in scope and consequently less targeted than the planned approach suggests. This was mainly a consequence of the differences in context across which PROWE necessarily operated. It is also perhaps an indication that for this group of staff in particular, this type of research project – where diverse participants use technologies which are not yet stable – presents particular challenges, particularly when those uses are a blend of professional and personal/informal.

The institutions involved – The Open University and the University of Leicester – are both based in the Midlands. They were chosen not for their physical proximity, but because they together represent the two largest distance learning providers in the UK university sector. The OU has c.180,000distance learning students and c.7000part-time teaching staff and the University of Leicester has 6,000 distance learning students and c. 350 part-time teaching staff. Although distance learning is not such an important form of educational delivery across other UK HEIs, there is an argument that there has been a convergence of distance learning and campus based teaching with the onset of blended e-learning. Many part-time students who receive instruction through blended e-learning are to some extent distance learners. Also many part-time lecturers will – like their distance teaching equivalents – be only infrequent visitors to the intuition’s main campus, perhaps attending only in evenings, or teaching mainly at satellite campuses. They may work part of their time at other institutions or in non-teaching jobs. There is a clear staff development issue for such distributed staff, particularly in terms of how staff development can be personalised for them and provide an equivalent to informal networks for passing on experience and sharing resources.

Because of the way that the project was introduced and maintainedat the two different institutions(see Context and Descriptions and Technology at OU and UoL) it has been easier to produce a picture of institution-wide practice at the OU than it has been at UoL. In part this was anticipated at the start when we realised that at UoL there were 11+ departments offering distance learning alternatives, eachwith a different approach to supporting its part-time tutors. This range of approaches varies from a part-time tutor who controls the content of a specific module and develops most of the resources with a high degree of autonomy, to one whose role is to mark student scripts and who may have little other contact. Every department in the UoL that offers distance education also offers its part-time tutors different types of teaching contract and different levels of staff support. There is also variation between the arrangements for different types of tutors within some of the larger departments (e.g. the ManagementSchool).Throughout the PROWE project, UoL has engaged in a continuing review of such arrangements. It is, for example, currently engaged in a major project to create a flexible management information system to meet the needs of all distance elearning courses.