Making Recommender Systems Work for Organizations
Natalie S. Glance / Damián Arregui / Manfred DardenneXerox Research Centre Europe
6 chemin de Maupertuis
38240 Meylan, France
+33 476 615 022/99
{glance, arregui, dardenne}@xrce.xerox.com
ABSTRACT
For the past two years, we have been investigating the use of recommender systems as a technology in support of knowledge sharing in organizations. Recommender systems are a way of extending the natural process of recommendation by word-of-mouth to networked groups of people. They are able to provide personalized recommendations that take into account similarities between people based on their user profiles. The community around recommender systems that has emerged in the past five or so years has focused on methods for constructing and learning user profiles, the exploration and testing of various recommendation algorithms, and the design of user interfaces, with applications primarily in the domains of electronic commerce and leisure/entertainment.
Thus far, we have focused our research in two areas: adapting recommendation algorithms and user profile construction methods to take into account prior information regarding the existing organizational social network; and addressing the incentive issues surrounding the use of a recommender system for knowledge sharing in an organization. In this paper, we describe principally the incentive issues that we have identified and how we have attempted to alleviate them. We also report and analyze results from an internal year-long trial of our recommender tool, the Knowledge Pump.
1INTRODUCTION
It is not enough to know that the 100 items returned by your search contain both of the words “knowledge” and “discovery.” It’s not even enough to know which ones are more or less relevant to the domain of “knowledge discovery.” What people really need to know is which papers are the ground-breaking ones, which contain important new results, which are the provocative ones, as well as who are the experts. Current search techniques based on indexing, retrieval and relevance feedback can’t give this kind of information, but recommender systems promise the ability to augment relevance judgements with these kinds of quality judgements.
Recommender systems are intelligent agents that provide a way to filter items by personalized measures of quality: users receive only those recommendations from their colleagues that they are most likely to perceive to be of high quality. Conversely, the person doing the recommending is ensured that only those people likely to be interested in what s/he has to share will receive the item. Actually, since measuring quality is highly subjective, it’s more accurate to say that recommender systems work by filtering by taste. Recommender systems learn their users’ tastes and recommend items to users by first matching users to each other. How recommender systems learn users’ tastes and how they perform the matching is to a large extent what differentiates current systems (as well as their domains and genres of application).
A natural evolution of recommender systems has been to also provide matchmaking services by bringing together people with similar tastes and interests. In this sense, many recommender systems have morphed into “communityware,” a new term for software which supports communities. However, most of these systems are oriented towards the needs and requirements of Internet users engaging primarily in leisure activities, and not towards the workplace and work activities. In the workplace, these matchmaking services can become ways to extend one’s professional network and to locate experts in a domain.
In this paper, we report on our work which focuses on re-orienting recommender systems towards the workplace. In work settings, the primary foci shift from sharing recommendations to sharing knowledge and from community-building to community support. Moving recommender systems from the Internet to the workplace also means turning “leisure-ware” into groupware, creating both new challenges and new opportunities.
When we think about putting recommender systems to work for organizations, we need to re-think a number of assumptions underlying Internet-based recommender systems: for example, the potential size of the user base (huge and still rising exponentially) and the raison d'être (recommendations oriented towards leisure and personal interests). In contrast, in work organizations, the potential user base will be relatively fixed and of more modest size. Also, the main impetus, or at least, a major impetus for usage, will be sharing information pertinent to workplace objectives, competencies, and interests.
On the Internet, a useful service can hope for a user base of thousands, if not millions, of users. Recommender systems that depend on statistical algorithms thrive on extensive usage. In contrast, recommender systems in primarily closed organizations of more modest size must implement new ways to make recommendations for a smaller user base. Many Internet-based systems using collaborative filtering techniques have already discovered the benefit of filtering first by content and then by taste. This two-tiered approach is likely to be even more important in work environments where differences in preferences cannot be accounted for simply by taking into account the opinions of yet more people rating yet more items.
However, filtering first by content and then by taste means calculating correlations based on far fewer rating pairs, a problem which is likely to be far worse in an organizational setting where interest domains may be populated by mere handfuls of people. These problems highlight the importance of going beyond automated collaborative filtering for making recommendation predictions. To make good predictions, organization-based recommender systems will have to take into account usage data across many systems, potentially including search engines, document management systems, possibly e-mail, and will have to finesse methods for combining the many kinds of evaluations into one form.
Recommender systems as groupware instead of leisure-ware also suffer even more from the classical critical mass problem. Within a work organization, a recommender system can be a valuable tool only if most (as opposed to many) people are using it. Like the telephone or the fax or e-mail, this kind of technology will be used at work only if most others use it as well. People in the workplace are likely to be more discouraged from sharing knowledge via a recommender system if only a fraction of their colleagues are signed up to receive recommendations, because then they will have to supplement their effort by using other communication media, such as e-mail. The critical mass problem exists in Internet-based recommender systems, as well, but arguably to a lesser extent, as the Internet can make up in numbers what it lacks in density.
Thus, it is vital in a work setting to investigate the incentive issues that arise in achieving acceptance and usage of the technology. These incentives include ease-of-use, integration with users’ software environment, and perhaps most importantly, immediate and sustained perceived benefit. This last issue is particularly difficult: how to overcome the well-known cold-start problem of recommender systems to provide useful and high-quality recommendations immediately?
In addition, we can expect the dynamics of group participation and usage to be qualitatively different in a work situation where people know each other by name and reputation. How can work-based recommender systems leverage the notions of trust, reputation and reciprocity to best serve working communities and help them serve themselves? Here, we can expect to learn both from other peer-based models for collaborative work and evaluation and from market mechanisms for privatizing public goods (evaluations being a public good). However, the true test will be in the usage: in this sense, work-centric recommender systems are just like their Internet cousins; they serve as platforms for novel social experiments and institutions.
Finally, while Internet-based recommender systems focus on creating communities by bringing people together, Intranet-based recommender systems should focus on supporting communities that already exist. The power of recommender systems to help people find each other is multiplied many times over in an organization: locating experts; reducing re-work by bringing together people working in the same area in different geographic locations; and identifying competencies, both established ones and emerging ones.
To study both the utility and usability of recommender systems for the workplace, we have implemented and deployed a research prototype we call the Knowledge Pump. Our objectives in designing the Pump were two-fold. The first was to help automate the process of sharing recommendations. The second was to support workplace communities of interest. With these ends in mind, we designed ways to provide user feedback on collective behavior and implemented market mechanisms intended to regulate the flow of recommendations. Below, we first give an overview of the current prototype and compare it with the current breed of recommender systems. Then, we discuss what we mean by community in a networked organization, since the notion of community is central to our work. Next, we take a closer look at issues of utility and usability of recommender systems in the workplace that we have encountered in the development and usage of the Knowledge Pump. Finally, we present usage analysis results and summarize user feedback from the first year of the Pump’s operation within our research centre. In the discussion, we describe the next steps we are taking, both in the directions of commercialization and on-going research.
2KNOWLEDGE PUMP: OVERVIEW AND RELATED SYSTEMS
The first prototype of the Knowledge Pump has been implemented in Java as a client-server system. The client runs as an applet off of a WWW browser. By riding on top of the Web and coding in Java we can get most of the way towards fulfilling one requirement for acceptance of any groupware tool with only one version of the code: cross-platform availability.
When a user first becomes a member of the Pump, s/he chooses a set of advisors from the list of current members and a set of communities, or domains of interest, from the list of communities suggested. Users are suggested to think of advisors as people whose opinions they most value and trust. What we mean by community and how the list of communities is created (and how it evolves) is explained in Section 2.1.
Figure 1: An abridged example of “What’s Recommended?” by the Knowledge Pump today.
The Pump provides its members with two principal functionalities on top of a WWW browser. These are (1) a shared bookmarking system and (2) a daily-updated list of recommended items (for example, articles, case studies, customer solutions, and, of course, WWW pages) classified by community of interest. The shared bookmarking system allows the user to bookmark items and save them as either private (for the user’s eyes only) or public. A public bookmark can be thought of as a review or a recommendation that the user shares with other members of the Pump. The Pump also helps users search and browse the set of shared bookmarks. In addition, there is a “What’s new?” button that generates a page of all the new submissions and reviews of previously submitted items for the past week which spans across all communities. The shared bookmarking facility and the search function as well as additional functionality will be discussed further in later sections. The software architecture and implementation of the system are described in [4].
Each day, the Pump sends users a new set of shared bookmarks likely to interest them. An example of “What’s Recommended?” by the Knowledge Pump today is shown in Figure 1. For each community to which the user belongs, the Pump delivers every day (by automatically re-loading the WWW page) a set of recommendations. Each recommendation is preceded by the Pump’s prediction of the user’s interest, shown as a number of stars. The recommendation itself is a hyperlink pointer to the item recommended. The next line after the item shows the list of reviewers who have rated and commented upon the item. Many of these names are likely to be known to the person receiving the recommendations from day-to-day work interactions. Finally, there is also a hyperlink to the detailed review page for the item: each reviewer’s rating, comments and the time of evaluation. The review page functions as a kind of “conversation” around the recommended item, and will be described further in Section 4.1.
The Knowledge Pump uses what we call community-centered collaborative filtering [4] to predict a user’s level of interest for unread items in each of the users’ domains of interest. This mechanism combines elements of social and content-based filtering. With respect to the latter, we currently rely on recommenders to classify items into a commonly agreed upon classification scheme. (The process for defining the classification scheme is discussed in the next section.) A different, or perhaps complementary, approach would be to use statistical techniques for content filtering as well, as do Balabanovic and Shoam [2] in Fab, a Web-page recommender system.
1
The second layer of social filtering – matching items to people by first matching people to each other – lies on top of the initial classification by community.[1] Community-centered collaborative filtering extends the technique of automated collaborative filtering presented by, for example, Shardanand and Maes [20] and Resnick et.al.[17] in the context of movie and music recommendations and Netnews recommendations respectively. These systems automate collaborative filtering using statistical algorithms to make recommendations based on correlations between personal preferences. Recommendations usually consist of numerical ratings, but could be also deduced from implicit ratings, such as time spent reading a text [12].
We’ve modified the standard approach in order to address the cold-start problem faced by recommender systems. The cold-start problem refers to the poor performance of automated collaborative filters when usage data is sparse, which is the case when the system first starts up, or, possibly, when a new community or domain of interest is created. Unfortunately, poor performance is likely to discourage the usage that would overcome the lack of usage data. The cold-start problem is likely to be even more significant in the workplace, where the goodwill of users is more rapidly consumed.
In extending automated collaborative filtering, we have tried to ensure that recommendations will be of immediate value for new members. One technique we employ is to bootstrap the collaborative filter by a partial view of the social network constructed from user’s lists of advisors. When the usage data is too sparse to make automated collaborative filtering feasible, the Pump instead relies first on a member’s advisors to make predictions. Over time as more usage data is collected, the automated collaborative filter kicks in to contribute to the Pump’s prediction of the user’s interest in an item. Of course, bootstrapping the Pump in this way only works in a setting where many members of the social network already know each other.
A number of other approaches to bootstrapping recommender systems have also been taken, which could be usefully combined with ours. For example, Shahabi et.al.[19] determine similarities between users from the WWW navigation paths using a server-side agent. Kautz et.al.[10] have developed methods for reconstructing social networks from the text of Web pages and hyperlinks between them. Nichols et.al.[14] describe the different kinds of implicit recommendations and associations that can be mined from usage data in the context of digital libraries. Pirolli and his colleagues [15, 16] examine how patterns of usage at a server site can be used to inform the subsequent Web navigations of other users and deduce properties of documents (faddish vs. sustaining interest, for example). Foner [3] has designed and implemented a multi-agent system for deducing common interests among people using matchmaking agents.
It’s a natural move to juxtapose recommendation services and matchmaking services; a number of recommender systems have extended their services in this direction. A major reason why Firefly[2] has become so popular is because it provides chat facilities alongside its movie and video recommendation service: the recommendations provide the seed for new relationships. Similarly, another movie recommendation system described by Hill et.al.[9] lets users know who has similar tastes to them and offers joint recommendations for more than one person. In a workplace setting, the power of recommendations to link people together has the potential to be extremely valuable. We are exploring a number of techniques for enhancing users’ sense of community, which we will discuss in Section 4. In particular, our use of market mechanisms for regulating the flow of recommendations, the several kinds of user feedback on collective behavior, and the support for community awareness are, to our knowledge, novel extensions of recommender systems.
2.1What does ‘community’ mean?
If recommender system-like technology is to succeed in the workplace, it will be for some of the same reasons that e-mail and Internet newsgroups have proved to be so useful: because it provides a new medium for communication and information sharing; and because it provides a new way to support communities.