Automated Assistance and Implicit Feedback

Investigating Automated Assistance and Implicit Feedback for Searching Systems

BernardJ.Jansen
School of Information Sciences and Technology
The Pennsylvania State University
329F ISTBuilding, University ParkPA16802
Email:

Michael D. McNeese

School of Information Sciences and Technology
The Pennsylvania State University
307IST Building, University ParkPA16802
Email:

ABSTRACT

Information retrieval systems offering personalized automated assistance have the potential to improve the searching process. There has been much work in this area for several years on a variety of systems. However, there has been little empirical evaluation of automated assistance to determine if it is of real benefit for searchers. We report the results of empirical evaluation investigate how searchers use implicit feedback and automated assistance during the searching process. Results from the empirical evaluation indicate that searchers typically use multiple implicit feedback actions, usually bookmark – copy. The most commonly utilized automated assistance was for query refinement, notable the use of the thesaurus. We discuss the implications for Web systems and future research.

1.Introduction

There has been considerable research into automated assistance in order to address some of the issues users have when interacting with information retrieval (IR) systems (Jansen, Spink, Bateman, and Saracevic, 1998; Yee, 1991). The need for automated assistance is especially acute with Web searching systems, as research shows that users of Web search engines have difficulty successfully implementing query syntax (Jansen, Spink, Bateman, and Saracevic, 1998), and the performance of major Web search engines (i.e., number of relevance documents retrieved within the first ten) is approximately 50% (Eastman and Jansen, 2003; Jansen and Spink, 2003).

Automated assistance systems usually attempt to assist the user during the search process by either executing search tactics for or offering assistance to the user in order to help locate relevant information. We define automated assistance as any expressions, actions or responses by an IR system with the aim of improving the information searching experience for the user as measured by some external metric. These external metrics are usually relevance related ones, such as precision (Korfhage, 1997).

However, there has been little empirical evaluation of automated assistance. Therefore, it is not clear whether or if automated assistance is beneficial to users during the search process. Is automated assistance helpful? If so, what type of assistance? Is it helpful for certain types of searches? When in the search process do searchers desire assistance? The research results presented in this article address a portion of these issues. We examine whether automated assistance is beneficial to users during the search process.

We begin with a review of literature concerning Web and IR systems offering automated assistance. We then provide a short description of the automated assistance application we developed and utilized in the user study. Next, we discuss the empirical test we conducted to evaluate the effect of automated assistance on system performance. We present the results of our experiment and the implications for Web IR system design and then discuss directions for future research.

2.Literature Review

Many searchers have difficulty effectively utilizing IR systems. These issues occur across the spectrum of IR systems, including online public access catalogs (Peters, 1993) and Web systems (Jansen, Spink, and Saracevic, 2000). For Web systems, issues include lack of query syntax (e.g., AND, MUST APPEAR, etc.), improper use of query syntax, retrieving too many results, retrieving zero results (Silverstein, Henzinger, Marais, and Moricz, 1999; Yee, 1991), among many others.

Although there has been considerable research and development on advanced searching features in Web search engines, users generally do not use these features (Spink, Jansen, Wolfram, and Saracevic, 2002) and have problems with them when they do (Jansen, Spink, and Saracevic, 1998). In order to assist the user with the searching issues and to better utilize advanced searching methods, there have been efforts to develop IR systems that automate various aspects of the searching process. We refer collectively to these as automated assistance systems.

Meadow and fellow researchers (1982a; 1982b) present one of the first design and analysis of a system offering contextual help. Chen and Dhar (1991) developed a system for key word selection and thesaurus browsing. Using the agent paradigm, researchers have explored intelligent IR systems for Web, including Alexa (1999) and Letizia (1995) to aid in the browsing process. ResearchIndex (Lawrence, Giles, and Bollacker, 1999) utilizes an agent paradigm to recommend articles based on a user profile.

From this brief literature review, it is apparent that there has been considerable work into developing automated assistance IR systems. There has been much less research into evaluating: how do searchers utilize these systems during the searching process. We conducted a user study utilizing an automated assistance application that we developed in order to investigate these questions.

3.Research Evaluation

We are interested in examining how users interact with automated assistance with the aim of improving system performance during a session. A session is one episode of a searcher using an IR system during which a series of interactions occur between the searcher and system. In this research, we specifically focus on implicit feedback actions and use of automated assistance.

We next provide a short description of the automated assistance techniques we employed, and the component we developed.

4.System Development

We designed and developed a client-side software component to integrate a suite of automated assistance features with a range of existing Web-based IR systems. Our system development goal was that the system would rely on implicit feedback, gleaning information solely from normal user – system interactions during the search process. The automated assistance component uses these interactions to determine what assistance to provide.

4.1System Design

The system builds a model of user – system interactions using action - object pairs (a, o) (Jansen, 2003). A single action - object (a, o) pair captures an instance of a user – system interaction. An a is some user initiated interaction with the system. An o is the receiver of that action.

A series of (a, o) pairs models a searcher’s chain of interactions during the session. The system can use these (a, o) pairs to determine the user’s information need and provide appropriate assistance by associating certain actions with specific types of assistance.

Using (a, o) pairs has several advantages compared to other methods of gathering information from a user during a session. The query is usually the only source of information from the user during traditional IR system interaction. Other techniques (e.g., answering questions, completing profiles, judging relevance judgments) require the user to take additional actions beyond those typical of user interactions during an online search.

Using (a, o) pairs, the user’s query is not the sole representation of the information need. The system gathers additional information from other user actions, such as bookmarking, printing, emailing, without requiring additional user actions. As such, the (a, o) pair methodology is ideally suited for the implicit feedback that one expects on the Web and other client – server architectures.

4.2System Overview

The system currently monitors the searcher’s interactions with the system, tracking actions of bookmark, copy, print, save, submit, and viewresults. Previous research has identified these actions as implicit indications of possible document relevance (Oard and Kim, 2001). There are currently three objects that the system recognizes, which are documents, passagesfromdocuments, and queries.

The system monitors the user for one of the six actions, via a browser wrapper. When the system detects a valid action (i.e., (a, o) pair), it records the action and the specific object receiving the action. For example, if a searcher was viewing this_Web_Page and saved it, the system would record this as (savethis_Web_Page). The system then offers appropriate search assistance to the user based on the particular action and the system’s analysis of the object. The more (a, o) pairs the system records, the more complex the model of the information need.

4.3Automated Assistance Offered

We currently focus on five user – system interaction issues and corresponding system assistance, which are:

• Managing Results: Searchers have trouble managing the number of results. Using the (submit query) pair and the number of results, the automated assistance application provides suggestions to improve the query in order to either increase or decrease the number of results. If the number of results is more than thirty, the application provides suggestions to restrict the query. If the number of results is less than ten, the system provides advice on ways to broaden the query. We chose thirty and ten results as the boundary conditions based on research studies showing that approximately 80% of Web searchers never view more than twenty results (Jansen, Spink, and Saracevic, 2000). However, one can adjust the result thresholds to any targeted user population.
• Query Refinement: In general, most Web searchers do not refine their query, even though there may be other terms that relate directly to their information need. With a (submit query) pair and a thesaurus, the system analyzes each query term and suggests synonyms of the query terms. The system uses the Microsoft Office thesaurus, but the application can utilize any online thesaurus via an application program interface (API).
• Query Reformulation: Some search engines, such as AltaVista (Anick, 2003) offer query reformulation based on similar queries from previous users. We incorporated this feature into our automated assistance system. With a (submit query) pair, the system accesses a database of all previous queries and locates queries within the database containing similar terms. The system displays the top three similar queries based on number of previous submissions.
• Relevance Feedback: Relevance feedback has been shown to be an effective search tool (Harman, 1992); however, Web searchers seldom utilize it when offered. In the studies on the use of relevance feedback on the Web (Jansen, Spink, and Saracevic, 1999), Web searchers utilized relevance feedback less than 10% of the time. In this study, we automate the process using term relevance feedback. When the (a, o) pairs of (bookmark document), (print document), (save document), or (copy passage) occur, the system implements a version of relevance feedback using terms from the document or passage object. The system provides suggested terms from the document that the user may want to implement in a follow-on query.
• Spelling: Searchers routinely misspell terms in queries, which will usually drastically reduce the number of results retrieved. A (submit query) pair alerts the automated assistance application to check for spelling errors. The system separates the query into terms, checking each term using an online dictionary. The system’s online dictionary is Microsoft Office Dictionary, although it can access any online dictionary via the appropriate application program interface.

4.4System Overview

The automated assistance system has five major modules, which are:

The Query Terms module uses the (submit query) pairs during a session. For each (submit query) pair, the module parses each query into separate terms, removing query operators such as the MUST APPEAR, MUST NOT APPEAR and PHRASE operators. The module then accesses the Microsoft Office dictionary and thesaurus, sending each term to the process. If there are possible misspellings, the module records the suggested corrections.

The Relevance Feedback module uses (bookmark document), (print document), (save document), or (copy passage) pairs. When one of these pairs occurs, the module removes all stop words from the object using a standard stop word list (Fox, 1990) and all terms from previous queries within this session. The system then randomly selects terms remaining from the results listing abstract that was displayed or from the passage of copied text, depending on the (a, o) pair.

The Reformulation module uses the (submit query) pair to provide suggested queries based on submitted queries of previous users. When a (submit query) pair occurs, the modules queries a database of all previous queries that contain all the query terms, attempting to find at least three queries to present to the user. If the database contains three queries that contain all the terms, the module selects these queries, unless one is identical to the current query. If the database contains more than three, the module selects the top three based on frequency of occurrence. If the database contains less than three, the module queries the database for queries that contain at least one of the terms, beginning with the first term in the current query. The module repeats the process until it has at least three queries to present to the searcher. One can alter the number of queries the module returns.

The Refinement module uses a (submit query) pair and the number of results retrieved to suggest alternate queries to either tighten or loose the retrieval function. If the system retrieves more than thirty results, the module first checks the query for the AND, MUST APPEAR, or PHRASE operators. If the module detects no operators, it reformulates the queries using the existing terms and the appropriate AND, MUST APPEAR, or PHRASE operators. If the module detects AND or MUST APPEAR operators in the query, the module refines the query with the PHRASE operator. If the module detects PHRASE operators in the query, the module does no refinement to tighten the query. If the system retrieves less than twenty results, the module performs a similar process to broaden the query by removing of AND, MUST APPEAR and PHRASE and replacement them with the OR operator.

The Tracking module monitors user interactions with the browser, including all interactions with the browser tool bars, along with the object of the interaction. The Tracking module then formulates the (a o) pair, passing the pair to the appropriate module.

The Assistance module receives the automated assistance from the Query Terms, Relevance Feedback, Reformulation and Refinement modules, presenting the automated assistance to the searcher via an ASP script, which the browser loads with the Web document. For the spelling assistance, each term is presented, followed by a list of possible correct spellings. The same format is followed for synonyms. Queries with spelling corrections, similar queries, relevance feedback terms, and re-structured queries are presented as clicked text (i.e., the searcher can click on these to generate a new search). Figure 1 presents a complete system diagram.

Figure 1: Automated Assistance Modules and Information Flow with Interface

5.User Study

In the following sections, we outline our empirical evaluation.

5.1Study Design

The backend IR system utilized for the empirical study was Microsoft Internet Information Service (IIS). The IIS system is running on an IBM-compatible platform using the Windows XP operating system and Microsoft Internet Explorer as the system interface. For the automated assistance system, we integrated the automated assistance application via a wrapper to the Internet Explorer browser.

The subjects for the evaluation were 40 college students (35 males and 5 females) attending a major U.S. university. All were familiar with the use of Web search engines. We gave them no additional training. We did administer a pre-evaluation survey to collect a variety of demographic and other information. Table 1 presents the pertinent demographic information.

Table 1. Demographic of Subjects
Age / Mean / St Dev / Mode / Total
21.4 / 1.96 / 21
Gender / Male / Female
35 / 5 / 40
Experience with Search Engines
< 1 Year / 1 - 3 Years / 3 -5 Years / > 5 Years
0 / 2 / 12 / 26 / 40
Self-Reported Skill Rating
1 (Novice) / 2 / 3 / 4 / 5
(Expert)
0 / 2 / 7 / 23 / 8 / 40
Google / Yahoo! / Alta Vista / Others
38 / 7 / 2 / 4 / 51
Search Engine Use (Daily) / MeanRange =
4.6 – 5.5 / St DevRange =
3.6 – 4.1
Search Engine Use (Weekly) / MeanRange =
30.5 – 33.1 / St DevRange =
27.2 – 28.6

The average age of the subjects was 21 years. Of the subjects, twenty-six reported more than five years experience using Web search engines. The subjects self-rated their searching skills. Of the forty subjects, thirty-one rated themselves as expert or near expert. None rated themselves as novice. We also asked which search engines they used frequently. The subjects could list more than one. The most frequently reported search engine was Google. There were four search engines listed once (AOL, MSN, Ask.com, Meta-crawler). Reported frequency of search engine usage per day (subjects could report a range) averaged 4.6 to 5.5 occurrences. Weekly search engine usage averaged 30.5 to 33.1 occurrences.

We utilized the Text REtrieval Conference (TREC) volumes number 4 and 5 as the document collection for the evaluation. The document collection is more than 2GB in size, containing approximately 550,000 documents.

Each TREC collection comes with a set of topics for which there are relevant documents in the collection. We loaded a list of topics in a spreadsheet and coded a script to select six topics at random. The topics selected, and the ones that we utilized are:

• Number 304: Endangered Species (Mammals)
• Number 311: Industrial Espionage
• Number 323: Literary/Journalistic Plagiarism
• Number 335: Adoptive Biological Parents
• Number343: Police Deaths
• Number350: Health and Computer Terminals

There were 904 TREC-identified relevant documents for the six topics, which is 0.2% of the collection.