Exploratory information searching in the enterprise: A study of user satisfaction and task performance.
Paul H. Cleverley
Department of Information Management, Robert Gordon University, Garthdee Road, Aberdeen AB10 7QB Email:
Simon Burnett
Department of Information Management, Robert Gordon University, Garthdee Road, Aberdeen AB10 7QB Email:
Laura Muir
Department of Information Management, Robert Gordon University, Garthdee Road, Aberdeen AB10 7QB Email:
No prior research has been identified which investigates the causal factors for workplace exploratory search task performance. The impact of user, task and environmental factors on user satisfaction and task performance was investigated through a mixed methods study with 26 experienced information professionals using enterprise search in an oil and gas enterprise. Some participants found 75% of high value items, others found none with an average of 27%. No association was found between self-reported search expertise and task performance, with a tendency for many participants to overestimate their search expertise. Successful searchers may have more accurate mental models of both search systems and the information space. Organizations may not have effective exploratory search task performance feedback loops, a lack of learning. This may be caused by management bias towards technology not capability, a lack of systems thinking. Furthermore, organizations may not ‘know’ they ‘don’t know’ their true level of search expertise, a lack of knowing. A metamodel is presented identifying the causal factors for workplace exploratory search task performance. Semi-structured qualitative interviews with search staff from the Defence, Pharmaceutical and Aerospace sectors indicates the potential transferability of the finding that organizations may not know their search expertise levels.
Introduction
The oil and gas exploration industry is a source of significant scientific and engineering activity. The industry seeks to identify and model subsurface hydrocarbon accumulations, responding to commercial opportunities requiring significant investment decisions in short periods of time (sometimes weeks). This time pressure combined with large amounts of diverse information is likely to create an environment of information overload. This makes it an ideal context in which to analyse challenging search tasks such as exploratory search (Nolan 2008). According to Marchionini (2006), search tasks can include lookup (known item) search (where there is a single correct search result or answer) and exploratory search (to investigate/learn) which may involve searching information resources for unknown quantities of information.
When searching for documents in oil and gas organizations, staff are often reliant on a small amount of textual metadata. This is because metadata is used to influence ranking of search results, may be the only way to locate and request some published digital information (as the contents may be confidential by default) or the item is physical in nature (Sawaryn et al. 2014, Liddell et al. 2003). Many project deliverables and supporting information are often stored in several places, poorly tagged and rarely cite lists of references (Andersen 2012, Quaadgras and Beath 2011, Garbarini et al. 2008). Smaller levels of investment and far fewer usage statistics compared to Internet search engines, combined with information silos, permissions management and information behaviours leads to challenges in providing effective enterprise search environments (Hawking 2004).
Critically, when the ‘standard model’ of information seeking is discussed, researchers may not always consider that various tasks within the model may be undertaken by intermediaries (Hearst 2009, Ch. 3). Where the question is not fully formed in the Geoscientists mind, they will often search themselves and potentially make serendipitous encounters (Cleverley and Burnett 2015a). Where an information need has been clearly defined, exploratory search tasks can be handed to mediators, to search on behalf of Geoscientists (Bichteler and Ward 1989) to benefit from greater search expertise and reduce the time Geoscientists spend on non-interpretative work such as information gathering. Mediators are typically information professionals (librarians, IM consultants, Data Managers (DM) or Technical Assistants (TA)). Usage data shows for the digital library used in this study, 70% of use is by information professionals. Ehrlich and Cash (1999) conclude the expertise of professional search intermediaries can be invisible to the company in which they work. How well exploratory search tasks are performed, may influence oil and gas technical analysis and decisions. Poor search can also miss evidence of fraud (Johnson 2013) and has caused fatalities in the health sector (Savulescu and Spriggs 2002).
There is a dearth of literature on integrated models explaining the causal factors for exploratory search task performance from an organizational perspective (Vassilakaki et al. 2014). Models (represented by a diagram) with associations between concepts are typically rooted more closely to the real world (Case 2012) and so are of value to theoretician and practioner alike. With regards to models on information behaviour and literacy research, Wilson (2008, pg.462) comments “many believe a position has been reached that professional education and research are irrelevant to practice”. The majority of Information Literacy (IL) research covers academic, public and Internet environments (Williams et al. 2014). According to Abram (2010, pg. 205), “We need more discussion and study of the unique needs and challenges of increasing information literacy skills in the workplace”.
There is a need for research which assesses search literacy in the oil and gas enterprise. The research aim is to develop a causal metamodel for exploratory search task performance in the workplace. A number of objectives will be undertaken in order to better understand the antecedent factors for search task performance. The next section reviews the literature including factors which affect search and associated models. This is followed by the research methodology, including experimental design, limitations and method of analysis. The results are presented with discussion to help the reader better understand each finding. The paper concludes with the presentation of the theoretical metamodel, implications for theory and practice and recommendations for further research.
Literature review
This section provides a background to the areas being studied, identifying gaps to inform the research questions. In particular, identifying factors which influence user satisfaction and search task performance as well as person, technology and organization ‘centric’ information models.
Search user satisfaction and performance
Satisfaction is a subjective judgement individual’s make comparing their experience to prior expectation. End user satisfaction of (and engagement with) computer systems has been measured through questionnaire instruments (O’Brien and Toms 2012, Doll and Torkzadeh 1988). Griffiths et al. (2007) advises caution in using user satisfaction as a measure of system performance, (pg. 150), “We need to study the relationships held between various user and environment characteristics and satisfaction”. Al-Maskari and Sanderson (2010) advocate the need to consider more than one factor at a time when researching user satisfaction. In a study of students, high levels of search satisfaction were shown for a task, despite missing key information (Wood et al. 1996). Despite this, many organizations continue to use user satisfaction to measure enterprise search (White 2012).
Search task performance is an objective assessment comparing the items located by the searcher to some form of ideal outcome. In a study of medical students (Sutcliffe et al. 2000) search performance was found overall to be poor, longer evaluation times and broadening/narrowing strategies led to better performance but they did not compensate for poor search term choice. Patterson et al. (2001) conducted an experiment using a large document collection pertaining to the Ariane 501 rocket accident, containing ten high value items. The resulting briefings from intelligence analysts that were of higher quality, were made by those that spent more time, read more documents and identified the higher value items. No measure of satisfaction was taken, so it is not known how they felt about their experience or how the individual or organization may have reacted to feedback. In a review of the ‘search experience’ variable, Moore et al. (2007, pg. 1537) observe “we found that very few studies attempted to use some objective form of measuring the level of user’s search performance”.
Exploratory search in the enterprise
The term ‘enterprise search’ typically refers to Information Retrieval (IR) technology which automatically indexes relevant enterprise content, providing a single place for staff to search without having to know where content resides (White 2012). Enterprise digital library IR systems populated from multiple locations arguably fall outside traditional definitions of enterprise search (as indexing is manual not automatic), but if approached from ‘search as a process’ perspective could be included.
In enterprise search most queries are single word (lookup) and often portrayed as not working well compared to Internet search engines (Andersen 2012). Many users want enterprise search engines to work like Internet search engines, but may be oblivious to the relevant content that can be missed during exploratory search tasks even using Internet search engines (Skoglund and Runeson 2009).
Unlike lookup search, exploratory search describes a range of activities from investigating and comparing to discovery and evaluation. By volume they may be responsible for 8-20% of all enterprise searches (Stenmark 2008). User interface scaffolding (Azevedo and Hadwin 2005) to guide and prompt the user (for example faceted search) have been shown to improve search performance (Fagan 2010). Interest in exploratory search user interface design is of considerable and ongoing interest (Yogev 2014, Golovchinsky et al. 2012, Yang and Wagner 2010, Marchionini and White 2008).
Exploratory search tasks have numerous characteristics; general, open ended, target multiple items, uncertain outcome, multi-faceted, involve query reformulation, other information behaviours and are ‘not easy’ (Wildemuth and Freund 2012, Kules and Capra 2008). Jiang (2014) proposes that exploratory search tasks exist in a continuum involving all these dimensions. When exploratory search tasks are investigated in the literature, there is a tendency to focus on tasks at the more complex end of the continuum (Wildemuth and Freund 2012). Simpler, ‘report like’ exploratory search tasks such as, locate all the information on A for B when C, have received less attention in the academic literature, despite these being commonplace in practice (Liddell et al. 2003).
Causal factors for exploratory search task satisfaction and performance
Griffiths et al. (2007) grouped existing literature on search user satisfaction into four themes; task, system (mixing technology and information quality), user and environmental. There has been (and continues to be), significant research on the impact of system (technology) quality on user satisfaction and search task performance (Al-Maskari and Sanderson 2010, Hildreth 2001, Frokjaer et al. 2000). This review has a focus on the task, user and environmental factors.
Task Factors
Bystrom and Jarvelin (1995) indicate work task complexity is a key variable in search success and user satisfaction. Enterprise information volumes are increasing rapidly which can contribute to information overload (Marcella et al. 2013, Hess 1999). The information overload phenomenon is a personal perception (Wilson 2001 pg. 113), “when the flow of information associated with work tasks is greater than can be managed effectively” and can negatively affect performance, cause anxiety and lower motivation (Bawden and Robinson 2009). With searchers making queries using an average of two words searching combined repositories containing millions of items, it is hardly surprising that many search results contain hundreds if not thousands of results (Cleverley and Burnett 2015b). Work task information overload is typically caused by a combination of information volume (and characteristics), organization design, time pressures (Crescenzi et al. 2013), search expertise, motivation and task type (Eppler and Mengis 2004). Bawden and Robinson (2009) suggest increasing IL may mitigate information overload. This has been taken a step further by some researchers with search task scoring rubrics to measure IL (Leichner et al. 2014). The effect of information overload on workplace exploratory search has received little attention in the literature.
User Factors
Individual differences such as gender (Enochsson 2005), familiarity with an IR system (Moore et al. 2007, Halcoussis et al. 2002) and personality (Heinstrom 2005) have been reported to influence information seeking behaviour. The personality trait of negative affectivity (Watson and Clark 1984) has been correlated with lower levels of search satisfaction (Woodroof and Burg 2003). Satisficers seek adequacy rather than optimality, ‘good enough’ (Simon 1956). Maximisers have higher levels of negative affectivity, seeking the optimal solution (Schwartz et al. 2002). There is little research on the impact of maximizing traits on workplace exploratory search.
Addison and Meyers (2013) divide IL into three areas; acquisition of information age skills, cultivation of habits of the mind and engagement in information-rich social practices. According to Armstrong et al. (2004, pg. 5), “…. Users need to respond to search results – possibly because there are too few or too many – and know when to stop searching”.
The effect of computer literacy, subject matter expertise, IR system and task familiarity on user satisfaction and search task performance has received significant interest in the literature (Smith 2014, Tang et al. 2013, White et al. 2009, Allen 1991). In a review of search experience literature, Moore et al. (2007) identified three methods used to collect search experience/expertise data; professional demographics, self-reporting and objective assessment. For objective assessment Moore only found measures based on frequency/time, not whether searchers had found the most relevant results.
Environmental factors
Montazemi (1988) identified several environmental factors in an organization which may influence user satisfaction of Information System (IS) usage including organizational size and design. Argyris and Schon (1978) describe the interventionist strategies organizations make when outcomes are not as expected, single loop (operationalize actions) and double loop (question the norms). Productivity gains derived from learning curves can be significant (Argote 1999). Attitudes and behaviours toward Knowledge Management (KM) and Organizational Learning (OL) are of considerable importance to the oil and gas industry due to repeated processes (e.g. well drilling), distributed teams and an aging workforce (Grant 2013). Where an organization measures and reflects on the performance of its search technology, it is typically obtained through Information Technology (IT) capability benchmarks (White 2014, Norling 2013), user satisfaction surveys (Meza and Berndt 2014) and search analytics (Romero 2013, Dale 2013) through a search Centre of Excellence (CoE) as suggested by White (2012). These analytics based interventionist approaches are likely to favour lookup searches.
Social cognitive theory (Bandura 2001) proposes that individuals can learn from social interaction and observation, learning new behaviours without necessarily trying them. The impact of social reality on search behaviours is evidenced by cultural differences in information searching (Marcos et al. 2013, Kralish and Berendt 2004). Information seeking in the workplace can be a collaborative social activity (Shah et al. 2013). However, time and resource pressures with short work deadlines, can also make it an isolated activity with few opportunities to learn from others. Experiential self-learning (Kolb 1984) is likely to play a key part in the information search process, where searchers adapt to the results provided by the IR system. Although several studies assess searcher performance (Tabatabai and Shore 2005, Wood et al. 1996), they do not examine the searchers and/or organizations response when presented with knowledge of their actual search task performance.