Comparative Performance Evaluation of Symantec Corporation's ACT! 4.0 with Gold Mine

June 16, 1998

Table of Contents

Table of Contents......

Executive Summary......

Conclusions......

USABILITY Test RESULTS......

Introduction......

Detailed Scores by Category......

Detailed Test Results......

Documentation/Tutorial/Help......

Interface......

Basic Contact Management......

Test MetHODOLOGY......

Hardware Configuration......

Software Configuration......

Tester Backgrounds......

Usability Test Procedure......

Overview of Methodology......

Usability Test Scenario......

Data Collection Questionnaire......

Detailed Test Methodology......

Tutorial / Help......

Interface......

Data Entry......

Editing and working with contact data......

Groups......

Communication......

Advanced contact management......

Scheduling......

Communication......

Specialized Features......

Backup......

Eliminate duplicates......

Data synchronization......

Data Import......

Helper Applications......

Customization......

This report was prepared by National Software Testing Laboratories (NSTL) under contract for Symantec Corporation (Symantec). NSTL does not guarantee the accuracy, adequacy or completeness of the services provided to Symantec or the data included herein. NSTL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO RESULTS TO BE OBTAINED BY ANY PERSON OR ENTITY FROM USE OF THE CONTENTS OF THIS REPORT. NSTL MAKES NO EXPRESS OR IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OF ANY PRODUCT MENTIONED IN THIS REPORT.

Executive Summary

NSTL is the leading independent hardware and software testing organization for the computer industry, dedicated to providing high quality services and test tools to hardware developers, software publishers, government agencies and corporations. NSTL has extensive experience developing and conducting objective tests to assess new and existing products for compatibility, performance, comparative performance, usability, acceptance (bug) testing, and BIOS evaluation. NSTL’s proficiency and thoroughness provide clients with a high quality, cost-effective means to conduct this testing. More information on NSTL is available through the World Wide Web at

Symantec Corporation (Symantec) requested a comparative review of the usability of its ACT! application versus GOLDMINE Software Corporation’s GOLDMINE.

The usability of these two products was evaluated using six testers. These testers spent one full day with each program, and then compared the programs’ relative strength in each of eighteen categories. The tested versions were those publicly available: ACT! version 4.0.0 (1998) and GOLDMINE Standard Edition version 4.00.127 (January 1998).

Conclusions

Testers overwhelmingly preferred ACT! to GOLDMINE. Across eighteen different ratings categories, GOLDMINE equaled ACT! for ease of use in only one area. In every other rating, ACT! scored higher.

ACT!’s superior usability is most pronounced in its documentation and tutorial – of key importance to new users. Contact communication and reporting also prove dramatically easier to learn and to use in ACT! than in GOLDMINE. For basic contact management and scheduling, the differential is not as great, but ACT! consistently receives higher ratings.

Although numbers are useful to summarize findings, the most valuable insights come from testers’ observations and comments. The detailed report of results (starting on page 7) includes specific commentary to complement the scores in each test category.

NSTL reports separate ratings for Ease of Learning and Ease of Use. This permits distinguishing between features that are easy to learn, but clumsy in use … and features that may be difficult to learn, but that incorporate convenient shortcuts to make routine use faster and easier. In the majority of cases, well-designed functions prove to be both easy-to-learn and easy-to-use.

Scores shown below are scaled from 1 to 10, with 10 as the highest possible score.

Ease of LearningEase of Use

ACT!7.8ACT!8.2

GOLDMINE6.9GOLDMINE7.0

USABILITY Test RESULTS

Introduction

Software usability testing analyzes the quality of interaction between user and product. Usability measurement is based on users’ own evaluation of their ability to learn and use the product, as well as on their overall satisfaction with the software. Such measurement can be assessed both quantitatively and qualitatively. NSTL collects both rating data and user comments in a detailed usability questionnaire and in a group debriefing session with testers.

Usability tests are designed to simulate real-word usage of the product under test. Testers are selected to be representative of the typical user audience for the given programs. Test scenarios are designed to present testers with results-oriented tasks utilizing basic program functions. Testers work to achieve the desired results relying only on the program and its related support materials.

The scores shown below reflect the rating consensus of testers achieved at their debriefing session. Analysis and representative tester comments follow, to shed light that numbers cannot.

Detailed Scores by Category

Detailed Test Results

The following pages detail NSTL’s findings in each key-rating category.

Documentation/Tutorial/Help

Testers evaluated the usability of documentation, the clarity and helpfulness of the tutorial, the ease of referencing the on-line help screens, the degree of context-sensitivity and the helpfulness of the information found there. They also evaluated the ease of referencing and the layout of the manuals.

They found ACT! 4.0’s tutorial to be very clear and informative as well as very helpful. Although the tutorial videos were very long they were easy to follow and were used frequently. “Probably the best demo program I’ve ever seen, it was like having a VCR on screen.” “It was like having my own personal tutor on screen”. GOLDMINE didn’t have a tutorial, only an intro that gave a very simplified explanation of the products capabilities and functions. “The info center was more of an advertisement than an informative tutorial.” “It’s only purpose seemed to be to promote the product”.

The testers found ACT’s on-line help and manuals were very helpful and context sensitive. The on-line help screens did seem to be a little cluttered but they were a good source for quick look-ups. “I found the reference manuals to be a much better source of information”.

GOLDMINE’s manuals were found to be descriptive and easy to reference but they lacked good graphical presentation. It was stated that the on-line help screens would be more helpful if the main points were bulleted. “The on-line help screens were not too cluttered”.

Interface

The testers were asked to rate and give comments on the general screen layout, the depth and clarity of menus, icons, toolbars, and menus, and the degree to which you have to switch around to different screens to accomplish a task. They were also asked to rate the ease of switching views, consistency of operations, and the efficient use of the status bar or other indicators.

They found that ACT!’s screen layout was cluttered especially when in full contact view, but it was easy to read and very comprehensible. Menus were mostly one layer deep, and the icon, toolbars, and menus were visually excellent. There was very little switching around to accomplish a task, and the switching of views was very easy.

The testers didn’t find GOLDMINE’s screen to be cluttered, but they weren’t happy with the over all design. One example that they gave was that when they used the standard toolbar instead of the basic toolbar you need to rearrange the screen so that all of the icons are visible. Some key functions are buried several layers deep (e.g., switching toolbars). The icons are easily identified and clear, and it is very easy to click and switch views. Although most tasks are linked together, you still have to switch screens. Often it took three or four screens to do one task.

Basic Contact Management

The testers evaluated the usability of the programs’ advisors, wizards, etc., for help in repetitive data entry. They tested the ease of finding contacts based on name, company, or other characteristics, the ease of editing/updating existing data and the ease of classifying contacts into groups.

The testers stated that with ACT! it was very easy for them to update contact and company information once it was retrieved. They found the lookup feature simple to use, but thought that an icon would have been more helpful. They commented that the wizard wasn’t very helpful when trying to learn how to classify contacts into groups.

When using GOLDMINE to make new contacts within the same company, the program has a feature to help avoid repetitive data entry which, the testers found to be helpful and time saving. They especially liked that titles such as Mr., Mrs., and Dr. are automatically added. However, one tester stated, “Some users might be offended since Ms. is not offered automatically.” They found it easy to edit data, and to look up contacts using query or the icon. They found it difficult to classify contacts into groups and the on-line help gave only a small idea on how to accomplish the task.

Basic Communication and Report

The testers were asked to evaluate the ease and usability of dialing a telephone to reach a contact, generating a report that lists contacts, and creating a quick note to a contact.

They found that with ACT! dialing the telephone was not as easy as the other tasks. The testers stated that “You need to make sure that the program is configured properly, a possible problem for more inexperienced users.” They found it easy to generate a report and utilize the contact menu. The testers also found it easy to create a note, One tester commented “you simply click the note icon and contact names are added automatically to the already formatted template.”

It was found to be just as difficult to dial the phone with GOLDMINE. The testers also found it difficult to generate a report that links contacts. Help needed to be accessed by the testers to complete this task since the command for this procedure was not obvious to them. They did find the help to be very useful though as it answered all of their questions and guided them through the procedure. One tester commented “It was easy to create a quick note to a contact, when using mail merge, the template automatically includes fields for name, address, etc. this was a big plus.”

Scheduling

Testers were asked to evaluate the ease and usability of setting appointments, tasks and recurring tasks, the ease of viewing and reporting planned activities, setting, changing, and viewing priorities, changing plans, setting reminder alarms, and changing (making exceptions to) recurring plans.

The testers found scheduling appointments with ACT! easy. They stated that the icons made it simple to quickly accomplish the task. Although, the group agreed overall that a “general” category needs to be added (e.g., meeting, call, and to-do). One tester stated “I found scheduling annoying when planning a “personal” meeting like a doctor’s appointment, because the program insists that company meetings, phone calls, only work with contacts that you have already created. So, I had to create a contact with “personal” as the name just so when I created an appointment my schedule would read; e.g., Dr.’s visit with personal instead of Dr.’s visit with Gail Miller or any other contact name in my list. I found this to be very annoying”. They found setting, changing and viewing priorities to be very simple to do. The testers stated “It was easy to change plans especially because the program reminds you when you try to schedule a conflict.” The testers found that changing recurring plans was simple and very useful.

With GOLDMINE, it was very easy to set appointments and tasks. The testers found it slightly harder to set recurring events as how you had to flip screens. They said it was easy to view and report activities but it would be better if clicking on zoom wouldn’t be necessary. They found changing priorities to be tedious since one must click on a separate properties box and type in the new priority. The testers found it to be a lot more time consuming than simply being able to click on the priority box to change it. It was easy for them to change plans, but only after they had learned how to use the properties feature. They stated that it was very easy to set the reminder alarms, but that they really weren’t loud enough. The testers stated that it was difficult to make exceptions to recurring plans. One tester commented “I was able to change one plan, but I then needed to set a whole new series of plans for the next week.”

Advanced Contact Management

The testers rated the ease of finding multiple contacts based on shared characteristics, logging completed activities, finding information in contact histories, and the ease of editing/updating existing data for multiple contacts at once.

The testers found that with ACT!, it was easy to find contacts through a key word search. They discovered that when you want to log completed activities all you do is simply click on an activity and it is moved to the history file, a feature the testers liked and found time consuming. They also found it very simple to look through the tab menu for each contact and to access the calendar which they though to be handy. The testers found it easy to edit and update existing data with group functions such as “add group note.”

With GOLDMINE the testers found it to be easy to find multiple contacts based on shared characteristics using the query function, but difficult to display them all in the same report unless a filter or group is created for them. They found it fairly easy to log completed activities once they located the find logging function. Since there is no keyword search function the testers found it difficult and were unclear on how to find information in histories One tester stated “It was difficult to discover how to add data for multiple contacts at once. I needed to use the help menu, yet it was easy to operate once I discovered how to do it.”

Advanced Communication and Reports

Testers rated the usability and ease of generating reports of activities, generating mailing labels, creating and sending e-mail, faxes, and “personalized” mass mailings.

The testers found the report function quite easy to use. It took them a little bit of time to learn how to generate mailing labels, but once they located the function they found it very easy and convenient to use. One tester commented “Everything could be done automatically! You can add by group, content or lookup”. They found it very easy to create, add multiple recipients and send e-mail. It was also very easy for them to create a fax using the simple WinFax icon. The testers commented that “It was very easy to create and generate mass mailings, you could easily make mail merge templates and personalized letters with Word.” They testers stated that they found this a wonderful part of the program

They found it fairly easy to generate a report of activities with GOLDMINE, but stated that the report looks very unappealing. The testers felt that the report needs to be in an easier to read format. They found it to be very difficult to figure out the path to take in order to generate mailing labels, but easy to type and send e-mail and faxes. They felt that creating mass mailings seemed to work ok.

Special Features

The testers were asked to comment on the ease and usability of backing-up contact data, finding and eliminating duplicates, synchronizing data (setting up and running), importing data from other sources, configuring helper applications, and the ease of customizing the program,