Log Analysis and Tuning Using Analytics and Tuning Studio

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted in examples herein are fictitious. No association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2007 Microsoft Corporation. All rights reserved.

Microsoft, SQL Server, Windows are trademarks of the Microsoft group of companies.

All other trademarks are property of their respective owners.

Table of Contents

1Overview: Log Analysis and Tuning

1.1Log Analysis and Tuning Scenarios

1.2Trace Logging Configuration

1.2.1Logging Grammars

1.3Log Analysis Framework

1.4Introducing Analytics and Tuning Studio

2Developing Applications for Easier Log Analysis

2.1Define Tasks and Set Task Status

2.2Label Sessions for Later Reporting

2.3Skip Logging for Confidential Data

2.4Use Application Log Events for Debugging

2.5Follow Naming Guidelines

3Log Analysis and Tuning: Analytics and Tuning Studio

3.1Reducing No-Recognitions Due to Out-of-Grammar Utterances

3.1.1Measuring Improvement with a New Grammar

3.2Session Analysis

3.2.1Find Repeat Callers

3.2.2Errors Affecting User Calls

3.2.3Locating Calls with a Custom Label

3.3Task Analysis

3.3.1Find the Most Popular Task and Task Flow

3.3.2Find Tasks on Which User Has Most Trouble

3.4Turn Analysis

3.4.1Finding Recognition Issues

3.4.2Finding the Most Popular Response to a Turn

3.4.3Turns with Unexpected TTS

4Best Practices for Speech Application Tuning

1Overview: Log Analysis and Tuning

This section provides an overview of analysis and tuning scenarios and summarizes the Microsoft Office Communications Server 2007 Speech Server trace logging configurations and Analytics and Tuning Studio.

1.1Log Analysis and Tuning Scenarios

Log analysis and tuning is a part of the speech application development cycle. Successful voice response applications in deployment are those that have been tuned on samples that are representative of the general caller population and the actions of users who are behaving as real end users will.

For example, during the design phase developers can try to predict the full range of input that users are likely to speak in response to a given prompt. After collecting data from a user trial, developers can examine what users really said to the system and update the grammars accordingly.

Predeployment Trials

Many developers roll out their applications in gradual phases to ensure the highest possible quality when it is time for full deployment. After design, debugging, and initial installation are complete, a phased deployment model such as the following is typically used:

  1. Initial pilot. The phone number of the system is given to colleagues, friends, and family with instructions to try out some tasks.
  2. Trial phase. A broader set of trial users is selected (either a subset of the expected caller set or a more extensive range of acquaintances) and more data is gathered.
  3. Further trials. One or more trials can be carried out with a broader or different set of users.
  4. Full rollout. This is followed by continual monitoring. Trace Log Configuration should be set to Default during the full rollout with periodic verbose logging (seeTraceLogging Configuration in thisdocument) during the monitoring period.

After each phase, different parts of the application can be tuned based on the user behavior observed, explicit user feedback about the system, or both.

What Should Be Analyzed and Tuned?

Log analysis offers the developer firsthand experience of users’ interactions with the system. All user-facing components of the application can be tuned to improve the system. Not only can big problems be uncovered, such as bugs that did not surface during the developer's own testing, but considerable improvements can be made to the user experience:

  • Grammars can be tuned to cover the most likely input.
  • Prompts can be made clearer.
  • Dialog flow can be tuned toward more efficient task completion.
  • Confidence thresholds can be optimized to minimize unnecessary confirmations.
  • Timeouts can be optimized for efficient collection of user input.
  • Commands can be enabled where users naturally expect them (and removed where they do not).

1.2Trace Logging Configuration

Speech Server logs data in the form of binary ETL files. Default logging configuration is set to a minimal set of events that are needed for generation of the reports provided with Speech Server.

Logging Tuning Events and Audio for a Tuning Cycle

Use the following steps to enable logging for Speech Server log analysis:

  1. In the Speech Server Administrator console,right-click the server name.
  2. Click Properties.
  3. Click the Trace Logging tab.
  4. Make sure the Enable trace loggingcheck box is selected.
  5. Select the Analytics and tuning events andApplication events check boxes.
  6. Select the All audio for check box, and then enter 100 in the percent of calls box.

Note:Including sample audio can make ETL files larger. Whether you sample audio depending on the available disk space.

  1. Click OK to close the properties window.

1.2.1Logging Grammars

Grammars are automatically logged at load time when logging of Analytics and tuning events is enabled. Speech Server loads grammars at two instances. At startup time, the grammars are loaded if they are specified in the application manifest file. Also, grammars are loaded at runtime when required. Grammar Tuning Advisor can use logged grammars and recognition audio from the ETL files. Logged grammars are needed if the original grammars are not available. If the service was started before the changes were made for the logging tuning events, it might be necessary to refresh the cache. Refreshing the cache ensures that the grammars are logged correctly in the ETL files.

  1. In the Speech Server Administrator console, right-click the server name.
  2. Click All Tasks, and then click Update Cached Resources.

1.3Log Analysis Framework

Importing Log File Data

To view log data using Analytics and Tuning Studio, it is necessary to import it into a SQL Server database. The import process is executed using the command-line utility MSSLogToDatabase.

By default, MSSLogToDatabase imports only the user input audio. During the Tuning cycle, to import the complete session audio, the/audio:session parameter must be passed in the command line.

When log data for only the reports is required, import the log data by passing /filter:reports parameter in the command line.

1.4Introducing Analytics and Tuning Studio

Analytics and Tuning Studio providesan easy way to see log information sorted into different views based on the granularity of information. It has three levels of granularity:session, task, and turn. Typically, a session contains one or more tasks and a task contains one or more turns.

  • Session.A session is generally equivalent to a call.
  • Tasks.A task is a focused dialog in the call whose aim is to get a specific set of information to achieve a goal. In a voice response workflow application, it is analogous to a SpeechSequenceActivity as defined by the developer. A session can contain multiple tasks. A task can contain multiple subtasks.
  • Turns.A turn is usually associated with a prompt or with a prompt and response. A task or session can contain multiple turns.

Analytics and Tuning Studio provides three levels of detail for each session, task, or turnview:report, list, and detail.

  • Report.Gives a quick reference to the most important details for the item in the form of colored graphs and charts.
  • List.Gives one line of data per session, task, or turn.
  • Detail.Gives detailed information for each session, task, and turn and any tasks or turns within them.

2Developing Applications for Easier Log Analysis

Making the correct decisions at application development time ensures that tuning the application from the logs is much easier. The following are suggestions a developer can use during application development.

2.1Define Tasks and Set Task Status

Defining tasks during voice response application development is useful when tracking business metrics such as transaction completion rates and for gauging the usability of parts of the application. The developer can use tasks to measure the characteristics (for example,the average duration and number of repetitions) for different dialogs paths in the application.

The use of tasks allows the developer to:

  • Structure the application into a hierarchy of tasks and subtasks.
  • Provide a means to signal the success or failure of the task.

The Task Report view in Analytics and Tuning Studio provides a snapshot of the performance of the various tasks defined in the application. The Task List view and Task Detailview can be used for detailed analysis.

In Voice Response Workflow Applications

  • SpeechSequenceActivity can be used to define a task. Task logging is automatically done for this activity. The default status associated with a task is Unset.
  • Some of the activities, such as FormFillingDialog and GetAndConfirm,have task logging enabled by default. If task logging on these activities is not required for business metrics or performance tuning, the developer can turn off task logging for these activities. This allows the developer to betterfocus task analysis on tasksimportant for tracking of business metrics.

  • SetTaskStatusActivity is used to set the status of a task on completion. Use this activity to also associate a message with the status. This message text can be used by the Task Reason report in Analytics and Tuning Studio to track both the reason of failures or success of an individual task as well as the number of tasks associated with a particular message. SetTaskStatusActivity is critical to tracking the success or failure of a task. The idea is to make sure that every possible exit out of a task sets the task status, unless you explicitly want it Unset.
  • Changethe task status based on execution.

Do this when the default task status is set to Failure and the task completion changes the status to Success. This can be used in cases where the task has a bailout and hence bailout leaves the task status as Failure. For example,a bailout is set to occur after three consecutive no inputs for a task. If the user does not provide any input three times in a row, execution exits the task. In this case, the status can start with the status set to Failure, and then if the bailout executes, the task status is appropriately recorded as failure.

The previous task sequence has a bailout set as shown by the icon on the top right corner. The task status is set to Failure to start.If the user provides the information, the task status is changed to Success. If the user bails out in between, the task status is recorded as Failure by default.

  • Set the task status appropriately for all the exit paths for a given task.

In the previous example, based on confirmation of the user input, two SetTaskStatusActivities are being used in the branches to set the task status accordingly with SetUserInputSucceeded and SetUserInputFailed.

In VXML Application

VoiceXML does not implement tasks with a completion state. However, with the Speech Server implementation of VoiceXML, developers can specify the task states in code. Accordingly, authors should signal the start and end of a task using the standard VoiceXML logging mechanism. See the following examples.

  1. Signaling theStart of Task

Syntax

<log label=”__EnterTask”>TaskName</log>

Example

<log label=”__EnterTask”>CollectUserInput</log>

  1. Signaling the Completion of Task

Syntax

<log label=”__ExitTask”

TaskName:TaskStatus:StatusMessage

</log>

Example

<log label=”__ExitTask”

CollectUserInput:Success:User Completed Task

</log>

In the previous examples, TaskName is a string that names the task, for example GetDate;TaskStatus is a string that should be set as Success, Failure, or Unset; and StatusMessage is a string that defines a reason for the completion status, for example User Hang-up.

Refer to the product help documentation TopicDesign VoiceXML Applications for Easy Reporting and Tuning.

2.2LabelSessions for Later Reporting

A given application might want to label or otherwise annotate a call at runtime to reflect the particular analytical need at the time of execution depending on the characterization needed for the business. For example, a distributor might want to classify its customers into different types, such as Gold and Platinum.This type of labeling, which provides information about the caller, helps give morespecific insights into an application’s performance and user experience.

Use the LogSessionLabelmethod to custom label a call at runtime. See the following example.

ApplicationHost.TelephonySession.LoggingManager.LogSessionLabel("CustomerSubscriptionPlan", “Gold”);

Default trace logging configuration is sufficient to log the trace event in the ETL file.

The calls can then be queried in Analytics and Tuning Studio based on filtering with the custom labels defined in the application and meeting a certain developer-defined criteria. Analytics and Tuning Studio provides a Session Label filter to accomplish this (seeLocating Calls with a Custom Label in thisdocument).

The custom label data exists in the UserEvents and UserEventTypes tables of the database.

2.3Skip Logging for Confidential Data

Some applications are required to ask for sensitive information such as a social security number or credit card number.This information must be kept completely confidential. Itmight be a security or privacy violation to record it in the logs.The developer can turn off logging for the dialog elements that collect confidential data.Disabling the logging for dialog elements that collect confidential data addresses privacyconcerns in log data. When logging is disabled, prompt text, recognized text, and the associated audio are not stored in the logs. Of course, data that is not logged is also not available for tuning. See the following screen capture from the Turn List view, which shows how the absence of confidential data is indicated in Analytics and Tuning Studio.

In Voice Response Workflow Applications

Also, be sure to turn off logging for any turns that repeat confidential information, such as when there is a statement or separate confirmation question. For higher level controls, such as GetAndConfirm and NavigableList, turning off logging for the parent activity is sufficient to turn off the logging for the confirm phase inside these activities. This precaution avoids any accidental data disclosure.

In VXML Applications

Standard VoiceXML does not support controlling the logging of turn data to protect privacy, but this feature is available for VoiceXML applications on Speech Server. This is platform-specific functionality, added by Speech Server through a proprietary <property>. See the general form in the following example.

<property name="com.microsoft.[component].LogResults" value="false" />

In the previous example, [component] is a reco, dtmf, or prompt.

2.4Use Application Log Eventsfor Debugging

Another important tracking facility is the use of application events.Application events can be used in exception handlers and application code to track application activity. It can be used for application diagnostics and call flow tracing purposes. This provides an ability to log more atomic information for application-specific data.

In Voice Response Workflow Applications

Use the four methods described in the following table.

Method / Description
LogApplicationData / This method logs an ApplicationDataEvent and can be used to log any useful data from the application.
LogApplicationError / This method logs an ApplicationErrorEvent and must be used in case of any error that the application encounters.
LogApplicationWarning / This method logs an ApplicationWarningEvent and can be used for logging information in cases of unexpected but non-fatal results.
LogApplicationInformation / This method logs an ApplicationInformationEvent and can be used for information logging. Logging configuration needs to be modified (Application Eventscheck box should be selected) to log application information events.

See the following code example.