Using the Cortana Intelligence Suite for Sales and Marketing Predictive Analytics

Page 1 | Using the Cortana Intelligence Suite for sales and marketing predictive analytics

Using the Cortana Intelligence Suite for sales and marketing predictive analytics

As Microsoft continues to evolve intoa productivity and platform company, the amount of data to analyze grows exponentially. This has created unique opportunities for data scientists to add significant value at Microsoft.Data science teams are becoming increasingly common across the enterprise.

As the business value of data science grows at Microsoft, Microsoft IT needsmore centralized and standardized ways to manage predictive models. We used Microsoft Azure and the Cortana Intelligence Suite together to create a platform that generates and manages predictive analytics in a new, different way.

We developed this platform approach to predictive analytics to improve discoverability and shared access to predictive modeling assets at Microsoft. Some of the benefits that this approach has provided include:

• Reduced the computing time require for data featurerefreshes by 93 percent.
• Reduced the resource hours required to develop predictive models by 20 to 80 percent.
• Published scores to a single catalog, improving discovery by the sales and marketing team.
• Reduced the resource hours required torefresh the model portfolio every monthby 89 percent.

Business challenge

With the increasing use of data science and scientists, comes the risk of redundant and disconnected efforts across such a large organization. Traditionally, data scientists workedindependentlyto build their own predictive models. This method wasn’t efficient; often, data scientists would create a new predictive model where one already existed for a very similar business problem. And the lack of standardization sometimes led to inconsistent results between predictive models. The following figure illustrates these business challenges.

Figure 1. The business challenges

When we started this process,we were spending more than 80 percent of the time required to create a predictive model on identifying datasets and generating features. We had over 100 on-premisespredictive models in our portfolio, but we weren’t managing them effectively.

Three data scientists managed the portfolio, each owning their own set of over 30 models, for which they had to maintain full end-to-end operations including data management, model retraining, model scoring, score publishing, and reporting.Using data scientists to manage data operations tasksis costly and is not a great use of their unique expertise. Many of the operational tasks are repetitive in nature, but without a standardized process or platform,they were difficult to automate.

Another challenge was that the models were entirely dependent on the expertise and availability of the data scientist whoowned them. For example, if someone had questions about a model, only the data scientist whobuilt the model could effectively answer them.

Defining our solution objectives

Our goal was to provide best-in-class, enterprise-scalable, cloud-based capabilities and services that would simplify development, deployment, and operationalization of predictive insights in support of modern marketing and sales transformation.

We started by looking at how we could use Microsoft Azure and the Cortana Intelligence Suite to optimize the management of our predictive model portfolio, including data management, model development, model maintenance (ongoing refresh), model publishing, and model operationalization. The platform approach was developed to allowdata scientists to achieve more usingprocess standardization and predictive insights democratization.We wanted the platform to support not just the predictivemodels that our team builds, but also to support models that other teams build.

To meet our goal, we defined our objectives for the predictive analyticsplatform. They included:

• Facilitating predictive modeling against large, sometimes unstructured, datasets ina variety of sources and formats.
• Using Azure services forcloud storage, computing, and orchestration.
• Providing a platform to centralize management and deployment of security and privacy policies for data sources.
• Facilitating collaboration on new data science problems and democratizing the creation and consumption of modeling insights.
• Enabling rapid model development to meet the accelerating pace of business data needs. – fostering collaboration by featuring unlimited extensibility (R, Python, etc.)
• Making predictive models easy to deploy as web services thatsupport request-response services and batch execution services.
• Automating modeling operations usingtraining/retraining pipelines.
• Governingmodel performance using astandard model evaluation framework.
• Curating a central catalog of predictive models featuring both owned experiments and third-party assets.

• Delivering predictive insights for key marketing and sales scenarios.

Creating a predictive analytics platform

The predictive analytics platform we designed is an enterprise-scale data mining and reporting platform built for developing and publishing predictive models. Intended for shared use across the company, the platform also facilitatesplatform-enabled collaboration scenarios. The platform was designed forthree primary stages of model development: feature preparation, model development, and score publishing. In addition, model operationalization is a foundational element of the architecture.

To build the platform, we first created acommon library of all existing features (variables and predictors) that had proved to be predictive in our past modeling efforts. We standardized and rationalizedacross these features,where necessary, to create a feature variablemart that supports the ongoing maintenance of existing models and serves as a feature catalog for new modeling efforts.

Next, we converted our portfolio of on-premises models to Azure Machine Learning (AML) experiments and created a catalog on SharePoint Online to assist marketers and sales professionals with asset discoverability. The platform provides a single place to publish and inventory predictive model results (scores). Sales and marketing professionals from across the globe have one catalog where they can discover, visualize, and consume relevant scores.

Finally, we introduced model operationalization that automates the data processes required to refresh features, rescore (and periodically retrain) models, and republish scores for all the models on the platform.

The Cortana Intelligence Suite provides key capabilities

Using key capabilities in Microsoft Azure and the Cortana Intelligence Suite, the predictive platform was built to address all of our defined objectives, as shown in the following figure.

Figure 2. Predictive analytics platform capabilities

Feature preparation using Azure Storage

Feature preparation involves all the activities necessary to assemble data for modeling—including data selection, derivation, enrichment, cleansing, and formatting. Our Azure-based feature variable martis a curated collection of precalculated variables (or features) spanning different business perspectives (e.g., organizations, partners, individuals) that have proven predictive value based on models already published on the platform. As new models are published to the platform, each model’s variables are added tothe feature variable mart.These features enable ongoing operationalization of that model and serve as reference features for other data scientists that may find them useful in future modeling scenarios.

The feature variable mart enables rapid data model prototyping

The variable mart and the centralized repository of existing experiments on AML support rapid model prototyping and more agile predictive analytics. The ability to use existing features and model experiments as a template for extending and developing new models is much more efficient than designing and creating experiments from scratch. The collaborative features of Azure ML Studio enable data scientists to easily share their workspace with others, who can then collaborate on the same experiments, or even clone an experiment, to use as a starting point for a completely new model.

Model Development via Azure Machine Learning

Model development involves training and validating multiple models (experiments) to inform selection of the model that best fits the business need. AML has powerful cloud-based predictive analytics capability that streamlines the model development experience for all data scientist skill levels. By focusing on AML-based experiments, the predictive platform offers modeling workspaces with best-in-class algorithms, an intuitive drag-and-drop interface, support for Rand Python custom code, cloud-based collaboration, and the ability to easily deploy models as ready-to-consume web services.It supports both batch execution services and request-response services.

The AML experiment catalog also reduces duplication of effort that occurs when data scientists can’t see what models their peers have already developed, which can be common in a company of our size.

Score Publishing via Azure API services

Publishing scoresinvolves delivering model outputs (scores) to business. All models managed on the predictive platform publish scores to a standardized Microsoft Azure/Microsoft SQL Server analytics publisher. In addition, a parallel model onboarding service permits non-platform (or third-party) model scoring outputs to be cataloged in our analytics publisher. As a single, centralized catalog of model scores, AP enables:

• Improved discoverability of predictive assets.
• Increased utilization of predicted scores in business workflows.
• A standard web services (API) layer to support direct integration of predictive assets within modern applications.

• Standard model evaluation frameworks for both usage and impact.

The single layer makes it easier for application developers as well; they only need to point their applications to one source of predictive output. Without this platform, anybody that builds a model and wants to integrate these capabilities would have to work directly with engineering teams that maintain the specific predictive models.

Model operationalization via Azure Data Factory

Model operationalization involves refreshing features, rescoring (and periodically retraining) models, and republishing scores. The predictive platform automates these activities using Azure Data Factory (ADF). For each model managed on the platform, ADF:

• Schedules refreshesof relevant features in the variable mart.
• Orchestrates the batch execution of the variable mart features against AML endpoints for batch scoring and periodic retraining.
• Provides standard model evaluation frameworks for both usage and impact.

• Configures model outputs for alignment to our analytics publisher schema to ensure compatibility with web services and reporting.

Operationalizing predictive analytics

From an operationalization standpoint, the refresh/retrain/republish process is now automated and requires minimal human intervention. The automated refresh is agile and adaptable to model inclusions or exclusions. The process also has automated pipeline and model performance monitoring built in, so that system admins and model publishers are notified of model refresh and publish status. The statistical performance of all the models is monitored and then archived during every refresh cycle.

Benefits of the new platform are illustrated in the following figure:

Figure 3. Benefits of the predictive analytics platform

• Because we migrated computing and processing workloads to the Azure Data Lake, we saw a 93 percent reduction in computing time requiredto refresh featurescompared to on-premises servers.
• The ability to create new predictive models based on cataloged experiments has reduced the time required to build model, and reduced the need for data scientists to create custom solutions. We are more agile and have seen a 20 to 80 percent reduction in the hours required to develop predictive models by using features in the variable mart. The range of savings varies based on how many features have to be built or can be based on existing feature variables in the mart to create a new scenario. Predictive models built using only the features that are available in the variable mart can see as much as an 80 percent time saving.

• Azure Data Factory automated and operationalized the model portfolio,which reduced the hours required to refresh the model portfolio every month by 89 percent.

Predictive Analytics is becoming a vital solution in our sales and marketing efforts at Microsoft. The predictive analytics platform has made it easier for other sales and marketing groups to onboard their experiments andtake advantage of its capabilities.

Marketing professional Philip Jordan says, “Western Europe used Azure Machine Learning, available in the Cortana Intelligence Suite, to analyze and segment accounts based on their 10-year Microsoft sales transaction history. That analysis generated some interesting findings:the top 13 percent of major accounts spent 18 times more than the bottom 2 percent of major accounts over a 10-year period—concluding that not all major accounts are equal. The basis of this early analysis is now being applied to the lead scoring models that score marketing leads generated through the system. This new insight will help ensure that sales professionalsreceive high-quality,prioritized leads based on customer action and insights.”

Conclusion

We developed this predictive analytics platform to enable discoverability and access to predictive modeling assets across Microsoft. As the platform continues to grow and the capabilities continue to evolve, it is our goal to empower more data professionals to engage, customize, and deploy their own predictive models.

For more information

Predictive analytics improves the accuracy of forecasted sales revenue

Using predictive analytics to improve financial forecasting

Cortana Intelligence Suite

Microsoft Azure- Cortana Intelligence Suite

Microsoft IT

microsoft.com/itshowcase

© 2016 Microsoft Corporation. All rights reserved. Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.

IT Showcase Article

microsoft.com/itshowcaseJuly 2016