Memory Dump File Concepts And Processes

Microsoftò Exchange Server 2003 and Microsoft Exchange 2000 Server

Memory Dump File Concepts and Processes

Microsoft Product Support Services w hite p aper

Written by Steve Justice

Published June 2004

Revised June 2005

Abstract

This white paper describes memory dump files, the tools that are used to collect them, and the techniques that support engineers use to troubleshoot the processes that Exchange relies on.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Microsoft , Windows , and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Contents

Introduction 1

Memory Dump file Concepts 2

Definition of a memory dump file 2

Information that a memory dump file captures 2

Differences between full memory dump files and mini memory dump files 3

Why it takes so long to create a memory dump file 3

Tools that are used to capture memory dump files 4

Memory Dump file Processes 6

Process stops responding 6

Performance problems 8

Crashes 9

Memory leaks 11

Reducing the time to upload memory dump file information 12

CONCLUSION 13

For More Information 14

Appendix A 15

APPENDIX B 17

APPENDIX C 19

APPENDIX D 20

Introduction

This white paper provides general guidelines for troubleshooting certain problematic scenarios. Do not assume that the steps that this white paper provides are the only methods that you have to use to troubleshoot any problem. Some circumstances may require more than one approach to determine the root cause of a problem.

Additionally, memory dump files are not always the best way for a support engineer to determine the root cause of a problem. Sometimes a memory dump file helps direct the troubleshooting, but does not point out the exact root cause of all problems. In those cases, additional troubleshooting steps must be used. Such additional steps may include using other tools or diagnosis techniques.

If a Product Support Services (PSS) support engineer is helping you solve a problem, the steps that you take may differ from the steps in this white paper. Troubleshooting the problems that are listed in this white paper is a complex process. Because of this, the type of information that is necessary to diagnose a problem is largely left to the support engineer’s discretion. The complexity of code-level troubleshooting requires many techniques to gather pertinent information.

The information in this white paper is not intended to supersede the judgment of a PSS support engineer who is diagnosing a problem.

The information in this white paper applies to:

· Exchange 2000 Server

· Microsoft Exchange 2000 Enterprise Server

· Exchange Server 2003

· Microsoft Exchange Server 2003 Enterprise Edition

Memory Dump file Concepts

The concepts in this white paper refer only to Exchange-related processes that run in User mode. If you have to diagnose a Kernel mode process, the steps and methods are different depending on the nature of the problem.

Definition of a memory dump file

A memory dump file is a copy of the contents of the virtual memory for a particular process. A debugger can create a memory dump file. The debugger writes the contents of virtual memory to a file on the hard disk. Later, another debugger can be used to open that memory dump file and treat the file as if it is a "frozen" instance of a live process that is being debugged. A support engineer can use this snapshot of virtual memory to examine things, such as the heap memory and the call stacks for every thread that was running for that process.

Information that a memory dump file captures

Memory dump files capture the contents of the virtual address space at one moment in time for a process that is running. That virtual address space contains almost everything that a process was using at that time to conduct its operations. Some of the most frequently used information in memory dump files is:

· Heap memory. This information includes objects or variables that are created by a process which is using statements in C++ such as "free," or "malloc" in C. Heap memory is generally used to determine what data was loaded at the time that the memory dump file was collected.

· Call stacks for all threads. These call stacks contain all the function calls that were being processed at the time that the memory dump file was collected, together with the variables that were on the stack. Call stacks are used to determine what was happening at the time that the memory dump file was collected.

· Thread environment blocks. These blocks contain information about a running thread. Support engineers can use this information to determine what the state of the thread was and what the thread ID is.

· Assembly c ode. The assembly code that each process is executing is loaded in virtual memory. A memory dump file can capture that assembly code, and a support engineer can use that code to help debug the problem so that the support engineer can determine how variables or processor registers reached their current state.

· Module i nformation. Most processes use multiple files when they are executing. For example, many DLL files are frequently used to load supporting libraries for a process. A process might contain files that were created by Microsoft or by other companies. A memory dump file captures information about those loaded modules so that their version information and assembly can be examined later.

Differences between full memory dump files and mini memory dump files

A memory dump file can collect a variety of information. Typically, a support engineer must have all the contents of virtual memory to troubleshoot a problem. In other cases, you might want to capture less information to focus on a specific problem. The debugger is flexible. This flexibility lets you limit the information that a memory dump file captures by collecting either full memory dump files or mini memory dump files:

· Full m emory d ump file s. These files contain the contents of virtual memory for a process. These files are the most useful when you are troubleshooting unknown issues. A support engineer can use these files to look anywhere in memory to locate any object, pull up the variable that was loaded on any call stack, and disassemble code to help diagnose the problem. The disadvantage of full memory dump files is that they are large. It also may take additional time to collect these files, and the process that is being recorded must be frozen while the dump file is created.

· Mini memory d ump file s. A mini dump file is more configurable than a full dump file and can range from only several megabytes (MB) up to the size of a full dump file. The size differs because of the amount of virtual memory that the debugger is writing to disk. Although you can gather mini memory dump files quickly and they are small, they also have a disadvantage. Mini dump files may contain much less information than full dump files. The information that a mini dump file gathers may be virtually useless to a support engineer if the area of memory that the support engineer has to troubleshoot was not captured. For example, if the heap memory is not written to the memory dump file, a support engineer cannot examine the contents of a message that was being processed at the time that the problem occurred. Useful information, such as the subject line and the recipient list, would be lost.

Why it takes so long to create a memory dump file

When a debugger has to copy the contents of virtual memory to disk, all threads for that process must be frozen at the same time. All threads can remain frozen until the memory dump file is finished, and then all threads are unfrozen at the same time. Because of this, a process such as the Store.exe process may cause clients to stop responding (hang) or report that the Exchange computer is unavailable. The time that is necessary to write a memory dump file is directly related to how much information must be written to disk. If a process is using 1.5 gigabytes (GB) of virtual memory, several minutes may pass before the memory dump file can be created. This behavior occurs because approximately 1.5 GB of data is written to the memory dump file. During that time, all threads can remain frozen to prevent unexpected behavior in the process.

Tools that are used to capture memory dump files

Many tools can capture virtual memory. Each of these tools uses some form of a debugger to create memory dump files.

The tools that PSS support engineers use most frequently are:

· Dr. Watson. This tool is a "just in time" debugger. Dr. Watson creates User.dmp files for a process whenever a second chance, unhandled exception is encountered. In most circumstances, a second chance, unhandled exception occurs because of an access violation, a stack overflow, or some condition that an application was not written to handle.

· ADPlus. This is a script that uses the CDB debugger to capture virtual memory. ADPlus can be configured to create memory dump files when hardware fails or a process stops responding. ADPlus has additional flexibility; it can capture multiple processes with one command-line instruction. ADPlus has two modes of operation: Hang mode and Crash mode:

o Hang mode noninvasively attaches to a process and creates a memory dump file that is based on command-line parameters. After the dump file is created, ADPlus detaches the debugger, and the process continues to run. ADPlus gives you a snapshot of what the process was doing at a point in time.

A typical ADPlus command line to capture a Hang mode memory dump file is:

adplus –hang –pn [ process name ]

o Crash mode invasively attaches to the process and waits for an exception to occur. By default, ADPlus creates a mini memory dump file for every first chance exception that is encountered. Whenever a second chance exception occurs, a full memory dump file is collected and the process is terminated. If ADPlus is in Crash mode, and the user decides to detach ADPlus by pressing CTRL+C, a full memory dump file is created, and then the process is terminated.

A typical ADPlus command line to capture a Crash mode memory dump file is:

adplus –crash –pn [ process name ]

· Userdump.exe. This is a command-line tool that can capture the virtual memory for one process when a process stops responding or an unhandled exception is encountered. Userdump.exe can create full memory dump files.

· Microsoft Windows ? Debugger. If a support engineer is attached to a process with the Windows Debugger, the support engineer can create either full dump files or mini dump files while debugging. This method gives the support engineer the greatest amount of flexibility to create memory dump files.

Note This white paper describes how to use the Adplus utility to configure the debugger to collect memory dump files. You can also use t he Windows Debugger , but the syntax that you must use is more complex and beyond the scope of this white pa per. Therefore, the equivalent Windows Debugger commands are not included for the examples in this white paper .

Memory Dump file Processes

Different approaches must be taken to use memory dump files to diagnose problems. Support engineers try to minimize the impact that gathering memory dump files has on a customer, but still try to collect all the information that they must have to troubleshoot a problem. To minimize the impact of information gathering, tools have been designed to help customize the way that memory dump files are gathered. This section provides some general guidelines about how to approach certain situations to bring problems to a resolution as quickly as possible. The information in this section is not intended to supersede the judgment of a PSS support engineer who is diagnosing a problem.

Note All the examples in this section assume that the Debugging Tools for Windows are in stalled on the Exchange server. T o install t he Debugging Tools for Windows , visit the following Microsoft Web site :