Avoid SAP HANA System Surprises with a Standard Operating Procedure Checklist
by Dr. Bjarne Berg, VP SAP Business Intelligence, Comerit, and Professor at SAP University Alliance at Lenoir Rhyne University
Key Concept
Writing a standard operating procedure (SOP) checklist and creating periodic tasks lists for other activities can assure that you have a very high degree of business continuity in your SAP HANA system. SAP HANA is a highly scalable platform with a substantial amount of built-in fault tolerance at the hardware and software level. Although system failures are rather unusual, no amount of cleverness can totally prevent issues if the use is not monitored regularly and preventive measures are not taken well in advance.
Learning Objectives
Reading this article you will learn:
- Learn what SAP tools are available to monitor your HANA environments and see what hardware options exists
- Get access to all of the 74 automatic HANA alerts and see when to schedule them and what to do when triggered
- Learn how to keep you BW system on HANA as small as possible and how to periodically remove unneeded data and internal files from your HANA system
There are many ways to install and operate your SAP HANA environments on premise. However, creating a daily, weekly, and monthly standard operating procedure (SOP) is a good way to ensure that the system stays welltuned and that potential issues are avoided. This is also known as the daily operations handbook. Iexplain what the different landscape options are and how you can start creating your own SOP for your data center.
The first decision you have to make when setting up your data center for your SAP HANA environments is to decide if you are going to place it onpremise, as part of an outsourcing agreement, or in the cloud. The on-premise approach is currently the most common. It basically means that you will have to integrate the hardware into your current data center and possibly into an off-site data center if you are implementing a high-availability (HA) solution.
A major consideration for the on-premise approach would be to make sure that your hardware fits into your existing chassis, racks, power outlets, cooling plan, and the outlay of your data center. For example, many are not aware that some of the largerSAP HANA systems,such as Lenovo’s x3850 x6, require a four-rack unit (U) height in a data center, but if you useLenovo’s x3950 x6, you need to double that size requirement (since that is basically two stacked 3850s). Other products,such as Cisco’s C880 M4 server,requirea 10U height. Therefore, it is very important to decide what hardware deployment options you are going with. As of September 2015, the most common forms of certified hardware are shown in Figure 1 and described in Table 1.
Figure 1 Common SAP HANA hardware platforms for on-premise deployment
Table 1 Characteristics of the certified SAP HANA hardware options
Customer SAP HANA Admin and Support Responsibilities
As you start with your plan to write an SOP, it is important that you be aware of the normal support, install, and monitoring roles,as well asthe responsibilities of SAP, your hardware vendor, and your own team. The normal support responsibilities can be summarized as shown in Table 2.
Table 2 Summary of key responsibilities
It is important to note that the responsibilities as outlined in Table 2 are based on an on-premise installation of SAP HANA and that no other support agreements are made with the hardware vendor, a cloud vendor, or outsourcing partners. Depending on how you write your support agreement with these vendors, some or all of the customer responsibilities may be assumed by these partners. The trick to making sure of what you are responsible for is to specify these activities in aservice level agreement (SLA) if you are using other vendors to support your systems and landscapes.
There are also different cloud options that some companies might consider. For each of these options the responsibilities of the customer are significantly different. First, you can have your SAP HANA system and applications delivered as a software as a service (SaaS). Under this offering you can get software applications such as SAP Business Suite, SAP Business Warehouse (BW), and SAP Rapid Deployment solutions (RDS) as SaaS SAP HANA cloud solutions from several vendors who then take over all customer responsibilities for daily monitoring, support, and maintenance.
Another option is the platform as a service (PaaS). This is normally provided as a solution in which the database, operating system, connectivity, and hardware are supported by a cloud vendor, but daily operations and monitoring of the application are the customer’s responsibilities. Finally, the lowest level of cloud offerings is known as infrastructure as a service (IaaS). As the name implies, you are normally responsible for all tasks as shownin Table 2, except the hardware maintenance, which is then hosted in the cloud.
However, in this article I assume that the support is for an on-premise implementation;that the customer is assuming the normal support, maintenance, and monitoring roles;and that a cloud solution is not in place.
System versus Landscape Administration
There are several tools and procedures that should be developed that are different based on a system or landscape administration perspective. For example, for system administration you should leverage SAP’s HANA Administration guide that can be downloaded on help.sap.com. guide is maintained and updated by SAP on a release basis. It shows you how to use the SAP HANA cockpit (a Fiori launchpad application) and SAP HANA studio for the main system administration, the core functions of high-availability and disaster recovery, scalability (up and out), and security administration. It also details how to manage and monitor applications for data provisioning and custom applications built in the Extended Services (XS) framework.
You can also monitor the system through the database (DBA) Cockpit in Solution Manager and leverage the Trouble Shooting and Performance Tuning guide from SAP when issues arise. However, from a landscape administration perspective,you leverage the Technical Operational manual from SAP and the DB control center, as well as any respective application support for the systems you might be running. So, when you start developing your SOP or daily operating handbook, you should start by familiarizing yourself with these very important documents and tools and think about system administration as different from landscape administration.Table 3 shows the key SAP resources for SAP HANAsystem and landscape administration.
Table 3 Key administration resources
SAP HANA System Monitoring Tools and Education [header 2]
You can also choose one or more ways to perform your system monitoring. For example, you can monitor system databases and also tenant databases (in MCOD/MCOS) Multiple Components One System – MOCS; and Multiple Components One Database – MCOD)by directly connecting to a database using the SAP HANAcockpit, the DBA Cockpit in Solution Manager, or through regular SAP HANA studio.
In SAP HANA studio in the administration perspective, you get access to most database and system information. There are several tabs that displays landscape, alerts (automatic scheduled monitoring jobs), performance statistics, disk volume information, configuration settings, overall system information, diagnostic files, and configuration of traces and trace files(Figure 2).
Figure 2 The SAP HANA studio Administration Console Perspective
Also, since Support Package Stack 9 of SAP HANA in late 2014, the enhanced SAP HANAcockpit is now a very interesting way to get access to a simple web-based monitoring application that shows you key statuses of your SAP HANA systems and databases.
As mentioned before, the SAP HANAcockpit is basically a Fiori launchpad site that you can also customize to show only the items you are interested in for daily operation monitoring (Figure 3).
Figure 3 Administration with the SAP HANA cockpit in Fiori
The customization of this application is a simple click-and-drag of the tiles (much like on your cell phone). You can also choose the refresh rate of the information in the SAP HANAcockpit. The application can run on a web browser and is therefore mobile and simple to deploy. The SAP HANAcockpit also has a Manage Databases app that allows you to monitor single and multi-tenant databases in SAP HANA.
As you click each of these tiles, a vast array of detail information is provided for your in-depth analysis and system monitoring. However, it is important to note that while the SAP HANAcockpit supports core administration of tenant databases (i.e., MCOS), SAP HANA studio and some command-line tools may still be required for key tasks for tenant databases. Frankly, the only minor drawback with the SAP HANAcockpit is that it may require additional licenses depending on what you bought with the initial license package.
At a higher level, the SAP Database Control Center (DCC) is also a Fiori application that allows you to monitor both SAP HANA and other typesof databases from a central application. As you become more familiar with these tools, you probably will find it useful to start with one or two of these and choose the others as alternatives when you are stuck on a certain task. Most system administrators include SAP HANA studio and either the DBA or the SAP HANA cockpit for daily monitoring.
To start to learn about these tools, first download and study the guides outlined in Table 1. A five-day SAP course called HA-200 “SAP HANA - Installation & Operations” is available for experienced support staff as well as for beginners.
Solution Manager and Landscape Virtualization Management (LVM)
Many of the tools used for system monitoring are also used for database monitoring. First, you can conduct many of the individual database admin functions through SAP HANA studio and the SAP HANA cockpit from a web browser. From there, you can make changes to the database system settings and also add users,privileges, and most standard database admin tasks. Also, just as you can forall SAP software, you can use Solution Manager for core monitoring, admin of multiple systems in your landscape, and as the backbone for Change and Transport System+ (CTS+)integration of transports between the systems in your landscape. Solution Manager can also be used to generate EarlyWatch reports periodically that show growth, use trends, and technical support information. You will also find the DBA Cockpit in Solution Manager(Figure 4). This tool allows you to monitor the SAP HANA database and exposes almost all the technical information you would otherwise find in the administrator console perspective in SAP HANA studio.
Figure 4 SAP HANA Admin and monitoring with the DBA Cockpit in Solution Manager
Solution Manager and the DBA Cockpit also support trace analysis, workload analysis,and exception analysis of SAP HANA databases. Most customers therefore find this tool invaluable when monitoring and managing SAP landscapes with both SAP HANA and other types of databases.
In addition to these tools, the LVM from SAP is also supported for SAP HANA(Figure 5). This toolallows you to conduct core operations of complex landscapes that are based on SAP HANA or non-SAP HANA servers. There is a standard edition of LVM that can be downloaded from SAP for free, and an Enterprise Edition equipped with more features requires a license before you can use it.
Figure 5 The LVM screen
When any of these administration and management tools aredeployed, it is important that your support staff that is monitoring, maintaining, and operating anSAP HANA landscapehave a good understanding of the capabilities of each of these.
Daily Operations SAP HANA Checklist
Afteryou decide on your monitoring tool, download and study the available support documentsin Table 1, and complete any of the other training methods you have selected, such as an SAP class, you are ready to start writing your SOP. The SOP should consist of daily operations, weekly jobs, and periodic upgrades and patches as supplied by SAP. In this section I look at the most common daily operations tasks that you will be doing.
While many prefer to have active or passive monitoring of systems, bestpractices are to have a combination of these. Passive monitoring usually means activating and scheduling some of the alerts available in SAP HANA studio. You can place thresholds on the alerts (i.e., when memory consumed exceeds a certain number of GB), and you can schedule how often the checks are performed on the database.When triggered, these alerts show up in the SAP HANA cockpit, DBA Cockpit, and SAP HANA studio in both detail and overview pages. Today, there are 74 standard alerts that come with the SAP HANA system (Figure 6).
Figure 6 SAP HANA alerts in SAP HANA studio
You can also setup email alerts if you have the system privilege CATALOG READ, the SELECT privilege on the _SYS_STATISTICS schema, and the system privilege INIFILE ADMIN. The first of these privileges is included in the standard SAP HANA role called MONITORING. This role can be assigned to non-system admin users. It allows other technical resources access to see what is happening inside the SAP HANA system without the ability to change anything.
There is also a list of historically executed alerts in SAP HANA studio, but be aware that this list is restricted to the last 1,000 occurrences from the last 30 days.Also, when an alert is triggered, a priority is assigned by the system. In general, there are four different priorities with different timing when action is recommended (Table 4).
Table 4 Alert priorities in SAP HANA
There are also 10 different categories of alerts relating to availability, backup, configuration, CPU, diagnosis files, disk, memory, security, sessions, and system. Deciding when to schedule these alerts and when to monitor them is a critical task of the SAP HANA administrator. By activating and monitoring the recommended daily and intra-day alerts through any of the tools outlined previously, you can detect any performance issues early.
To get started, take a look at Table 5 as the first step of your own tailored daily SAP HANA admin SOP. In this table you find all the available SAP HANA automated alerts, alert IDs (so that you can find them in SAP HANA studio), the suggested frequency when these alerts should be activated or monitored, descriptions, and SAP’s official recommendation on how to resolve any issues.
Check type / ID / Time / Description / SAP recommended admin actionAvaila-bility / 0 / Intra-day / Identifies internal statistics server problem / Resolve the problem. For more information, see the trace files. You may need to activate tracing first.
3 / Intra-day / Identifies inactive services / Investigate why the service is inactive, for example, by checking the service's trace files
4 / Intra-day / Restarted services- services that have restarted since the last time of the check / Investigate why the service had to restart or be restarted, for example, by checking the service's trace files
21 / Daily / Identifies internal DB events / Resolve the event and then mark it as resolved by executing the SQL statement ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>
22 / Intra-day / Notification of all alerts- If any alerts since the last check is triggered / These alerts can trigger email blasts to specified recipients. Investigate the alerts.
23 / Intra-day / Notification of medium- and high- priority alerts- Since the last check is triggered
24 / Intra-day / Notification of high-priority alerts- Since the last check is triggered
31 / Daily / License expiry-If the disks to which data and log files are written are full. A disk-full event causes the DB to stop / Obtain a valid license and install it. For the expiration date, see the monitoring view M_LICENSE.
41 / Daily / In-memory DataStore activation- If a problem with the activation of an in-memory DataStore object exists / For more information, see the table _SYS_STATISTICS.GLOBAL_DEC_EXTRACTOR_STATUS and SAP Note 1665553
70 / Periodic / Consistency of internal system components after system upgrade / Contact SAP support
78 / Daily / Connection between systems in system replication setup- Closed connections between primary or secondary system / If connections are closed, the primary system is no longer being replicated. Investigate why connections are closed (i.e., network problem) and resolve the issue.
80 / As needed / Availability of asynchronous table replication- Monitors error messages related to async table replication / Determine which tables encountered the table replication error using system view M_ASYNCHRONOUS_TABLE_REPLICAS, and check the corresponding indexserver alert traces
Back-up / 28 / Periodic / Most recent savepoint operation- How long ago the last savepoint was defined; that is, how long ago a complete, consistent image of the DB was persisted to disk / Investigate why there was a delay defining the last savepoint and consider triggering the operation manually by executing the SQL statement ALTER SYSTEM SAVEPOINT