DRAFT
Daily Operations and Administration of Your HANA environments
By Dr. Bjarne Berg, VP SAP Business Intelligence, Comerit and
Professor at SAP University Alliance at Lenoir Rhyne University
There are many ways to install and operate your HANA environments on premise. However, a creating an daily, weekly and monthly Standard Operating Procedure (SOP) is a good way to make sure that the system stays well-tuned, and that potential issues are avoided. This is also known as the daily operations handbook. In this article we look at what the different landscape options are and how you can start creating your own SOP for your data center.
The first decision you have to make when setting up your data center for your HANA environments is to decide if you are going to place it on-premise, as part of an outsourcing agreement, or on the cloud. The on premise approach is currently most common and it basically means that you will have to integrate the hardware into you current data center and possibly an off-site data center if you are implementing a high-availability (HA) solution.
A major consideration for the on-premise approach would be to make sure that your hardware fits into your existing chassis, racks, power outlets, cooling plan and the outlay of your data center. For example, many are not aware that some of the larger HANA systems like for example Lenovo’s x3850 x6 requires a 4U height in a data center, but if you are using the Lenovo’s x3950 x6, you will need to double that size requirement (since that is basically two stacked 3850s). While other vendors such as Cisco’s C880 M4 Server requires 10U height. So, it is very important to decide what hardware deployment options you are going with. As of September 2015, the most common forms of certified hardware can be summarized as:
Figure 1: Common HANA Hardware Platforms for On-Premise Deployment
Table 1: Certified HANA Hardware options as of September 2015.
Customer HANA Admin and Support Responsibilities
As you start with your plan to write an SOP, it is important that you are aware of the normal support, install, monitoring roles and the responsibilities of SAP, your hardware vendor and your own team. The normal support responsibilities can be summarized as in Table 2 below.
Table 2: Summary of Key Responsibilities
It is important to note that the responsibilities as outlined in Table 3, is based on an on-premise installation of SAP HANA and that no other support agreements are made with the hardware vendor, a cloud vendor, or an outsourcing partners. Depending on how you write your support agreement with these vendors, some or all of the customer responsibilities may be assumed by these partners. The trick to make sure on what you are responsible for is to specify these activities in a Service Level Agreement (SLA) if you are using other vendors to support your systems and landscapes.
There are also different cloud options that some companies might consider. For each or these options the responsibilities of the customer are significantly different. First, you can have your HANA system and applications delivered as a “Software as a Service” (SaaS). Under this offering you can get software applications such as SAP Business Suite, SAP Business Warehouse (BW), and SAP Rapid Deployment solutions (RDS) as SaaS HANA cloud solutions from several vendors who then take over all customer responsibilities for daily monitoring, support and maintenance.
Another option is the “Platform as a Service” (PaaS). This is normally provided as a solution where the database, operating system, connectivity and hardware is supported by a cloud vendor, but where daily operations and monitoring of the application is the customer’s responsibilities. Finally, the lowest level of cloud offerings is known as “Infrastructure as a Service: (IaaS). As the name implies, you are normally responsible for all tasks as illustrated in Table 3, except the hardware maintenance which is then hosted in the cloud.
However, in this article we are going to assume that the support is for an on-premise implementation and that the customer is assuming the normal support, maintenance and monitoring roles and that a cloud solution is not in place.
System vs. Landscape Administration
There are several different tools and procedures that should be developed that are different based on a system or landscape administration perspective. For example, for system administration you should leverage SAP’s HANA Administration guide that can be downloaded on help.sap.com. This guide is maintained and updated by SAP on a release basis. It shows you how to use the HANA cockpit (a Fiori LaunchPad application) and HANA studio for the main system administration, the core functions of high-availability and disaster recovery, scalability (up and out), security administration, as well as how to manage and monitor applications for data provisioning and custom applications built in the extended services (XS) framework.
We can also monitor the system through the DBA Cockpit in Solution Manager and leverage the Trouble Shooting and Performance Tuning guide from SAP when issues arise. However, from a landscape administration perspective we leverage the Technical Operational manual from SAP and the DB control center, as well as any respective application support for the systems you might be running. So, when you start developing your SOP/Daily Operating handbook, you should start by familiarize yourself with these very important documents and tools and think about system admin as different from landscape administration.
Table 3: Key SAP resources for HANA System and Landscape Administration
HANA System Monitoring tools and Education
You can also choose one or more ways to perform you system monitoring. For example, you can monitor system databases and also tenant data bases (in MCOD/MCOS) by directly connecting to a database using the HANA cockpit, the DBA Cockpit in Solution Manager, or through regular SAP HANA studio.
In HANA Studio in the administration perspective, you get access to most database and system information. There are several tabs that displays landscape, alerts (automatic scheduled monitoring jobs), performance statistics, disk volume information, configuration settings, overall system information, diagnostic files and configuration of traces and trace files.
Figure 2: The SAP HANA Studio Administration Console Perspective
Also, since SPS9 of HANA in late 2014, the enhanced HANA Cockpit is a now very interesting way to get access to a simple web based monitoring application that shows you key statuses of your HANA systems and databases. As mentioned before, the HANA Cockpit is basically a Fiori LaunchPad site that you can also customize to show only the items you are interested in for daily operation monitoring.
Figure 3: Administration with HANA Cockpit in Fiori
The customization of this application is a simple click-and drag of the tiles (much like on you cell phone). You can also choose the refresh rate of the information in HANA Cockpit and the application can run on a web browser and is therefore mobile and simple to deploy. The HANA Cockpit also have a ‘Manage Databases’ app that allows you to monitor single and multi-tenant databases in HANA.
As you click on each of these tiles, a vast array of detail information is provided for your in-depth analysis and system monitoring. However, it is important to note that while the HANA Cockpit supports core administration of tenant databases(i.e. MCOS), HANA Studio and some command-line tools may still be required for key tasks for tenant databases. Frankly, the only minor drawback with the HANA Cockpit is that it may require additional licenses depending on what you bought with the initial license package.
At a higher level the SAP Database Control Center (DCC) is also a Fiori application that allows you to monitor both HANA and other type databases from a central application. As you get more familiar with these tools, you probably will find it useful to start with one or two of these as choose the others as alternatives when you get stuck on a certain task. Most system administrators include HANA studio and either the DBA or the HANA cockpit for daily monitoring.
To get started to learn about these tools, first download and study the guides outlined in Table 1 and thentake the 5-day SAP Course called HA-200 “SAP HANA - Installation & Operations”. This course is strongly recommended for experienced support staff as well as for beginners and should be taken before you start writing your own SOP.
Solution Manager and LVM Tools
Many of the tools used for system monitoring is also used for database monitoring. First, you can conduct many of the individual database admin functions through HANA studio and the HANA cockpit from a Web browser. From here you can make changes to the database system settings and also add users and privileges and most standard database admin tasks.Also, like all SAP software, you can use Solution Manager (SolMan) for core monitoring and admin of multiple systems in your landscape and as the backbone for CTS+ integration of transports between the systems in your landscape. Solution Manager can also be used to generate EarlyWatch reports on a periodical basis that shows growth, usage trends and technical support information. You will also find the DBA Cockpit in Solution Manager. This tool allows you to monitor the HANA database and exposes almost all of the technical information you would otherwise find in the administrator console perspective in HANA Studio.
Figure 4: HANA Admin and monitoring with the DBA Cockpitin Solution Manager
Solution Manager and the DBA Cockpit also support trace analysis, workload analysis and exception analysis of HANA databases. Most customers therefore find this tool invaluable when monitoring and managing SAP landscapes with both HANA and other types of databases.
In addition to these tools the Landscape and Virtualization Manager (LVM) from SAP is also supported for HANA. This allows you to conduct core operation of complex landscapes that may have HANA and non-HANA based servers. There is a standard edition of LVM that can be downloaded from SAP for free, and there is an Enterprise Edition that has more features that require a license before you can use it.
Figure 5: SAP Landscape and Virtualization Manager (LVM)
When any of these administration and management tools isdeployed, it is important that your support staff, that is monitoring, maintaining and operating a HANA landscape, has a good understanding of the capabilities of each of these. This can be obtained through regular SAP training classes. If you are new to these tools, you might start with one or two, and integrate them into your support landscape and add more as your experience level increase.
Daily Operations HANA Checklist
Once you have decided on you monitoring tool, downloaded and studied the available support documents in Table 1 and completed the 5-day HA-200 HANA - Installation & Operationsclass from SAP, you are ready to start writing your Standard Operating Procedure. The SOP should consist of daily operations, weekly jobs and periodic upgrades and patches as supplied by SAP. In this section we take a look at the most common daily operations tasks that you will be doing.
While many prefer to have active or ‘passive’ monitoring of systems, best-practices are to have a combination of these. Passive monitoring usually means activating and scheduling some of the alerts available in HANA Studio. You can place thresholds on the alerts (i.e. when memory consumed exceeds a certain number of GB), and you can schedule how often the checks are performed on the database. These alerts when triggered show up in the HANA cockpit, DBA Cockpit and in HANA studio in both detail and overview pages. Today there are 74 standard alerts that come with the HANA system.
Figure 6: SAP HANA Alerts in HANA studio
You can also setup email alerts if you have the system privilege CATALOG READ as well as SELECT privilege on the _SYS_STATISTICS schema, and also the system privilegeINIFILE ADMIN. The first of these privileges is included in the standard SAP HANA role called MONITORING. This role can be assigned to non-system admin users and allow other technical resources access to see what is happening inside the HANA system without the ability to change anything.
There is also a list of historically executed alerts in HANA studio, but be aware that this list is restricted to the last 1,000 occurrences from the last 30 days.Also, when an alert is triggered, a priority is assigned by the system. In general, there are 4 different priorities with different timing when action is recommended:
Table 4: Alert Priorities in SAP HANA
There are also 10 different categories of alerts relating to Availability, Backup, Configuration, CPU, Diagnosis Files, Disk, Memory, Security, Sessions and System. So, when deciding when to schedule these alerts and when to monitor them is a critical task of the HANA administrator. Activating and monitoring the recommended daily, and intra-day, alerts through any of the tools outlined previously will allow you to early detect any performance issues.
To get started, take a look at table 5 as the first step of your own tailored daily HANA admin SOP. In this table you will find all of the available HANA automated alerts, alert IDs (so that you can find them in HANA Studio), the suggested frequency when these alerts should be activated/monitored, descriptions and SAP’s official recommendation on how to resolve any issues.
Table 5: Admin Monitoring Frequency, Alert IDs and SAP recommended actions
Check Type / ID / Time / Description / SAP Recommended Admin ActionAvaila-bility / 0 / Intra-day / Identifies internal statistics server problem. / Resolve the problem. For more information, see the trace files. You may need to activate tracing first.
3 / Intra-day / Identifies inactive services. / Investigate why the service is inactive, for example, by checking the service's trace files.
4 / Intra-day / Restarted Services- services that have restarted since the last time of the check. / Investigate why the service had to restart or be restarted, for example, by checking service's trace files.
21 / Daily / Identifies internal DB events. / Resolve the event and then mark it as resolved by executing the SQL statement ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>.
22 / Intra-day / Notification of all alerts- if any alerts since the last check is triggered / These alerts can trigger email blasts to specified recipients. Investigate the alerts.
23 / Intra-day / Notification of medium and high priority alerts- since the last check is triggered
24 / Intra-day / Notification of high priority alerts- since the last check is triggered
31 / Daily / License expiry-If the disks to which data and log files are written are full. A disk-full event causes DB to stop / Obtain a valid license and install it. For the expiration date, see the monitoring view M_LICENSE.
41 / Daily / In-memory DataStore activation- If a problem with the activation of an in-memory DataStore object exists / For more information, see the table _SYS_STATISTICS.GLOBAL_DEC_EXTRACTOR_STATUS and SAP Note 1665553.
70 / Periodic / Consistency of internal system components after system upgrade / Contact SAP support.
78 / Daily / Connection between systems in system replication setup- closed connections between primary/ secondary system. / If connections are closed, the primary system is no longer being replicated. Investigate why connections are closed (i.e., network problem) and resolve the issue.
80 / As needed / Availability of asynchronous table replication- Monitors error messages related to asynch table replication. / Determine which tables encountered the table replication error using system view M_ASYNCHRONOUS_TABLE_REPLICAS, and check the corresponding indexserver alert traces.
Back-up / 28 / Periodic / Most recent savepoint operation- How long ago the last savepoint was defined, that is, how long ago a complete, consistent image of the DB was persisted to disk. / Investigate why there was a delay defining the last savepoint and consider triggering the operation manually by executing the SQL statement ALTER SYSTEM SAVEPOINT.
32 / Periodic / Log mode LEGACY- If the DB is running in log mode "legacy". Log mode "legacy" does not support point-in-recovery and is not recommended for productive systems. / If you need point-in-time recovery, reconfigure the log mode of your system to "normal". In the "persistence" section of the global.ini configuration file, set the parameter "log_mode" to "normal" for the System layer. When you change the log mode, you must restart the DB system to activate the changes. It is also recommended that you perform a full data backup.
33 / Periodic / Log mode OVERWRITE- If the DB is running in log mode "overwrite". Log mode "overwrite" does not support point-in-recovery (only recovery to data backup) and is not recommended for prod systems.
35 / Daily / Existence of data backup / Perform a data backup as soon as possible.
36 / Daily / Status of most recent data backup / Investigate why failed, resolve the problem, and perform a new data backup as soon as possible.
37 / Daily / Age of most recent successful data backup / Perform a data backup as soon as possible.