BMC Performance Manager for WebSphere BusinessIntegration
Basic Best Practices
By Ross Cochran
Table of Contents
Purpose of paper
Product architecture
MMA
BMC Performance Manager for WebSphere MQ for z/OS and OS/390 Pre-installation Activities
Rule creation and debugging
Rules - Schedules
Rules – Menu command passwords
Rules – Queue Manager
Rules - Queue
Rules – Channel
Rules – Event Presentation
Utilities
Utilities – Environment Probe
Utilities – FFST Dumps
Utilities - Debug
Utilities – Set Message Destination
Utilities – Changing z/OS User Ids and Passwords
Utilities – MQ error logs
Utilities – Locate queue command
Utilities – Queue Activity Report
Utilities – Channel Listeners and Channel Initiators
Distributed Systems - Channel Listeners and Channel Initiators
z/OS - Channel Listeners and Channel Initiators
Utilities - Queues
Utilities – MQ Events
Event-based vs. Threshold-based Management
Utilities – Message Transit Monitor
BMC Extensions
Message management
MQ log management
Purpose of paper
This paper covers the best practices BMC Performance Manager for WebSphere Business Integration. The paper assumes users understand WebSphere MQ, BMC Performance Manager (aka PATROL), and BMC Performance Manager for WebSphere Business Integration(aka the MQ KM). It is not a:
Install guide
Complete “how-to” guide
Debug guide
User Guide
Product architecture
MMA
The BMC Performance Manager for WebSphere Business Integration application management suite for D/S is based on the BMC Middleware Management Architecture (MMA). MMA code enables the PATROL Agent to communicate with WebSphere MQ. MMA code running on a PATROL-managed host can also communicate with MMA code running on a non PATROL-managed host (such as z/OS). The MMA code is combined within the MMA_base.kml. The BMC Performance Manager for WebSphere Business Integration, which passes requests to MMA from the Agent and Console, is grouped under the AMQ_base.kml.
MMA consists of several components, the most important of which are
•MMA Client
•MMA Server
•MMA Com Manager
The MMA Client handles the requests from the PATROL Agent or Console. The MMA Client sends commands to the MMA Server if the request is for a local queue manager or to the MMA Com Manager if the request is for a remote queue manager. A default number of clients are active (4 in the case of Windows). However, the user can set the minimum and maximum number of active clients. The MMA Server takes requests from the MMA Client and issues commands to WebSphere MQ, and then passes the response to the request back to an MMA Client. The MMA Com Managersends requests destined for remote queue managers to another MMA Com Manager on a non-PATROL Agent host (such as z/OS).
Tip – MMA is a self-managing architecture. In general users will never know it is present. If MMA appears to be causing a problem in the MQ network, please contact BMC Support before attempting to debug an MMA issue.
Two KML’s are requiredfor distributed systems:
- MMA_BASE.kml
- AMQ_BASE.kml
Two KML’s are optional depending on functions:
- MLP_BASE.kml
- MQSI.kml.
MLP_Base.kml is only required if users desire MQ log (circular or linear) management.
The MQSI.kml is only required if users want to manage an MQ broker.
WebSphere MQ events are processed by an event listener. When WebSphere MQ events are generated the event listener process passes the event to the MMA Server, which in turn passes information back to an MMA Client and then back to the PATROL Agent. After the Agent updates the parameter, it will signal the PATROL Console that the event has occurred.
The event listener process is a product component that is automatically started. This threaded (multi-process) application keeps all event queues open exclusively for input, immediately removes any MQ event messages that appear, and propagates events to one or more publish queues. The publish queue selection is determined by subscription requests from one or more subscribing programs. To subscribe to events, any program can put a message to the event listener’s command queue (BMC.LISTENER.COM by default) with the name of the desired publish queue.
The following example is a sequence of commands that establishes PUB.QUEUE as a publish queue on queue manager QM1:
amqsput BMC.LISTENER.COM QM1
subscribe A QM1 PUB.QUEUE
The event listener maintains a list of publish queues on its subscription queue, BMC.LISTENER.SUB by default.
Tip – The BMC event listener can not get MQ events unless users enable events on the applicable MQ objects
Tip – Do not try to start or stop the BMC event listener manually.
Tip – The BMC Event Listener can not open the MQ event queues if they are all ready exclusively opened by another application.
On z/OS the user must start the MMA Server and the MMA Communications Manager manually. The MMA Server (MMASRV by default name) issues commands to, and accepts replies from, WebSphere MQ queue managers. It receives its requests through the MMA Communications Manager.
The MMA Communications Manager (MMACOM by default name) handles the communications between the MMA Server on the z/OS host and an MMA Communications Manager on a distributed systems host that is running a PATROL Agent with BMC Performance Manager for WebSphere Business Integration. The MMA Communications Manager uses TCP/IP for communications.
On z/OS the Event Listener started task (MMAELSR) and the Message Transit Monitor started task (AMQMTMS) are started automatically.
Tip – Users can safely rename the MMASRV and MMACOM started tasks.
Tip – Do not rename the MMAELSR and AMQMTMS started tasks.
Tip – Start MMASRV before MMACOM. Shutdown MMACOM before MMASRV.
When adding a z/OS host from the PATROL Console, the LISTENER queue prefix applies to the 2 queues that are created on each discovered Queue Manager. These are the COM and SUB queues that BMC Performance Manager for WebSphere Business Integration uses as a mechanism to subscribe to published MQ events. The publish queue will also be created on each queue manager. Its name is derived from the product code (AMQ), the host name (identifies the host from which this add host operation is originating) and the port number used for PATROL Agent communication on this host.
WARNING: Any hostname containing a dash (‘-’) is very problematic. The dash becomes part the queue name and is passed to a WebSphere MQ ‘DEFINE QLOCAL’ statement. Because dash is an illegal character in WebSphere MQ, the DEFINE QLOCAL statement will fail.
The DNS name or IP address of the host that users are adding is required, as is the port number to be used by the MMA Communications manager.
Tip – The PATROL Agent connecting to the z/OS system needs to be able to resolve the z/OS DNS name. The z/OS system must be able to resolve the PATROL Agent system name. Please also verify the firewall(s) around the z/OS system is open for the selected IP port number.
For successful interaction between BMC Performance Manager for WebSphere Business Integration and z/OS, two z/OS user IDs and passwords are required. The first user ID is used by the MMA Server to authenticate itself into z/OS. Users should consider setting up a special z/OS user ID for this purpose. Because only the MMA Server makes use of this ID (no real human will ever login with it), users should consider making this a “no logon” user ID that only the MMASERVER started tasks can use. If the password for the userid expires, then the PATROL distributed systems agentwill not be able to manage the z/OS queue managers. Thus, in addition to making this a no login ID, users should consider assigning it a non-expiring password. The second user ID pertains to the person issuing z/OS MQ commands from the PATROL console. This is a standard TSO user ID that needs at least full MQ authority (access to the six RACF MQ classes).
There are also two user IDs needed for PATROL operations.
The first is the PATROL Console ID, or the ID of the PATROL user. The second ID pertains only to the PATROL Agent. Authentication requirements are thoroughly described in both the information pertaining to the types of consoles available to the user, and in the PATROL Agent documentation.
In addition to privileges and permissions that all PATROL users must have, the PATROL Console and Agent also need the following authorizations:
full authorization by the WebSphere MQ Object Authority Manager (OAM)
authorized as a member of the MQM group
Tip -If the only WebSphere MQ activity is being monitored by a PATROL Agent is activity that is being processed on z/OS—that is, if the PATROL Agent is not monitoring any local WebSphere MQ activity—it does not need to be authorized by WebSphere MQ OAM or be a member of the MQM group.
BMC Performance Manager for WebSphere MQ for z/OS and OS/390 Pre-installation Activities
Often, certain items require a “long” lead time on z/OS systems. Here is a summary of those items.
Secure 100 Cylinders of DASD.
Secure a copy of product license key from the BMC Rep. The product code is WMZ.
Two TSO user IDs are required with authority of all 6 MQ security classes (listed below). It is suggested one of the two IDscan not logon interactively, can only execute in batch, and can only be used by the MMASRV started task. It is also suggested the same ID have a password that does notexpire. The security classes are:
MQCONN MQCMDS MQQUEUE MQPROC MQNLIST MQADMIN
The following started tasks need the same RACF access as noted in the previous step. They should run continuously. They should be started in the order listed and shutdown in the reverse order listed.
- MMASRV
- MMACOM
- MCMNM (Only required if want MQ Administration)
- MMAELSR (Started automatically by BMC)
- AMQMTMS (Started automatically by BMC)
The following datasets need APF authorization:
HLQ.SCSQLOAD
HLQ.SCSQAUTH (Probably already authorized).
HLQ.BBLINK
HLQ.PGMLIB
HLQ.SCEERUN
HLQ.BMCPSWD
The queue manager JCL must be modified to specify the use of BMC Extensions (The BMC Extensions are an optional feature discussed later in this paper).
A dedicated TCP/IP port is required. MMACOM listens on port 1987 (Users can change it). Verify all applicable firewalls have been addressed.
If MQ events are desired (and not currently enabled), verify amble log space is available to accommodate WebSphere MQ event entries.
The ‘SYSTEM.ADMIN.COMMAND.QUEUE’ needs to be defined and needs the MQ attributes of SHAREABLE (versus NOSHARE) and DEFSOPT needs to be SHARED (verses EXCL).
Rule creation and debugging
The BMC Performance Manager for WebSphere Business Integrationproduct takes action through the use of rules. Without rules, the product monitors but does not take action. Think of rules as the components that make things happen.
Tip – No rules are enabled out-of-the-box.
Tip – These rules are independent of any PATROL recovery actions or PATROL Action Methods.
The rules are accessed from the WebSphere MQ icon. Please make a note of this as users will forget where the rules are located. The rules are specific to BMC Performance Manager for WebSphere Business Integration, in other words, the recovery actions provided by core PATROL functionality are completely independent of actions invoked via these rules. For example, the actions invoked by the BMC Performance Manager for WebSphere Business Integration rules are unknown to the team supporting core PATROL.
Rules - Schedules
A schedule is a rule that creates a time frame to which other rule types can refer. It is best to setup the schedules before creating other rules even though users can create a schedule when users create a new rule. Rules can be based on GMT times or local PATROL Agent times.
Tip - It is best to use local PATROL Agent times (especially in a crisis situation when users forget to figure in the GMT offset) when setting up a schedule.
Rules – Menu command passwords
All menu commands from the GUI can be password protected. This means all the commands are displayed on each user’s PATROL Console, however each command can be password protected before it is invoked. The first step is to set passwords for each level. Then assign a level to each password-protected command. Be careful because there is no confirmation field into which users retype passwords. The password must be right the first time. Also record the passwords in a safe location. There is no facility for viewing or retrieving passwords.
Tip - Creating a Menu Command Authority Profile has no effect unless users reference it in a queue manager rule. This is a common error users make with the feature. They write a Menu Command Authority Rule, but forget to assign the associated Profile in the queue manager rules.
NOTE: Menu Command Authority has no connection to the underlying MQ security layer (OAM, RACF, ACF2, and Top Secret). Correctly implemented, this feature simply “protects” the BMC Performance Manager for WebSphere Business Integration command from unauthorized use.
Rules – Queue Manager
There is a general rule of thumb regarding rules. Write thefirst rules generically, in other words do not tightly define the criteria for rules by using very definitive host names and object names. By using very definitive host names and object names users may never get a rule to execute.
Tip - Start out with host names such as “*”. Once rules are working, start to tighten them down with more specific object and host names.
The three situations inwhich a queue manager rule is most useful are:
- Restart a queue manager
- Allow The BMC Performance Manager for WebSphere Business Integration to manage more than 100 local queues
- Restart the channel initiator
An obscureproduct feature allows PATROL to send events to z/OS as Write to Operator (WTO) messages. Depending on the queue manager rule, the WTO messages can be for distributed queue managers, z/OS queue managers, or both. Get with the z/OS System Programmer on this part. Unless users have training in z/OS, the information in these panels will be meaningless to users. The essential thing to know is that the WTO messages need a z/OS priority and a z/OS console ID.
Tip – Users have used the WTO feature to enable D/S queue manager management from z/OS.
Rules - Queue
The main reason for queue rules is to determine where (under which icon) a queue or group of queues will appear. If the queue names follow some type of consistent naming convention (it does not have to be a good one, just a consistent one) then the queue rules will work super.
Rules – Channel
The two primary reasons for writing a channel rule are to:
- Automatically restart channels
- Depict an MQ channel event on the PATROL console
There is only one way to automatically restart a channel. The rules do not allow restarting a channel based on channel status, but only restarting based on the MQ event(s) of the associated transmission queue.
Tip– When the channel is automatically restarted by a channel rule, the channel restart is attempted one time, and one time only.
Rules – Event Presentation
This menu command allows users to specify the way in which BMC Performance Manager for WebSphere Business Integration events are presented. There are several settings in regard to Event Presentation Rules. However the most-used setting is to generate SNMP Traps. Specify yes to generate an SNMP trap for a PATROL MQ_SNMP event. If users do not do specify yes, regardless of the core PATROL Agent settings, users will not get MQ related events.
Utilities
Utilities – Environment Probe
A powerful and little used utility is the PATROL for WebSphere MQ Environment Probe. The PATROL for WebSphere MQ Environment Probe is similar to the PATROL Environment Probe in that it is very useful for tracking down possible causes of problems that might be more difficult to find by simply looking at “About” information. Because it runs from the Windows Start Menu, this utility can also be run at times when BMC Performance Manager for WebSphere Business Integration for some reason cannot load.
The utility returns a screen that is scrollable both side-to-side and top-to-bottom and that contains good deal of information that is potentially very useful in tracking down environmental problems:
Install directory
Windows System directory
Operating System
Drive(s) available
Disk space
Processor type
OS version
PATROL_HOME
WebSphere MQ Version release type PTF CSD
BMC Performance Manager for WebSphere Business Integration version
MQ KM Versions AMQ MQI MMA MQX QSTATS MLP MCM QMM
Tip – The first time users install the product – Get a screen capture of the PATROL for MQ Environment Probe.
Utilities – FFST Dumps
On distributed systems, First-Failure Support Technology (FFST) information is recorded in files in the MQ error directory. These files contain information about unexpected errors, often severe and unrecoverable. Details include such information as the function stack at time of error. Generally, the information is meaningful only for purposes of reporting a problem to IBM support. Many times these dumps are referred to as the .FDC dumps. Some systems may produce hundreds of dumps; some systems produce none. Although it would mean using excessive disk space, users may keep hundreds. However, it is recommended users only keep a few weeks’ worth. Use the Delete Based on Age button to specify how long to keep dumps.