Managing MQSI Dev Environments

MQSeries Integrator Version 2:

Managing MQSI Development Environments

Robert J. Knaus, AIM Architect

Department 5MPA, Austin, Texas

April 24, 2001

First Edition, April 2001

This edition applies to Version 1.0 of Managing MQSI Development Environments and to all

subsequent releases and modifications unless otherwise indicated in new editions.

© Copyright International Business Machines Corporation 2001. All rights reserved. Note to US Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule contract with IBM Corp.

Notices

The following paragraph does not apply in any country where such provisions are inconsistent with local law.

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore this statement may not apply to you. References in this publication to IBM products, programs, or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM licensed program or other IBM product in this publication is not intended to state or imply that only IBM's program or other product may be used. Any functionally equivalent program that does not infringe any of the intellectual property rights may be used instead of the IBM product. Evaluation and verification of operation in conjunction with other products, except those expressly designated by IBM, is the user's responsibility.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, 500 Columbus Avenue, Thornwood, New York 10594, USA.

The information contained in this document has not be submitted to any formal IBM test and is distributed AS-IS. The use of the information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item has been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.

Trademarks and service marks

The following terms, used in this publication, are trademarks of the IBM Corporation in the United

States or other countries or both:

  • IBM
  • MQSeries
  • MQSeries Integrator
  • MQSI
  • AIX
  • DB2
  • System 390

The following terms are trademarks of other companies:

  • NEON New Era of Networks, Inc
  • Sun Solaris
  • HP/UX
  • Oracle
  • Sybase
  • SQLServer
  • WindowsNT
  • Windows 2000

Introduction

An enterprise with multiple development projects and a centralized integration point needs to specify how to control the development of message flows that execute on shared MQSI brokers. If there is no coordination of development and deployment, then errors may occur in the MQSI broker due to conflicts in MQ queue names, message set definitions, message flow definitions and topology.

This paper recommends that MQSI domains be created for each development organization, a separate domain be created to do Integration Test, and one or more domains be created for Production. An MQSI domain contains one Configuration Manager and one or more Brokers. A code management system should be used to manage the versions of the files exported between MQSI Configuration Managers. The tasks to do this are explained in this paper.

Related Information

There are related MQSeries Integrator supportpacs found at

IC01 – Message Flow Versioning Utilities is referenced in this document.

IC04 – Change Management and Naming Standards Examples offers guidelines on developing with MQSI.

Overview of MQSeries Integrator

In an MQSI V2 Domain, there are two types of servers involved - the Configuration Manager (CM) and one or more Brokers (BK). The Configuration Manager manages the development environment and uses the configuration repository and the message repository. Each developer uses the Control Center GUI on their own NT System for creating MQSIV2 message flows. If necessary, each developer uses the NEON Rules GUI and NEON Formatter GUI for developing NEON message flows. The Control Center communicates directly with the Configuration Manager Server as a Java MQSeries Client application. The NEON GUIs read and write information into a NEON database using a database client. NEON GUIs do not interact with the Configuration Manager.

During a Deploy operation, the MQSI Configuration Manager server uses MQSeries with specific queue names to send the runtime environment information in an XML/MQSeries message to the Broker, where it is stored in the Broker Database. NEON rules and formats do not require a deploy operation since the database is shared with the broker.


Figure 1: MQSI Development Basic Configuration

In order to provide installation flexibility, MQSI allows the distribution of its components on different systems and databases, with some restrictions. As shown in figure 1 above, the user interfaces run as clients on a WindowsNT (and Windows2000) systems. The Configuration Manager also runs on WindowsNT (and 2000). The databases may be distributed in the network. The Configuration Repository and Message Repository must be DB2 databases.

The MQSIV2 Broker is available on WindowsNT (and 2000), Sun Solaris, and AIX. IBM has stated that the Broker will be available on HP/UX and on System 390.

The NEON Database may be on DB2, Oracle, Sybase and SQLServer. The Broker Database may be on DB2, Oracle, Sybase and SQLServer.

When the Control Center deploys the configuration, the Configuration Manager sends MQSeries messages to each Broker. These messages contain information that is used to update the Broker Database. These messages contain XML although they are not documented and do not represent an interface to the Broker.

Prototype Development Process

A typical development process would be:

  • The Message Administrator imports message definitions from COBOL Copybooks or C structures into the message repository (MRM) or defines messages using the CM GUI.
  • Developers use the message definitions to build message flows. Message flows are checked out first (to prevent collisions) and worked on at the developer’s workstation. When finalized, the message flow is checked back in and deployed into Development’s MQSI Broker.
  • Developers use standard MQ tools (AMQSGET/ PUT) to generate messages for testing the flows. The Control Center may be used to start and stop traces on message flows and execution groups. Trace nodes may be added to message flows during testing.
  • Developers of NEON Rules and Formats may use the NEON Visual Tester as well as NEON commands to validate rules and formats.
  • Tools and utilities that are useful for testing may be found at These “Supportpacs” should be reviewed: MA0J, MA7A, IH01, IH02 and IH03.
  • When ready, the flows are promoted to an integration test broker, where testers can perform the system and integration tests, i.e. verify that the messages are properly distributed to all subscribers. MQ loop-back or real data feeds can be used depending on the project. Final tests are run against the real targeted applications.
  • The Integration Test environment can be set up to be a mirror copy of Production, and it is from this environment that flows are exported and promoted to Production.
  • Depending on the test type, some specialized tools might be used to speed up the testing phase (e.g. stress test and message generators).

Managing Brokers

MQSIV2 is designed so that the Control Center manages the Brokers. The MQSIV2 domain consists of the Configuration Manager and its databases and one or more Brokers and its (their) databases. The Brokers may have the same topology or be different. An enterprise would have at least one domain used for development. It would also have at least one domain used for integration test and production. It could also choose to separate the integration test and production into separate domains.

Figure 2:Development to Integration Test to Production depicts an environment with multiple development domains and a single domain used for integration test and production. Within the integration test and production domain there are multiple brokers used for testing and multiple brokers used for production.

This configuration might be suitable when new and changed message flows must be put into production quickly or in relatively small installations with a single test system and one or two brokers.

The Development Configuration Manager’s repositories are exported to the Integration Control System’s repositories. As the information is imported, the test team to insure it conforms to standards and naming conventions should review it. The integration and production level tests are then performed. Once test criteria are met, the Integration Control System’s Control Center deploys the topology to the Production Broker(s). The parameters and options that govern the broker’s performance are set as part of the deploy step.

Figure 2: Development to Integration Test to Production

Separate Domains for Integration Test and Production

The configuration shown in Figure 2 may be extended as shown in Figure 3:Separate Test and Production below. The export/import process is used to send the information from the Integration Control System to the Production Configuration Manager. The Production Control Center is then used to deploy the runtime information to the production MQSI brokers.

Figure 3: Separate Test and Production


Export/Import

There are three similar but slightly different functions to export the Configuration Repository, the Message Repository, and the NEON Formats and Rules. These operations are done separately.

  • The Configuration Manager GUI is used to export and import the Configuration Repository, that is, the message flow and configuration data. The file produced is in XML format.
  • The mqsimrmimpexp command is used to export the data in the message repository. In the command parameters, the -e flag indicates export and the name of one or more message sets to be exported are specified. The same command with the -i flag is used to import messageset information. The file produced by the export is in XML format.
  • The command NNRie is used to export and import NEON rule definitions. It produces a readable text file that is not XML.

Remember that in MQSI exporting and importing is used to create a separate copy of the data for use by another domain. The deploy operation transmits configuration information to Brokers in the same domain

The Control Center is used to deploy information to MQSI Brokers. The menu entry File -> Deploy is used. This deploys information to all of the brokers. It can transmit a complete replacement configuration or a delta configuration.

NEON Rules and Formats do not use a deployment process. The rules and formats reside in a database that must be available to the Broker. When rules or formats are created/changed by the GUI, the NEON rules are changed on the database directly. There are line mode commands with good granularity to export and import individual NEON objects, so this can be automated.

When using NEON rules with MQSIV2, there are two separate processes that must be related:

  1. The export/import of NEON data from the Development domain to the Integration Test domain and then to the Production domain, and
  2. The deploy operations from the Configuration Manager to each of the Brokers.

The recommended sequence is to export/import the NEON objects first, and then deploy the MQSIV2 configuration.

There is an MQSI supportpac, MQSI V2 Message Versioning Utilities, which should be reviewed for Systems Management. The Supportpac number is IC01. The URL is The readme file from this support pack is included below in the section Versioning Utilities Supportpac.

Versioning

Versioning is not automatic at the present time (MQSI V2.01). Customers can use the versioning utilities in the MQSI V2 Message Versioning Utilities supportpac. As a suggested best practice, whenever a message flow is changed, first right click ---> duplicate on the message flow, then rename it with a version number appended as part of the name. That way, if there is a problem, it is easy to deploy the previous version to the Broker to restore it. If this is not done, the metadata for the message flow is overwritten by the updated message flow. To restore it, an administrator would need to re-import a backup copy of the repository.

Figure 4: Controlling Versions


Replacing Message Flows at Runtime

Using the Control Center, an operator or administrator can control the operation of each broker including deploying changes to it while it is running. In MQSIV2, deploy operations happen in real-time – there is no need to stop the Broker. If a message flow with the same name is re-deployed, the Broker will dynamically quiesce the old message flow, replace its definition and then start using the new flow when the next message is received in its input queue

.

This is not true for the NEON message flows. The mqsinrfreload command tells the Broker to reload some or all of the NEON rules and formats (individual components of an application group). All message flows using the NEON rules engine must first be stopped before the reload command is executed. The NEON rules engine is started, stopped, and paused from the command line and may also be controlled as an NT service.

Versioning Utilities Supportpac – IC01

The utilities in the supportpac manage message flow definitions created with the MQSIV2 Control Center. The combination of the naming scheme used by the message flow developer and the file name created by the utility may be used to differentiate versions. These versions may be checked into and out of a code control system as files thereby allowing them to be specified as part of a release or package.

mqsifiltermsgflows utility

The mqsifiltermsgflows utility takes an export file (exported from the

MQSeries Integrator Version 2 Control Center), and creates a 'filtered'

export file containing a single message flow. It does NOT modify the

original export file.

To use the mqsifiltermsgflows utility:

1) Create an export file by using the Control Center File-Export

function, and save this file into the <mqsi_root_dir>\Tool directory.

2) From the <mqsi_root_dir>\Tool directory, run the command:

mqsifiltermsgflows export-file message-flow-label

where export-file is the name of the file exported from the Control

Center, and message-flow-label is the name of the message flow to

extract from the export file.

This will create a file in the same directory with the name

'<label>.xml', where <label> is the label of the message flow contained

within it. If the second parameter is left blank, all message flows are

retained and a new export file with the suffix '.filtered.xml' is

created (all information not relating to message flows is removed).

This file may then be re-imported into the Control Center using the

File-Import function. You can combine more than one of these messages

flow files into a single export file using the mqsicombinemsgflows

utility.

mqsicombinemsgflows utility

The mqsicombinemsgflows utility takes a list of 'filtered' export

files, as produced by the mqsifiltermsgflows utility, and combines

them into a single XML export file, ready to be imported by the

File-Import function of the MQSeries Integrator Version 2 Control

Center. These files must exist in a separate directory, and this

directory must contain no other XML files. The combined XML export file

is put into this directory when the command has executed.

To use the mqsicombinemsgflows utility:

1) Copy the necessary 'filtered' export files into a new directory.

Ensure that there are no other XML files in this directory (i.e.

files with the '.xml' extension).

2) From the the <mqsi_root_dir>\Tool directory, run the command:

mqsicombinemsgflows directory new-export-file

where directory is the path of the directory in which the 'filtered'

export files are located, and new-export-file is the name of the

file into which these export files are combined.

After you have done this you can import the message flows using the

Control Center File-Import operation. This adds them to the workspace

and the local repository. From there you can edit them and/or check

them in to the shared configuration.

mqsideletemsgflows utility

The mqsideletemsgflows utility reads in a Control Center export file (as

produced by the File-Export function of the MQSeries Integrator Version

2 Control Center). It looks at all the message flows it finds in the

workspace within that export file (other than IBM primitives), and if

none of them are locked, deletes them directly from the shared

configuration in the configuration repository. It behaves just like the

Control Center, in that it accesses the Configuration Manager via

MQSeries to perform the necessary operations against the message flows.

Note that when deleting, a message flow is considered to match if it has

a label the same or a UUID the same. So the net result is that any

message flow with either the same label or the same UUID will be

deleted.

To use the mqsideletemsgflows utility to delete the message flows named

in a previously created export file:

1) Edit the mqsideletemsgflows.ini file, and change the values of the

three properties to specify the location of the Configuration Manager

to which you wish to connect (this is the equivalent of the