Tivoli Workload Scheduler and System Automation for z/OS Integration in an Enterprise of Multiple Plexes
September 2013
IBM Advanced Technical Skills
Art Eisenhour, Certified I/T Specialist:
Introduction
This paper discusses the areas of consideration when multiple Sysplexes that are running NetView for z/OS and System Automation for z/OS are integrated to work with a single Tivoli Workload Scheduler for z/OS system. It is the assumption of the authors that NetView for z/OS and System Automation for z/OS (SA) have already been installed in each Sysplex and that the users know how to customize SA policies. It is also assumed that Tivoli Workload Scheduler for z/OS (TWS) has been installed and is scheduling operations across the multiple Sysplexes, and that the users know how to customize, administer and schedule operations using TWS.
Special Notices
This document reflects the IBM Advanced Technical Skills understanding on many of the questions asked about the integration of System Automation and Tivoli Workload Scheduler for z/OS. It was produced and reviewed by the members of the IBM Advanced Technical Skills organization and additional subject matter experts. This document is presented “As-Is” and IBM does not assume responsibility for the statements expressed herein. It reflects the opinions of the IBM Advanced Technical Skills organization. If you have questions about the contents of this document, please direct them to Art Eisenhour ().
Trademarks
The following terms are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries: IBM, Netview, Parallel Sysplex, System z, Tivoli, z/OS and zSeries. A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml .
Other company, product and service names may be trademarks or service marks of others.
Acknowledgements
Co-authors:
John Cross, IBM Global Services
Adam Palmese, System z™ Technical Software Sales
Special thanks to Mike Sine, IBM Advanced Technical Skills for reviewing this document and providing valuable feedback.
Contents
Introduction 2
Special Notices 2
Trademarks 2
Acknowledgements 2
Overview 4
Move a Tivoli Workload Scheduler for z/OS (TWS) Controller to an alternate plex using System Automation for z/OS (SA) 5
Prerequites: 5
How to coordinate the management of a TWS Controller across an enterprise 6
Objectives: 6
Steps: 6
Examples of STARTUP and SHUTDOWN procedures: 6
How to query and control a remote TWS Controller with SA 8
How to send requests from TWS to SA using Conventional workstations 8
How to send requests from TWS to SA using Automation workstations 8
How to send requests from TWS to SA using a Batch Job 10
How to use TWS to initiate a planned move 10
How to bring up TWS in disaster recovery mode 11
SA Application STARTUP policy to start the controller with alternate options: 11
REXX exec example to start the controller with alternate options: 11
How to deliver Netview z/OS commands through TSO 11
Security 12
Summary and Further Information 13
Appendix A - AOFRYCMD setup 14
Appendix B - TWS and SA for z/OS Communication Flow 15
An enterprise consisting of multiple z/OS sysplexes or mono-plexes can have work scheduled by a single Tivoli Workload Scheduler for z/OS (TWS). However, it may be necessary at times to change the location of the Controller from one plex to another. This whitepaper describes how to automate the move of a Controller in a multi-plex environment using System Automation for z/OS (SA), and how to send requests from TWS on any plex to SA on any plex.
Overview
This figure illustrates an enterprise with one active TWS Controller that can be moved from one Sysplex to another through the coordinated operation of the separate SA systems. The enterprise consists of multiple separate Sysplex systems and each Sysplex consists of one or more z/OS images. Each Sysplex has an SA Automation Manager. And each z/OS image has a TWS Tracker and an SA Netview Agent. The disk storage that contains the TWS databases and plan files are switchable from one Sysplex to another as are the network connections related to the TWS Controller.
Overview of a planned move
Operations management has scheduled that SYSPLEX1 will be removed from service from 3 a.m. Sunday until 3 a.m Monday to allow upgrades to the hardware of the z/OS images, and the TWS controller is to be moved to SYSPLEX2. TWS will send a request before 3 a.m. Sunday to SA on SYSPLEX1 to stop the Controller started task on z/OS image MVS1A. SA will stop the TWS Controller and related server tasks. When the shutdown is complete, SA will order the network and disk storage switching to occur. When the switching operations are complete, SA on SYSPLEX1 will send a request to the SA Automation Manager on SYSPLEX2 to start the TWS Controller subsystems on MVS2A. When the TWS startup on MVS2A is complete, TWS will resume job scheduling automatically, and System Automation on SYSPLEX2 will notify the SA systems on the other plexes of the current location of the TWS Controller using global variables. The procedure to move the TWS Controller from SYSPLEX2 to SYSPLEX1 can proceed in similar manner or be left for another time.
Overview for unplanned outages
When the TWS controller cannot be restarted on the primary z/OS image or a hot standby backup within the local sysplex, then TWS disaster recovery procedures must be followed as described in the TWS Customization and Tuning documentation in the section on Disaster recovery planning. Otherwise, events might be lost and the files may not be in a consistent state. Dual job-tracking should be utilized and the alternate Controller started with JTOPTS CURRPLAN(NEW). Many of the techniques described for planned moves can be utilized, but preparations have to be made and procedures documented as to when and how the move will be executed.
The following sections describe how to use SA and TWS functions to enable these moves.
Move a Tivoli Workload Scheduler for z/OS (TWS) Controller to an alternate plex using System Automation for z/OS (SA)
Prerequites:
¨ Netview-to-Netview connections must exist between the plexes for RMTCMDs.
¨ Every z/OS image must have a TWS Tracker subsystem that is network connected to the Controller through either TCP/IP or SNA.
¨ The SA Application for the TWS Tracker should be defined to be active on all z/OS Systems.
¨ Every plex where the Controller may be located must have an SA system.
¨ The SA Application Group for the TWS Controller must be defined in a z/OS system group in each plex. Since only one instance may be active at a time, the Application Group should be ONDEMAND and the desired availability of the Applications should be ALWAYS.
¨ The TWS Controller options must enable SA provided exit EQQUXSAZ for Automation workstations, and/or EQQUX007 for Conventional (non-Automation) workstations
¨ The IP addresses or SNA LUs associated with the Controller must have a dynamic switch capability to move with the Controller.
¨ The TWS databases and Controller related files must be on DASD volumes that can be switched or quickly replicated to the target system when the Controller is moved.
¨ Documented procedures and REXX Execs / CLISTS must be defined to switch the DASD and Network connections when the TWS Controller is stopped on the active plex.
¨ The TWS administrator must be familiar with Disaster recovery planning as documented in the Customization and Tuning guide (Chapter 12), and TechNote 1104339, http://www-01.ibm.com/support/docview.wss?uid=swg21104339
Note: An unplanned move of the TWS Controller to another plex requires Disaster recovery procedures unless the Controller is stopped normally.
How to coordinate the management of a TWS Controller across an enterprise
Objectives:
· SA Policy definitions will support the move of a TWS Controller in a single policy.
· All SA systems will know the location of the active Controller by Netview domain name.
· All SA systems will know where to move the Controller when the active controller stops.
Steps:
1. Code a REXX exec to perform a Netview SETCGLOB command to set common variables in the startup and shutdown procedures described in the next section. Here is an example named “SETVAL” to customize to your needs.
/* REXX */
/* SETVAL: Issue the SETGLOB command */
PARSE ARG VAR1 VAL
'SETCGLOB 'VAR1' TO ' VAL
/* */
2. Define where to move the Controller when it is stopped. Here are two options using variables:
· Store the location to move the Controller in SA Systems AUTOMATION SYMBOLS. For example, using the Overview figure on page 3, set AOCCLONE9 for System MVS1A to be CNM21 as the “move to” domain for the Controller when it is active on MVS1A. And, set AOCCLONE9 to CNM31 for System MVS2A. The value in AOCCLONE9 can be retrieved within procedures as variable &AOFAOCCLONE9.
· Let TWS set a global variable that defines the move to location domain and submit the request to stop the Controller for the move. This will be illustrated later.
3. Define procedures in the SA policy for the Controller Applications for STARTUP and SHUTDOWN to set a common variable across the multiple Netview domains where the Controller may be located. This variable will indicate the location of the TWS Controller task when it is UP and have the value DOWN when it is stopped. This variable can then be queried via a PIPE or NCCF command or REXX EXEC.
Examples of STARTUP and SHUTDOWN procedures:
STARTUP, phase POSTSTART:
Set a common global variable in all SA systems to indicate where the Controller has started.
SA Application policy - Command Processing: POSTSTART
----------------------------------------------------------
Command Text
CNM11: SETVAL GLOBAL1.TWS.CONTROL.ACTIVE CNM&SYSCLONE.
CNM21: SETVAL GLOBAL1.TWS.CONTROL.ACTIVE CNM&SYSCLONE.
CNM31: SETVAL GLOBAL1.TWS.CONTROL.ACTIVE CNM&SYSCLONE.
Replace CNMxxx with the appropriate Netview domain names.
SYSCLONE values can be used to distinguish the domains and are resolved at ACF load time.
How to perform an initial startup of the TWS Controller with a NetView command:
The TWS Controller Group should be the object that is managed. This ensures that all related applications and automation tasks are active where the Controller is located. An initial startup command could look like this in which you specify the full resource name and target domain:
INGREQ {TWS_CONTROL/APG/SYSNAME} REQ=START SCOPE=ALL OUTMODE=LINE VERIFY=NO TARGET={DOMAINID}
Remember, only one instance of the Controller and related tasks may be active at a time in the enterprise.
SHUTDOWN, phase FINAL:
Set the common global variable value to "DOWN" to indicate that the Controller is stopped, then execute network and DASD switching, and request that the controller be started in the next domain.
SA Application policy - Command Processing: SHUTFINAL
----------------------------------------------------------
Command Text
CNM11: SETVAL GLOBAL1.TWS.CONTROL.ACTIVE DOWN
CNM21: SETVAL GLOBAL1.TWS.CONTROL.ACTIVE DOWN
CNM31: SETVAL GLOBAL1.TWS.CONTROL.ACTIVE DOWN
exec DASD and Network switching commands
INGREQ TWS_CONTROL/APG/&SYSNAME. REQ=START SCOPE=ALL OUTMODE=LINE VERIFY=NO TARGET={DOMAINID}
Where DOMAINID is the move to location variable, such as: &AOFAOCCLONE9, or a variable that is defined by TWS, e.g. &GLOBAL1.TWS.CONTROL.MOVE2LOC. See the section, “How to use TWS to initiate a planned move”, later in this whitepaper.
How to inititiate shutdown of the active TWS Controller with a NetView command:
INGSET TWS_CONTROL/APG/* REQUEST=MAKEAV* SOURCE=* OUTMODE=LINE
How to query and control a remote TWS Controller with SA
Here are 2 options:
1. Execute SA command, INGOPC with the option TARGET={domainid}, to send requests to a remote Controller. The target domainid could be the variable set by the STARTUP POSTSTART procedure described above.
2. If the Controller is defined to have a SNA LU, then the LU name can be defined in the SA OPC control specifications for the local Tracker resource.
How to send requests from TWS to SA using Conventional workstations
The Conventional (NVxx) non-Automation command workstations have been available for a long time and many applications use this interface. However, this interface submits requests only to the SA system that is located on the same sysplex as the TWS Controller and may have limited applicability in a multi-plex environment.
The following SA Product Automation for OPC definitions: OCS and ODM, will allow the TWS Controller to send requests to the local SA system using Conventional workstations.
Note: rather than use specific domain names, specify a value of SYSPLEX to let SA identify the active controller anywhere within the sysplex.
OCS:
TWS_CONTROLLER TWS Controller TWSC
OPCA PCS OPC Controller details
Entry Type : Controller Details PolicyDB Name : HASL34
Entry Name : TWS_CONTROLLER Enterprise Name : HASL
OPC PCS entry name: TWS_CONTROLLER
Enter or update the following table names:
Netview domain name. . . SYSPLEX
OPC controller subsystem TWSC ^ replace TWSC with your TWS Controller MVS subsystem name as defined in IEFSSNxx.
ODM:
DOMAINID Conventional (NVxx) Workstation Domain Map
AOFGDYN9 Code Processing : DOMAINID
Cmd Code 1 Code 2 Code 3 Value Returned
NV01 SYSPLEX
NV02 SYSPLEX
In the following example, the TWS workstation named NV01 is defined as a Conventional non-Automation workstation:
Work station name : NV01
WORK STATION TYPE ===> G General
DESTINATION ===> ________ ^ The destination name must be blank.
Options: AUTOMATION ===> N
The following TWS operation will submit a request from TWS to the local SA system to stop the resource named PCICS5 using a Conventional workstation:
NV01 010 00.00.01 PCICS5__ STOP_ ^ PCICS5 is the resource name,
STOP is the Op TEXT
How to send requests from TWS to SA using Automation workstations
TWS operations assigned to Automation workstations can be used to issue SA requests against resources located anywhere within the enterprise regardless of where the TWS Controller is located. The OPC OCS definitions should be the same as for Conventional workstations. The Automation workstation destination must name a valid active Netview domain or be blank.
In the following example, the TWS workstation named SA01 has been defined with a specific Netview domain name destination and does not require an SA OPC ODM entry.
Work station name : SA01
WORK STATION TYPE ===> G General
DESTINATION ===> HVNFA___ ^ The destination name must also be defined in the Controller ROUTOPTS, e.g. USER(HVNFA)
Options: AUTOMATION ===> Y
The SA01 Operations INGREQ commands are defined within the Operation AUTOMATION INFO, Command Text. This application will stop a database task, run a batch update, and restart the database task by canceling the stop request.
SA01 015 STOPDBC: INGREQ DSNTSADB/APL/HCB$
REQ=STOP,TYPE=NORM,OUTMODE=LINE,TARGET=HVNCB ^ TARGET= Netview Domain
to process the request
CPU1 040 UPDATEDB ^ UPDATEDB is a database update batch job
SA01 055 STARTDBC: INGSET CANCEL DSNTSADB/APL/HCB$
REQUEST=MAKEUN* SOURCE=* OUTMODE=LINE TARGET=HVNCB
In the above example, TWS sends the request to SA through the local Netview PPI to the domain named in the Workstation destination. SA in turn sends the request to the domain specified in TARGET=, and the target domain can be in a remote plex.