Fon’s Batch Procedures
TABLE OF CONTENTS
01.PREFACE......
11.1Document Control......
21.1.1Document Revision and Distribution......
31.2Purpose and Scope......
41.3Relationship to Other Standards......
51.4Amendment History......
61.5Cross Reference......
71.6Terminology and Acronyms......
82.INTRODUCTION......
92.1Scope of Service......
102.2Job Descriptions......
112.2.1Batch Execution Team Leaders......
122.2.2Batch Execution Operators......
132.3Training......
143.PROCEDURES......
153.1Provide Production Batch Cycle Monitoring Support......
163.1.1Perform Shift Turnover......
3.1.1.1Prepare Shift Turnover......
3.1.1.2Review Shift Turnover......
03.1.2Document Batch Cycle Phone Messages......
13.1.3Monitor Production Batch Jobs With CA-7......
3.1.4.1CA7 Queues......
3.1.4.2Determine Job/Cycle Status with CA-7 List Commands......
3.1.4.3Determine status of Time Jobs in CA-7......
3.1.4.4Place Jobs on Hold with CA-7......
3.1.4.5Demand in a Job with CA-7......
3.1.4.6Delete a Job from CA-7 Production Queue......
3.1.4.7Restart a Job through CA-7/CA-11......
3.1.4.8Force a JOB Complete Using CA-7......
3.1.4.9Post a Condition to a Job......
3.1.4.10List Documentation for a Job......
3.1.4.11Use the Forecast Commands......
3.1.4.12Use the List Log and List Prior Run Commands......
3.1.4.13Use the HELP Function Through CA-7......
3.1.4.14Production Scheduling Interface (PSI)......
3.1.4.15Search for Logon Ids for Jobs that were put on Hold......
3.1.4.16Escalation Procedures when a Cycle is Running Behind......
03.1.5ABEND Resolution......
3.1.5.1Initial Response to ABENDS......
3.1.5.2Research / Resolve Common ABENDs......
3.1.5.2.1Common Space Problems......
3.1.5.2.2Code Override Statements for SMS SB37 ABENDs Involving Dataclasses......
3.1.5.2.3Job Failed - JCL Error......
3.1.5.2.4Job Not Run - JCL Error......
3.1.5.2.5Migrated Datasets......
3.1.5.2.6Tape Errors......
3.1.5.3Abend Tracker Database......
3.1.5.3.1Sign into the database......
3.1.5.3.2Monitor abend entries......
3.1.5.3.3Mark an entry resolved......
3.1.5.3.4Secondary escalation......
03.1.6Use On-Call SE Response Facility......
3.1.6.1 Contacting Batch Scheduling oncall......
03.1.7Code Basic JCL......
3.1.7.1Reset PGENs......
3.1.7.2Code Override Statements......
3.1.7.3Code CA11 Restarts......
3.1.7.4Code Checkpoint Restarts......
03.1.8Monitor and Follow-up on Late Cycles......
13.1.9Investigate Pending Tape Mounts......
23.1.10Update Job Performance Groups......
33.1.11Perform HOLD, RELEASE, CANCEL and PURGE Functions in SAM......
43.1.12Use SAM to Monitor Looping Production Batch Jobs......
53.1.13Use SAM to Monitor Swapped Out Production Batch Jobs......
63.1.14Use SAM to Perform Modifications to a Job’s Output Criteria......
73.1.15Use SAM to Display Initiators, WTORs, and the MVS System Log......
83.1.16Use SAM to Display JES2 Queue, JES2 Held/Non-Held Output Information......
93.1.17Monitor Production Batch Jobs Using OMEGAMON......
103.1.18Use the XENQ Display Facility to Research Contention Situations......
113.1.19Monitor Transmissions Through Network Data Mover (NDM)......
3.1.19.1Sign on to NDM......
3.1.19.2View NDM Transmits In Process......
3.1.19.3View NDM Transmits Already Ended......
3.1.20 Cancelling Jobs in IMS on 231A......
03.2Use $AVRS......
13.2.1Use SAR (Sysout Archive Retrieval System)......
23.3Support Production Respool Reports......
33.4 ACCESS VPS AND VTAM MENU PANELS
43.5Reset VPS Printers......
53.6Acquire Credit Union Printers / Terminals to a Zone......
63.7Reset Passwords......
73.7.1Reset Password Guidelines:......
83.8VANTIVE Print Request Tickets......
93.9VANTIVE Tickets......
103.10Send a Message to a TSO User......
113.11Activate an EID......
123.12Change Logmodes......
133.13Search for Data Using the TSO Search Facility......
143.14Prepare and Respond to Initial Program Loads (IPL)......
153.15Prepare for Scheduled IPLs......
163.15.1Respond to an Unscheduled IPL......
173.15.2Respond to a System Crash......
183.16Support Account Migrations......
193.17Documentation Change Control......
20APPENDIX A - QUALITY RECORDS......
21APPENDIX B - Commonly Used Datasets / Filenames......
22APPENDIX C - Common ABEND Codes......
23APPENDIX D - Batch Execution Training Progression......
INTRODUCTION
3.PROCEDURES
3.1Provide Production Batch Cycle Monitoring Support
Production batch cycles are monitored on the MVS operating system and UNIX system throughout each shift to ensure cycles complete according to customer requirements. Production batch cycle monitoring requires the execution of the following actions:
3.1.1Perform Shift Turnover
Batch Execution operators perform shift turnover to communicate cycle status, outstanding issues, and any special instructions between shifts. This includes two processes: Prepare Shift Turnover and Review Shift Turnover. Where applicable, account specific turnover procedures are documented within account specific Batch Execution work procedure documents.
3.1.1.1Prepare Shift Turnover
- Using TSO MON, update as necessary and complete by the end of the shift.
A) Type TSO MON from the ISPF primary option menu (SAIPC1A).
B) Type the appropriate monitor number.
C) Type the generation (0 is for the current date).
D) Type the action taken (E (edit) or B (browse)).
E) Document and detail all account status’s and special instructions.
F) Document problems relating to the system.
G) List any pending items. This is to ensure that the person taking over the cycle can follow-up on pending items in an appropriate and timely manner.
H) In the comment section of TSO MON, list discrepancies and the Ses that have been contacted. This is to ensure that the person taking over the cycle will not call on the same item.
- Update the paper Monitor log sheet to document individual cycle status and all account specific checklist items by the end of the shift. The Monitor log sheets (ICSBEX**) contain a list of the account(s) each operator is responsible for during that shift as well as specific account check-list items that are to be monitored.
- Day shift: using the appropriate monitor sheet, place an N/C and monitor initials next to the cycles that are not scheduled for that day.
- All shifts: using the appropriate monitor sheet, indicate which cycles have completed by initialing next to the cycle.
- All shifts, using the appropriate monitor sheet, initial all check-lists items to denote that checklist items are addressed.
- Update the ABEND Worksheet form (IMSBEX01) with any outstanding ABENDs to be turned over.
- Give verbal turnover to the oncoming shift
- Discuss problems, special changes, or any related information.
- Answer any questions from the oncoming shift.
- Document the answers in turnover for reference when necessary for review.
- Update / close all ABEND records.
3.1.1.2Review Shift Turnover
- Read on-line turnover.
- Status the assigned cycles.
- Check for outstanding ABENDs. Outstanding ABENDs should be documented on the ABEND Worksheet (IMSBEX01).
- Check for previous ODATEs that have not started to execute.
- Receive verbal turnover from previous shift.
- Check Temp Change Members on SAIPC1x:
PSC1x.OACSC.TEMPCHN1 / For changes effective on Monday
PSC1x.OACSC.TEMPCHN2 / For changes effective on Tuesday
PSC1x.OACSC.TEMPCHN3 / For changes effective on Wednesday
PSC1x.OACSC.TEMPCHN4 / For changes effective on Thursday
PSC1x.OACSC.TEMPCHNW / For changes effective on Friday thru Sunday
Note: x = System ID. (A or B)
- After receiving turnover, cycle monitors will use appropriate monitor turnover log checklists.
- Release the out-going cycle monitor. After turnover, the incoming cycle monitor will release the out-going cycle monitor if all turnover items have been answered and all completed ABEND records have been closed. Written and on-line turnover must also be updated even if there were no problems to report.
- File previous day’s Monitor log sheet, if completed.
3.1.2Document Batch Cycle Phone Messages
Timely and accurate communication is essential to our ability to support our customer’s production batch cycles. Due to the importance of production batch related phone calls, minimum requirements for documenting these phone messages have been established and are listed below.
- Who the message is for (cycle monitor)
- Caller’s name
- Caller’s account or customer name
- Caller’s remote site (if relevant)
- Caller’s phone number
- Description of the request/problem
Upon documenting the information above, the individual taking the message will promptly convey the information to the intended cycle monitor. Once messages are processed, they may be discarded. Phone messages are not considered quality records.
3.1.3Monitor Production Batch Jobs With CA-7
The CA-7 Commands Guide contains additional information about the commands used in these work instructions. The following examples can be used to assist with CA-7 instructions.
Using the Forecast facility in conjunction with the List commands will be a very effective way to monitor and status cycles in CA-7. The Forecast facility projects the scheduled workload for a given period of time. The List commands are used to show the status and modify the jobs in the different CA-7 queues. When entering CA-7 commands, the cursor needs to be above the function line. To get into CA-7 type E.7.7. from the ISPF menu.
3.1.4.1CA7 Queues
CA7 uses different queues to monitor batch cycles. A few queues are detailed below.
- Request Queue - When a job is scheduled to run.
- Ready Queue - When all requirements for the job are satisfied.
- Active Queue - When the job is executing in the system.
- Prior Run Queue - List the last time that a job executed.
3.1.4.2Determine Job/Cycle Status with CA-7 List Commands
- Select option 7 and then option 1 from the Scheduling Products Menu.
- Type XQ,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Wild cards can also be used, such as WCBS* to show all WCBS jobs. This command will display jobs in the Request queue and allow the status of those jobs to be updated.
- Type LQ,JOB=Y or LQ,ST=ABND,JOB=*
- Where Y= is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Wild cards can also use used , as in the previous examples. This command will show the status on all production activity in CA-7. The second command will show you all the ABENDed jobs. This command is helpful in determining the queue in which a job currently resides.
- Type LJOB,JOB=Y,LIST=ALL
- Where Y is any job cycle being monitored such as WCBSTCAR or MCAJCW01. Wild cards can also be used as in the previous examples. This command will show all the information on a job such as in-conditions and prose.
3.1.4.3Determine status of Time Jobs in CA-7
This can be done by executing either command "LQ,JOB=TIME*" or "XQ,JOB=TIME*" in CA7. If there is a time job in the queue with a time that has already past for the current day, the cycle monitor needs to manually post the submit time requirement.
If there are system problems, check account cycle jobs(current julian date) that are waiting for a submit time through the PSI function QUEUES in the START TIME column.
3.1.4.4Place Jobs on Hold with CA-7
There are two ways to place a job on hold with CA-7. You can use the Hold command or go through the XQ screen.
- Type HOLD,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. CA-7 will prompt you for text and the job will be held.
- The second way is through the XQ screen. Type XQ,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Wild cards can also be used, as in the previous examples. Move the cursor down to the job you want to hold and type an H in front of it. CA-7 will prompt you for text for holding the job. Type in your text and the job will be held.
3.1.4.5Demand in a Job with CA-7
When demanding a job in with CA-7 it is very important to know if there are jobs that are going to be triggered. It is a good idea to demand out a job in Hold status and then release it.
- Type DEMAND,JOB=Y
- Where Y is any job you want to demand out.
- Type DEMANDH,JOB=Y (To force out a job on Hold)
- Where Y is any job you want to demand out on hold.
- Type DEMANDH,JOB=Y,SCHID=Z (To force out a job on Hold with a certain Schedule ID).
- Where Y is any job you want to demand out and Z is the Sched ID the job resides in.
- Type DEMAND,JOB=Y,SET=NTR
- Where Y is any job you want to demand out and SET=NTR is to force out without any triggering.
3.1.4.6Delete a Job from CA-7 Production Queue
Use the XQ screen to delete a job from CA-7.
- Type XQ,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Wild cards can also be used, as in the previous examples. Move the cursor down to the job you want to cancel and type an C in front of it. CA-7 will prompt you for text for canceling the job. If the job is in a Skeleton status you will have to enter a Top-Line command: CANCEL,JOB=Y,REASON=____,FORCE=YES.
3.1.4.7Restart a Job through CA-7/CA-11
There are two ways to restart a job with CA-7.
- Type RESTART,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. This command will restart a job awaiting restart in the Request queue.
- Type XQ,JOB=Y
- Where Y is any job being monitored such as WCBSTCAR or MCAJCW01. Move the cursor down to the job to be restarted. Type an F in front of the job. This will pull up the CA-7/CA-11 restart panel. Place an X in front of the Resubmit For Production function. CA-7 will display a prompt for some documentation on why the job is being restarted. The cycle monitor can also force complete a job from here. Be CAREFUL when force completing a job because any follow on jobs will execute.
- REQUEUE,Job=Y
- This command will requeue a job that is hung in the Ready queue.
- To restart a job with CA-11 Type XQ,JOB=Y
- Where Y is any job being monitored, such as WCBSTCAR or MCAJCW01. Move the cursor down to the job to be restarted. Type F in front of the job. This will pull up the CA7/CA11 restart panel. Place an X in front of the CA-11 RESTART/RERUN function. CA-11 will have the Jobstep and Procstep in the CA-11 restart fields. You can overtype these fields also.
3.1.4.8Force a JOB Complete Using CA-7
- Type XQ,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Move the cursor down to the job you want to Force Complete and type a F in front of it. This will take you to the Restart screen. Type an X in front of the FORCE COMPLETE function. CA-7 will prompt you for text on why you forced complete the job.. Be CAREFUL when force completing a job because any follow on jobs will execute.
3.1.4.9Post a Condition to a Job
Use the XQ screen to add a condition to a job in CA-7.
- Type XQ,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Wild cards can also be used. As in the previous examples. Move the cursor down to the job you want to add a condition to and place an X in front of it. Cursor down to the condition that is to be remove and type an X in front of it.
- Type ADDRQ,JOB=Y,DEPJOB=X
- Where Y is the job in the queue, and X is the job you want to add to Y as an additional dependency.
3.1.4.10List Documentation for a Job
Use the LPROS command to list documentation for a particular job.
- Type LPROS,JOB=Y
- Where Y is any job cycle you are monitoring such as WCBSTCAR or MCAJCW01. Wild cards can also be used, as in the previous examples. This command will show you the documentation the scheduler or programmer has provided.
3.1.4.11Use the Forecast Commands
Use the Forecast command to forecast the scheduled work for a given period of time. Successor jobs triggered by predecessor jobs or any jobs triggered by data sets created in the predecessor jobs may also be included.
- Type FJOB,FROM=(mmddyy,hhmm),TO=(mmddyy,hhmm),JOB=**
- This command will show you all the jobs that will run for a specific period of time by jobname, or you can use wild cards as in the previous examples. You can replace JOB=** with SYS=** to forecast by System ID.
- Type FRJOB,JOB=Y
- Where Y is any job cycle that is being monitored, such as WCBSTCAR or MCAJCW01. This command will show a reverse job flow (reverse trigger flow) based upon information from the database.
- Type FSTRUC,JOB=Y,SCHID=NN
- Where Y is the job you want to forecast from and NN is the schedule ID you are forecasting from. This command will show you the cycle flow from the job you specify.
3.1.4.12Use the List Log and List Prior Run Commands
- Type LRLOG,SPAN=*,JOB=Y
- Where SPAN=* is the amount of days or time you want to list and Y is the Job you want to search for. This command will list information from the CA-7 Run Log. These events include job and network completions and exception events such as restarts, force completes, and cancels.
- Type LPRRN,JOB=Y
- Where Y is any job being monitored. This command will list job information from the prior run queue. The prior run queue contains information about the last successful completion of each job.
3.1.4.13Use the HELP Function Through CA-7
- Type HELP
- This will take you to the CA-7 Help screen or the CA-7 Online Tutorial screen
- Type in a Y at the INDEX: prompt. This will give you an Index of all the CA-7 commands you can get help on.
- Overtype the word INDEX with a Command you want to get help on. Such as LQ or XQ. The CA-7 Help will give you Syntax and Examples of commands.
3.1.4.14Production Scheduling Interface (PSI)
Access PSI
- Enter “E” from the Primary Option Menu to enter the EDS Corporate Application screen.
- Enter “7” - Scheduled Products screen.
- Enter “P” - Production Scheduling Interface screen.
4. There are several areas displayed , however you want to enter “1”for active queue information..
5. Enter Job name and System name, or asterisk in both fields.
Queue Display
The information displayed on the screen is what is in the queue at that moment. Jobs will not display until the time they have been scheduled into the queue. In order to see a job you may have to go to CA-7 Account Forecast utility. Do not rely on PSI to find the information you need use any other option available.
Sixteen different fields are displayed using two queue panels (use PF11 to view the second screen).
Three of the fields ,”QUE”,JOB STATUS” and “ENTRY MODE” have variable entries as listed
below. All other fields do not display variable information.
1. QUEUE- information displayed will be one of the following:
- REQ - Job is in the Request Que.
- RDY - Job is in the Ready Que.
- ACT - Job is in the Active Que.
3. JOBS WITH READY TO RUN STATUS IN PSI
- Check in SAM if job took JCL error
- Check SAM if job is waiting for initiator
- Check job in CA7 using RM.1 if the job is waiting for resources
4. JOB STATUS - information will be one of the following:
- ABENDED - Job has abended.
- DUP JOB -Duplicating job in the request Que.
- EXECUTING - Job is executing.
- HOLD - Job is on hold.
- JCLERROR - Job has JCL error.
- JOBFAIL - Invalid job card or JES card.
- READY TO RUN - Job ready to run in the Ready Que.
- REQUEUED - Job has been requeued.
- RETRY - Job in Retry status.
- SKELETON - Job in SKELETON Status.
- WAIT RQMTS - Job is waiting on submit time.
- Z-DOWN - ZOOM is inactive.
- Z-ERROR - ZOOM has encountered a severe error.
- Z-JCL FAIL - ZOOM could not resolve the JCL.
- Z-PENDING -CA-7 is notifying ZOOM to process the job.
- Z-QUEUED - Zoom has queued the job for processing.
- Z- RUNNING - Zoom is resolving the JCL.
- Z-SSCAN - JCL has been resolved and is waiting for CA-7’s schedule scan.
4. ENTRY MODE - Job entry mode has three possibilities as follows: