Control-M File Watcher Utility (ctmfw)
The ctmfw utility can be used to detect[(] the following file related events:
· Successful completion of file transfer.
· Creation of a file.
· Deletion of a file
Therefore ctmfw can be used to activate a job or for performing a task (e.g. sending an email message or adding/deleting conditions) that is dependent upon the creation or deletion of a file. This utility runs as a process on the client machine and waits for the creation or deletion of the file (or files – the utility can monitor single or multiple files) as specified. The ctmfw utility monitors the files as follows:
· File transfers; file is initially detected and the job continues to monitor the size of the file. When the file reaches a specified minimum size and does not increase in size for a specified period of time, ctmfw either completes with a status of OK or executes a specified DO action. DO actions can consist of adding or deleting conditions or executing a command[†].
· File size is ignored when files are created/deleted and a wildcard used in the ctmfw parameter as part of the filename.
· The ctmfw utility must first detect the existence of a file before it can detect its deletion.
The utility can be used as a Control-M job, command or issued directly on the command line. For a single file the command format is as follows;
ctmfw filename [mode] [min_size] [sleep_int] [mon_int] [min_detect] [wait_time]
All parameters after the filename are optional, however they must be specified in order and defaults will be assumed where values are omitted. If you only specify three parameters then the mon_int, min_detect and wait_time values will be set as per their defaults. For example;
ctmfw /tmp/data/ CREATE 100 10
Will result in the following command being generated;
ctmfw /tmp/data/ CREATE 100 10 10 3 0
The utility generates 3 alternative return codes; 0 - indicates that the task was successfully completed in the given time frame, 1 - indicates that the file was not deleted or created in the given time frame and, 2 - indicates that the ctmfw task timed out (this RC is only generated when ctmfw is run in command line mode). See Appendix D for a summary of exit codes and actual results.
When watching for multiple files the command format is as follows;
ctmfw –input rules_filename
Where rules_filename is the complete path name of the file containing the definitions for each file to be detected. The rules file contains two sections;
· ON_FILEWATCH statements identifying which files to detect, specific criteria for each file, and the action to take upon detection or non-detection. Any number of ON_FILEWATCH statements can appear in a rules file.
· Default values to use for the ON_FILEWATCH statements. The MIN-SIZE, MIN-DETECT and WAIT-TIME parameters are the same as for a single file. Two additional parameters are available; i) INTERVAL is the sleep interval (in seconds) between successive scans for all the files. This parameter replaces individual sleep_int and mon_int parameters for each file. The default value is 10; ii) FROM_TIME is the starting time for detecting all the files and is used with WAIT_TIME to identify the time frame for detecting and monitoring the files. This parameter is expressed in 24-hour, hhmm format. The default value is 0000.
Two points to note; all keywords must be entered in uppercase and, in a rules file, if a wildcard (*) is used in a filename in an ON_FILEWATCH statement, the name and path must not be embedded in quote marks. If any mandatory parameter is omitted from a rules file, the default value for that parameter is used (i.e. same rules as watching single files). If entered, they must appear in this order;
#**********************************************************************************************
# Mandatory Parameters
INTERVAL <60> # Sleep interval (seconds)
FROM_TIME <0000> # Starting time for all files (hhmm)
MIN_SIZE <0> # Minimum size for all files (bytes)
MIN_DETECT <3> # Number of iterations for all files
WAIT_TIME <0> # Time limit for all files (minutes)
# ON_FILEWATCH statements
ON_FILEWATCH <filename> [mode] [min_size] [min_detect] [wait_time]
[from_time] [cyclic_interval]
THEN
<action>
ELSE
<action>
END_ON
#**********************************************************************************************
In this example the # symbols are comments and defaults are shown for all mandatory parameters. The from_time is the time from which to watch for the specified file action; cyclic_interval is the interval between multiple operations of detecting the file (in minutes). This interval must be greater than the value for wait_time. If the cyclic_interval is 0, only one attempt to detect the file will be performed. The <action> strings refer to any one of these four actions;
· DO_COND <cond name> <cond date> <+|-> - will add (+) or delete (-) a condition.
· DO_CMD <command> - will execute a valid command under the command interpreter.
· DO OK - terminate ctmfw will status OK.
· DO NOTOK [exit code] - terminate ctmfw will status NOTOK. Exit code is optional and
replaces normal return code. If a cyclic_interval is included in
the ON_FILEWATCH statement then ctmfw will only terminate
on a DO OK/DO NOTOK statement.
If the file is detected and the size remains static within the time frame (CREATE) or the file has been deleted (DELETE) then the DO commands in the THEN block are executed. If the file is not detected/deleted within the time frame, the statements following the ELSE block are executed. The ctmfw utility terminates when either all the files in the Rules file have been processed or a DO_OK/ DO_NOTOK action is executed. The Do commands that can be defined within the rules file can be used in conjunction with the standard Do commands as defined under the Job Processing Definition[†].
Example
A job processing definition is created to implement a File Watcher job. The file must arrive between 19:00 to 22:00 and be created in the \tmp directory under the name trans.dat. The minimum file size is 100 bytes. The detection process should be performed each minute. The file size monitored every 10 seconds, and the number of intervals where the file size remains static is 5. If the file is not detected by 22:00, an alert should be sent to CONTROL-M.
Parameter / ValueJob Name / FileWatch.
Men Name / FileWatch
Owner / <controlm_user>
From Time / 1900
Command Line / ctmfw /tmp/trans.dat CREATE 100 60 10 5 180
On Statement/Code Processing
Stmt / *
Code / COMPSTAT=0
DO Cond / file_trans_dat_ok Date: ODAT Sign: +
Stmt / *
Code / COMPSTAT=1
DO Shout / To: CONTROL-M Text: “File trans.dat did not arrive on time”
Environmental Modifications
As File Watcher is a Control-M utility, the system makes the assumption that you are running under either the Control-M Server or Control-M Agent user-id (i.e. the owner field on the Job Processing Definition is set to either ctmserver or ctmagent). Clearly it is quite possible that there is a requirement to run this utility (or one of the many other utilities) under another user-id. If this is the case then you should fully test the job in question and ensure that the environmental variables are set as required, that the user has the necessary executable library in their path and that the read/write permissions are adequate. With regard to the environmental variables, this is a full list of possible settings that the non-Control-M user-id (i.e. not either ctmserver or ctmagent) may have to supply;
· CONTROLM
· CONTROLM_USER
· CONTROLM_DATABASE
· LIBPATH or LD_LIBRARY_PATH
· ORACLE_BASE
· ORACLE_HOME
· ORACLE_SID
· NLS_LANG
With regard to the ctmfw, only the CONTROLM variable appears to be required (and this is only when the utility is operating in DELETE mode). Therefore, instead of making permanent changes to system profiles, it may be easier to adopt the following command when starting the utility;
export CONTROLM=$CONTROLM/nd305p/30576001/ctm/ctmagent/ctm;
/nd305p/30576001/ctm/ctmagent/ctm/exe_AIX/ctmfw /home/sqcadm/ctm_test.log DELETE …
this entire line should be placed on the command line of the primary panel of the Job Definition Profile (the correct parameters should be placed after the DELETE command).
Appendix A - CTMFW Parameters
filename / Path of the file to be detected. The file name can include mask character * to represent any number of characters (including none). When including an * as part of a file name in a Rules file the name should not be enclosed in quotation marks. When a wildcard is used, the ctmfw utility does not monitor the size of the file.
Mode / CREATE / Detects creation of a file. If filename includes a wildcard, the size of the file is ignored. NB: If a mask is specified for the filename, the ctmfw utility will end successfully after detection of the first file that matches the mask.
DELETE / Detects deletion of a file. When the ctmfw utility is run in this mode, it first checks for files that match the specified name. After the file is detected, the ctmfw utility checks at the specified interval for deletion of that file. NB: If a mask is specified as the filename, the ctmfw utility will end successfully only after all detected files that match the specified mask have been deleted.
min_size / Minimum file size in bytes. This parameter is ignored when using wildcards in filename or when using DELETE mode. Default: 0 (i.e. any size detected).
sleep_int / Interval between successive attempts to detect the existence/deletion of a file (in seconds). Default: 60.
mon_int / Interval between attempts to monitor the size of a file after it is detected (in seconds). This parameter is ignored when using wildcards in filename or when using DELETE mode. Default: 10.
min_detect / Number of attempts to monitor file size where the size remains static and greater than or equal to min_size (indicating successful creation of the file). This parameter is ignored when using wildcards in filename or when using DELETE mode. Default: 3.
wait_time / Maximum time (in minutes) to run the process without detecting the file at its minimum size (CREATE) or detecting its deletion (DELETE). If the file is not detected/deleted in this specified time frame, the process terminates with an error return code, as described on the bottom of page 1 (above). Default: 0 (i.e. no time limit).
Appendix B - Further Reading
· Control-M Quick Introduction (DAP guide/Word document).
· Enterprise Controlstation User Guide (BMC manual/PDF format).
Appendix C – Do Commands
The Do commands are called into action once the criteria of an On statement have been satisfied. These are created in the steps panel of the Job Processing Definition (see ‘steps’ section on page 18 of the Control-M Quick Introduction for full details of how to create these on/do actions). Note that when Control-M encounters conflicting on/do actions then it will refer to a set list of priorities (e.g. if Do Cond conflict with the defined Out Conditions setting then Do Cond will over-rule the specified Out Conditions). See Chapter 5 of the ECS User Guide for a full list of how Control-M deals with these conflicts. Below is a full list of possible Do commands.
Parameter / PurposeDO Autoedit / N.b.: on the Control-M/Desktop this utility is called Do SetVar. Ina rerun of a job this is used to alter variables in the Job Definition.
DO Cond / Used to add or delete a condition when the On statement criteria are met. There are 3 sub-parameters; condition is the descriptive name of the condition, date is the date stamp assigned to the condition (automatically created, e.g. ODAT in this field means that Control-M will fill in the date stamp for the day on which the job was originally ordered), and sign is either – (deletes a condition) or + (adds a condition).
DO Forcejob / Forces a job to be placed in the Active Jobs File regardless of its scheduling criteria. As soon as the submission criteria have been met then the forced job will run.
DO Mail / Sends an email message to a recipient when the conditions in the On statement are met. The email title and body can use Autoedit variables to generate some of the text. This can also be done via the shout utility, although the mail utility is easier.
DO NotOK / Sets the job’s completion status to NotOK regardless of the actual completion status.
DO OK / Sets the job’s completion status to OK regardless of the actual completion status.
DO Rerun / Resubmits the job for processing in accordance with the Max Rerun and Interval parameters. Both of these sub-parameters are required for reruns (the first limits the number of times that a job may rerun, the latter is the gap, in minutes, between runs).
DO Shout / Specifies that a message be sent to a destination. Shouts can be sent to terminals, the Control-M logs, email recipients, file and logical destinations. See also do mail.
DO Stop Cyc / Do Stop Cyclic prevents further iterations of the current cyclic job.
DO Sysout / Specifies how the job’s output should be handled.
Appendix D – File Watcher Summary