Intel Secure File Transfer (SFT)
Content Replication
How To Configure Guide
Document InformationName / Intel Secure File Transfer (SFT) Content Replication How To Configure Guide
Version / v1.0
Date Created / April 27, 2011
Date Updated / September 25, 2011
Legal Notices and Disclaimers
Disclaimers
INTEL CORPORATION MAKES NO WARRANTY OF ANY KIND WITH REGARD TO THIS MATERIAL, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. INTEL CORPORATION ASSUMES NO RESPONSIBILITY FOR ANY ERRORS THAT MAY APPEAR IN THIS DOCUMENT. INTEL CORPORATION MAKES NO COMMITMENT TO UPDATE NOR TO KEEP CURRENT THE INFORMATION CONTAINED IN THIS DOCUMENT.
THIS SPECIFICATION IS COPYRIGHTED BY AND SHALL REMAIN THE PROPERTY OF INTEL CORPORATION. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED HEREIN.
INTEL DISCLAIMS ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY PROPRIETARY RIGHTS, RELATING TO IMPLEMENTATION OF INFORMATION IN THIS SPECIFICATION. INTEL DOES NOT WARRANT OR REPRESENT THAT SUCH IMPLEMENTATIONS WILL NOT INFRINGE SUCH RIGHTS.
NO PART OF THIS DOCUMENT MAY BE COPIED OR REPRODUCED IN ANY FORM OR BY ANY MEANS WITHOUT PRIOR WRITTEN CONSENT OF INTEL CORPORATION.
INTEL CORPORATION RETAINS THE RIGHT TO MAKE CHANGES TO THESE SPECIFICATIONS AT ANY TIME, WITHOUT NOTICE.
Legal Notices
Intel software products are copyrighted by and shall remain the property of Intel Corporation. Use, duplication or disclosure is subject to restrictions stated in Intel's Software License Agreement, or in the case of software delivered to the government, in accordance with the software license agreement as defined in FAR 52.227-7013.
The Intel logo is a registered trademark of Intel Corporation.
Other brands and names are the property of their respective owners.
Copyright 2009, Intel Corporation, All Rights Reserved
Document Version History
Version / Date / Document History1.0 / May 3, 2011 / Created by Jeremy Morrissey
Table of Contents
Legal Notices and Disclaimers
Disclaimers
Legal Notices
Table of Contents
1.Introduction
2.SMTP/Email Alert Option
3.File Alert Option
4.Controlling the Size of Content Replication (maxfilesize and maxfilecount)
5.Logging File Names Contained in Replic TAR Files
6.Generating a LIST file for files extracted onto Destination Systems
7.Using a custom working directory to avoid using the default user’s TMP folder
8.Replacing the use of the FTP “mget” command with “get –r” or another command
9.Minimizing Command Line Parameters using Configuration File
10.Logging to SYSLOG on Unix/Linux systems
11.Using SSH Public Keys On Windows Platform
12.Unix/Linux System Configuration Test Scripts
13.Developing custom alerting/error logging procedures
14.Script State Tracking
15.Using resendArchive.pl
16.Using skipfile.pl
17.Using restoreskippedfile.pl
18.Changing Your Password Used in SFTP
19.Migrating To an New Intel SFT Environment
20.Troubleshoot Issues
1.Introduction
The purpose of this guide is to provide detailed installation instructions on how to configure various features of the Intel SFT Content Replication capability on a Source or Destination system.
This guide depends on the Intel SFT Content Replication Install Guidesfor Unix/Linux or Windows procedures being followed to create an installation of this capability.
2.SMTP/Email Alert Option
Intel SFT Content Replication has the option that can be configured to send email alerts when an ERROR is logged during the execution of the ReplicateDir.pl or DownloadFiles.pl PERL scripts. All logs written by the script during the course of the script execution are send in the body of the email address.
Prerequisites for Unix/Linux:
The “sendmail” program must be configured on the system to be able to send email
You must have a valid email address that can be used in the FROM address of the email alerts.
You must have one or more valid email addresses to send the alerts to
Prerequisites for Windows:
Microsoft .NET FW 2.0 must be installed on the system
A valid SMTP host name that can be used to connect over the network port 25 to send email
You must have a valid email address that can be used in the FROM address of the email alerts
You must have one or more valid email addresses to send the alerts to
IMPORTANT INFORMATION BEFORE ENABLING
- All users should be aware that when --loglevel= command line parameter is set to DEBUG a lot of information can be containing in the log and any error alerts that are triggered. Depending on your system constraints, this may result in a delay in receiving the alert or alert failing to be sent. It is recommended to use default --loglevel= setting when alerting is enabled. Refer to the troubleshooting guide for how --loglevel= parameter is set.
- Windows users that are configuring passwords in their intelsftconfig.ini files should avoid enabling this alerting feature when --loglevel= command line parameter is set to DEBUG. Refer to the troubleshooting guide for how --loglevel= parameter is set. To disable, set the ALERT_SMTPTOADDRESS to empty.
To configure this capability follow these steps:
- Prior to following the below steps, you must have followed the steps found in the Intel SFT Content Replication Install Guide (for either Windows or Unix/Linux) to configure your system as the Source and/or Destination system.
- Locate the following files in your installation
…/intelsft/Services/DownloadFiles/intelsftconfig.ini
…/intelsft/Utilities/ReplicateDir/intelsftconfig.ini
- Open each of the above configuration files for editing
You will find the below properties that need to be edited in these files. Edit each property using the instructions below.
Property / ValueALERT_SCRIPT / With a standard installation, this property should be set to ALERT_SCRIPT=perl sendAlert.pl
The value points to a PERL script that is used to send the email alert and is provided in a standard installation.
If you do not find this property in your config file, add it with the value above.
ALERT_SMTPHOST / Windows users need to set this to the host name of an SMTP server that this sytem can connect to and send email
Unix/Linux users – leave this value blank
ALERT_SMTPTOADDRESS / The email addresses to send the alerts to
Windows users need to use the semicolon (;) as the delimiter for multiple email addresses
Unix/Linux users need to use the comma (,) as the delimiter for multiple email addresses
ALERT_SMTPFROMADDRESS / A valid email address to send the alerts from. This should be recognizable to the recipients of the alert and should not be treated as spam or junk mail.
ALERT_SMTPSUBJECT / The text that shows in the subject line for all email alerts. The default value is “Intel SFT CR Alert”. The actual subject line will be appended with a colon (:) and the name of the script that produced the error, such as “: ReplicateDir” or “: DownloadFiles”
- If the scripts are running on the Linux/Unix platform and your sendmail program is not found in the /usr/sbin/sendmail path then update the ALERT_SCRIPT configuration property value to pass an additional command line parameter indicating your sendmail program path
For example, if your sendmail program path was “/usr/bin/sendmail” then use the following:
ALERT_SCRIPT=perl sendAlert.pl --sendmail=/usr/bin/sendmail
- Verify that alerting was configured correctly and email alerts can be sent
Run the following command to test your alerting configuration. This will not trigger a replication process and is ok to run even if your system is not a Source system.
perl replicatedir.pl --source=test --data=test --archive=test --transfer=test --loglevel=4
The recipients configured should expect to see an alert with the subject similar to “Intel SFT CR Alert : ReplicateDir”
The alert body should be similar to what is shown below:
2011-05-15 08:14:09 232 INFO: replicateDir.pl 20110916 started
2011-05-15 08:14:09 232 DEBUG: init: begin
2011-05-15 08:14:09 232 ERROR: init: unknown transfer type 'test'!
2011-05-15 08:14:09 232 INFO: replicateDir.pl 20110916 stopped
3.File Alert Option
Intel SFT Content Replication has the option that can be configured to write a log file to a configurable path on your system when an ERROR is logged during the execution of the ReplicateDir.pl or DownloadFiles.pl PERL scripts. All logs written by the script during the course of the script execution are contained within this log file. The file name is uniquely named using the process id + a date time stamp.
You must have the below versions or later of both of the scripts to configure this option. The version and instructions on how to check the version can be found in the current version release notes found here: ftp://ftp.intel.com/Pub/middleware/intelsft/cr/releasenotes.txt
ReplicateDir.pl: 20110808
DownloadFiles.pl: 20110715
IMPORTANT INFORMATION BEFORE ENABLING
- Windows users that are configuring passwords in their intelsftconfig.ini files should avoid enabling this alerting feature when --loglevel= command line parameter is set to DEBUG. Refer to the troubleshooting guide for how --loglevel= parameter is set. To disable, set the ALERT_SMTPTOADDRESS to empty.
To configure this capability follow these steps:
- Locate the following files in your installation
…/intelsft/Services/DownloadFiles/intelsftconfig.ini
…/intelsft/Utilities/ReplicateDir/intelsftconfig.ini
- Open each of the above configuration files for editing
Locate the ALERT_SCRIPT property in the configuration files
Change the ALERT_SCRIPT property to have the following file:
ALERT_SCRIPT= perl sendFileAlert.pl
IMPORTANT REMINDER: Depending on your system or how you are running the perl commands, you may be required to specify the full path to the “sendFileAlert.pl” script in the above property value. If so, an example for you to reference may be:
ALERT_SCRIPT= perl /usr/example/sendFileAlert.pl
Locate the ALERT_DIRECTORY property in the configuration files
If you cannot find the ALERT_DIRECTORY property, then add this property
The value of this property must be where you desire to have the alert log files written. The full path must be specified and the user account running the perl scripts must have write access. For an example property and value within your configuration:
ALERT_DIRECTORY=/usr/alertlogs
- Verify that alerting was configured correctly and email alerts can be sent
Run the following command to test your alerting configuration. This will not trigger a replication process and is ok to run even if your system is not a Source system.
perl replicatedir.pl --source=test --data=test --archive=test --transfer=test --loglevel=4
A file should be written to the directory that you configured for ALERT_DIRECTORY within the intelsftconfig.ini files.
The file name format is ERROR_<PID>_YYYY-MM-DD_HH_MM_SS.log
The file contents should be similar to what is shown below:
2011-05-15 08:14:09 232 INFO: replicateDir.pl 20110916 started
2011-05-15 08:14:09 232 DEBUG: init: begin
2011-05-15 08:14:09 232 ERROR: init: unknown transfer type 'test'!
2011-05-15 08:14:09 232 INFO: replicateDir.pl 20110916 stopped
4.Controlling the Size of Content Replication (maxfilesize and maxfilecount)
There may be a need to control the size of the content being replicated using the ReplicateDir.pl command. By default the ReplicateDir.pl command will package all files it finds to replicate and transfer them in one package (tar.gz file). The size of this package may need to be controlled on your system. This can be achieved using the following two options:
- Specifying a maximum size in bytes of files to be added to the package
And/or
- Specifying a maximum number of files to be added to the package
Regardless of the option used above, the files that are not packaged will queue up and be packaged in a subsequent run of ReplicateDir.pl.
You must have the below versions or later of both of the scripts to configure these options. The version and instructions on how to check the version can be found in the current version release notes found here: ftp://ftp.intel.com/Pub/middleware/intelsft/cr/releasenotes.txt
ReplicateDir.pl: 20110808
DownloadFiles.pl: 20110715
The following command line parameters can be added to the ReplicateDir.pl to use these options:
--maxfilesize=<number of bytes>
When the parameter --maxfilesize is present in the command line to ReplicateDir.pl only files totaling up to that file size will be replicated in that run. All others will wait until the next ReplicateDir.pl script run. An exception is when an individual file size is greater than this --maxfilesize value and in this case, this file will be packaged in a tar file by itself and all others will wait until the next time ReplicateDir.pl script is run. This is to avoid skipping these files and never replicating them.
--maxfilecount=<number of files>
When the parameter --maxfilecount is present in the command line to ReplicateDir.pl only that number of files will be replicated in that run. All others will wait until the next ReplicateDir.pl script run.
5.Logging File Names Contained in Replic TAR Files
Intel SFT Content Replication has the feature to log the file attributes of files that are replicated between the Source and Destination systems. This feature is something must be enabled and is not enabled by default.
You must have the below versions or later of both of the scripts to configure this option. The version and instructions on how to check the version can be found in the current version release notes found here: ftp://ftp.intel.com/Pub/middleware/intelsft/cr/releasenotes.txt
ReplicateDir.pl: 20110808
DownloadFiles.pl: 20110715
To configure this feature follow these steps:
- To configure the Source system (that uses ReplicateDir.pl) edit your ReplicateDir.pl command line to add the below additional command line parameter:
--logfilesinarchive=Y
- To configure the Destination system (that uses DownloadFiles.pl) edit your …/DownloadFiles/intelsftconfig.ini configuration file to enable this feature
Open the ../intelsft/Services/DownloadFiles/intelsftconfig.ini file or your custom configuration file you are using in your command line for downloadFiles.pl
Locate all the properties within this file that have a property name LSP or end with LSP and notice that they call a perl script called “auto_untar_gz.pl”
Edit each of those properties so that an additional command line parameter is added to. The additional command line parameter is “--logfilesinarchive=Y”. For example:
LSP = perl auto_untar_gz.pl --logfilesinarchive=Y --extractdestpath=…
IMPORTANT REMINDER: Depending on your system or how you are running the perl commands, you may be required to specify the full path to the “auto_untar_gz.pl” script in the above property value. If so, an example for you to reference may be:
LSP = perl /usr/example/auto_untar_gz.pl --logfilesinarchive=Y --extractdestpath=…
6.Generating a LIST file for files extracted onto Destination Systems
Intel SFT Content Replication has the feature to create a verbose list file containing file attributes of files that are replicated between the Source and Destination systems. This feature is something must be enabled and is not enabled by default.
The main difference between this feature and the logging feature described above is that this feature creates a separate file for each package of files extracted on the destination system whereas the logging feature logs the same info into the log that is configured for the downloadFiles.pl script.
You must have the below versions or later of both of the scripts to configure this option. The version and instructions on how to check the version can be found in the current version release notes found here: ftp://ftp.intel.com/Pub/middleware/intelsft/cr/releasenotes.txt
ReplicateDir.pl: 20110916
DownloadFiles.pl: 20110916
This feature specifically uses options on your tar program configured
tar –tvf <archive file>
To enable:
1)Open the DownloadFiles/intelsftconfig.ini files or your custom configuration file that you may have configured downloadFiles.pl to use
2)Edit all property values for properties that are prefixed with “LSP” to contain the command line parameter of “--extractfilenames=Y”.
As an example:
LSP=perl auto_untar_gz.pl --extractfilename=Y
--extractedfilenameslog=/home/logpath/ --filetoprocess=
Once enabled, for every content replication downloaded by downloadFiles.pl, you will find a file created with the following naming convention in the path that --extractedfilenameslog parameter is configured with.
<replic_*.tar file name>.<date / time stamp>_<PID>.log
Example file names:
The below is an example of the contents when ran on SUSE Linux
7.Using a custom working directory to avoid using the default user’s TMP folder
The default folder that the scripts use for temporary files is the user’s Operating System managed TEMP folder.
All command lines support an optional --sftwd= parameter that when specified will instruct the script to use that folder and not your user accounts “temp” folder that the Operating System provides to you.
replicateDir.pl … [--sftwd=<full path to your custom folder>] …
downloadFiles.pl … [--sftwd=<full path to your custom folder>] …
8.Replacing the use of the FTP “mget” command with “get –r” or another command
Some SFTP programs do not have support for the FTP “mget” command. These SFTP programs are able to use other FTP commands that provide the same functionality; therefore, can be configured for use in the scripts.
Edit the following file:
…/intelsft/Services/DownloadFiles/intelsftconfig.ini
Review the file for a MGET_COMMAND property. If you do not find one, then add one: “MGET_COMMAND=”
Set the MGET_COMMAND to the FTP command that replaced “mget” for your SFTP program
For example, to replace “mget” with “get –r” (recursive get), then you would have this property set:
MGET_COMMAND=get -r
9.Minimizing Command Line Parameters using Configuration File
The information in this sections applies only to the version stated below or later versions. The version and instructions on how to check the version can be found in the current version release notes found here: ftp://ftp.intel.com/Pub/middleware/intelsft/cr/releasenotes.txt