Create an SFTP activity
Create an activity that executes basic SFTP commands on a remote server.
Before you begin
Roles required: activity_creator or workflow_admin
About this task
You can create a custom activity that manages files and directories on a target host or copies a file from one SFTP server to another. The file content is streamed through a MID Server, which avoids having to store the data on the hard drive of the MID Server host machine.
Note: This activity requires the credentials of a user who can execute SFTP commands on the source host. The Copy File activity, provided in the activity pack, requires separate credentials to access the target host.
Procedure
Create a custom activity.
This action creates a custom activity using a template.
After setting up general properties and creating input variables, configure the SFTP Execution Command.
| Command | Type of activity to create. The choices are:- Copy file - Create directory - Get file list - Remove file or directory - Rename file or directory - Set file attributes Note: The fields that display on the form depend on the command you select. |
| Source host | Name or IP address of the server containing the files targeted by the activity. |
| Source port | Port number to use to communicate with the source server. The default port number is 22. |
| Source file path | Full path to a file \(`/temp/test_data_file.txt`\) or directory \(/`temp/test_dir`\) on a source host depending on the selected command. Field available on following commands: - Copy file - Remove file or directory - Set file attributes |
| Source directory | Path to the directory where the source files are located. Field available on following commands: - Create directory - Get file list |
| Source files | Names of specific source files to include in the file transfer. Separate the file names with semi-colons. This field supports a semi-colon separated list of wild-card patterns. For example: *.txt; a?cd.pdf. If this field is blank, all files are included. Field available on following command: Get file listNote: For information about how the values in this field affect options in other fields, see the logic explanation which follows. |
| Is a directory | If checked, specifies the file path is a directory. Field available on following command: Remove file or directory |
| Credential tag for source | Specific credential tag this activity must use to run SFTP commands on the source host. |
| Credential tag for target | Specific credential tag this activity must use to run SFTP commands on the target host. Field available on following command: Copy file |
| Excluded files | Names of specific source files to exclude from the file transfer. The activity acts on all other files found in the source directory or subfolders. Separate the file names with commas. This field supports comma-separated list of wild-card patterns. For example: *.txt, a?cd.pdf.. Field available on following command: Get file list Note: For information about how the values in this field affect options in other fields, see the logic explanation which follows. |
| Include subfolders | If checked, includes the files from subfolders in the source directory. Field available on following command: Get file list Note: For information about how your selection affects other fields in the form, see the logic explanation which follows. |
| Target host | Name or IP address of the server to which the files are being transferred. Field available on following command: Copy file |
| Target port | Port number to use to communicate with the target host. The default port number is 22. Field available on following command: Copy file |
| Target file path | Full path to a file on a target host. Field available on following commands: - Copy file - Rename file or directory |
| Suffix for temporary file | If a file exists on a target host, this command enables a temporary suffix to use for a file name. If this field contains a value, the activity first copies the source file to a temporary file on the target host using targetFilePath + tempFileSuffix as the name. Upon completion, the activity renames the file to the actual target file name. If this field is blank, the activity copies the source file directly to the target file and overwrites it, if it exists. Field available on following command: Copy file |
| UID | User ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. The UID and GID numbers are internal values returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This flow is demonstrated in the SFTP File Transfer workflow. Field available on following command: Set file attributes |
| GID | Group ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. The UID and GID numbers are internal values returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This flow is demonstrated in the SFTP File Transfer workflow. Field available on following command: Set file attributes |
| Permissions | File or directory permissions to set for the user and group specified. This value must be expressed as an integer, such as 16877, which defines these permissions: rwxr-xr-x. The permissions numbers are internal values returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This flow is demonstrated in the SFTP File Transfer workflow. Field available on following command: Set file attributes |
| Access timestamp | Override the timestamp when the file or directory was last accessed. The access and modification timestamps must be set together as a pair. Field available on following command: Set file attributes |
| Modification timestamp | Override the timestamp when the file or directory was last modified. The access and modification timestamps must be set together as a pair. Field available on following command: Set file attributes |
| Size in bytes | Size of a file, expressed in bytes. Field available on following command: Set file attributes |
| Required MID Server capabilities | MID Server with the appropriate MID Server capabilities for connecting to the source and target servers. |
The system uses this logic to determine which files to move from the source host:
- If the Source files field is empty, the system selects all the files in the source directory. Otherwise, it only selects those files whose names match one of the file name patterns given in the field.
- If the Excluded files field is empty, the system excludes nothing. Otherwise, it excludes those files whose names match one of the file name patterns given in the field.
- The exclude rule has a higher preference than the include rule. If a file name matches one of the file name patterns in the Excluded files field, it does not get into the selection regardless of the include rule.
- When the Include subfolders check box is cleared, the system looks only in the source directory for files to include or exclude. Otherwise, it looks in the source directory and any of its subfolders for files to include or exclude.
**Note:** You can map parameter values in a test payload to variables in the **Outputs** tab automatically. See [automap output variables](automap-output-variable-inputs.md).
What to do next
- Use auto-mapping to generate outputs and parsing rules (recommended for JDBC)
- If you do not use auto-mapping, you can manually create output variables and create parsing rules
Parent Topic:Orchestration custom activity templates
SFTP template execution parameters
You use execution parameters to create the input process script in the Preprocessing form.
For descriptions of the command fields, see Configure the SFTP execution command.
Note: You must use the executionParam. prefix with all variables in this table.
| Name | Variable | Type | Usage |
|---|---|---|---|
| Command | command | String | Identifies the function of this activity. The possible commands are: - Copy file - Create directory - Get file list - Remove file or directory - Rename file or directory - Set file attributes |
| Source | source | String | Identifies the source host. |
| Source port | source\_port | Integer | Identifies the port used to communicate with the source host. |
| Source directory | source\_directory | String | Path to the source directory of the files to be moved. |
| Source files | source\_files | String | Names of the files on the source to be moved. |
| Excluded files | excluded\_files | String | Lists the files excluded from the operation. |
| Source file path | source\_file\_path | String | Path to the source files to be moved. |
| Target host | target\_host | String | Identifies the target host. |
| Target port | target\_port | Integer | Identifies the port used to communicate with the target host. |
| Target directory | target\_directory | String | Full path to the target directory on the target host. |
| Target file path | target\_file\_path | String | Full path to a file or directory on the source host or on the target host. For example, this value is used in the Rename File or Directory activity in the activity pack. |
| Temp file postfix | temp\_file\_postfix | String | Temporary file name extension used by the Copy File activity when moving a file. If this field contains an extension, the source file is copied to a temporary file using target file name + temp_file_postfix, before being renamed to the actual target file. If this field is blank, the source file is copied directly to the target file. |
| Is directory | is\_directory | Boolean | Indicates whether the given file path is a directory. |
| Include subfolders | include\_subfolders | Boolean | Indicates whether the system looks into subfolders of the source directory for files to include and exclude. |
| File attribute uid | file\_attribute\_uid | Integer | User ID associated with a file or directory. |
| File attribute gid | file\_attribute\_gid | Integer | Group ID associated with a file or directory. |
| File attribute permissions | file\_attribute\_permissions | Integer | File or directory permissions for the user and group specified. |
| File attribute atime | file\_attribute\_atime | Integer | Access time stamp from the file attributes. |
| File attribute mtime | file\_attribute\_mtime | Integer | Modification time stamp from the file attributes. |
| File attribute size | file\_attribute\_size | Integer | Size of the file, in bytes. |
| Source credential tag | source\_credential\_tag | String | Credential aliases for Orchestration activities used to run the command on the source host. |
| Target credential tag | target\_credential\_tag | String | Credential tag used to run the command on the target host. |
| MidCapabilities | midCapabilities | String \(comma separated\) | List of required MID Server capabilities. |
SFTP post-processing parameters and payload parsing
Use these parameters to create a post-processing script, payload parsing, and tagging.
| Name | Variable | Type | Usage |
|---|---|---|---|
| Output | output | String | Contains output data returned by the query. |
| EccSysId | eccSysId | String | Contains the Reference ID associated with the ECC Queue input message. |
| ErrorMessages | errorMessages | String | Contains the error messages returned from the query. If no error messages are returned, this value is null. |
| Tags | tags | Hashmap of tag values returned from the SSH command | Contains the tags used to extract output using the SSH commands. The tag output is delimited by double percentage signs, as in |
Configure the SFTP execution command
Use the input variables you created to configure the command that Orchestration executes on the SFTP target.
Before you begin
Create the input variables you need in the Inputs form before you can advance to the Execution Command stage.
Role required: activity_creator, admin
Note: You can test the JDBC connection between the MID Server and the target without having to run the activity in a workflow context. For details, see test template outputs.
Procedure
Drag variables from the list of inputs and drop them into command fields.
The system formats the variable in the proper syntax for the command.
In this example, the Create directory command displays in the Command field.
- Complete the fields shown in the table.
| Field | Description | Command |
|---|---|---|
| Command | Type of activity to create. The choices are:- Copy file - Create directory - Get file list - Remove file or directory - Rename file or directory - Set file attributes Note: The fields displayed on the form depend on the command selected. | All |
| Source host | Name or IP address of the server containing the files targeted by the activity. | All |
| Source port | Port number to use to communicate with the source server. The default port number is 22. | All |
| Source file path | Full path to a file on a source host. | - Copy file - Remove file or directory - Set file attributes |
| Source directory | Path to the directory where the source files are located. | - Create directory - Get file list |
| Source files | Names of specific source files to include in the management action. Separate the file names in this list with semi-colons. This field supports the use of wild cards. If this field is blank, all files are included. For information about how the values in this field are affected by options in other fields, see the section below the table. | Get file list |
| Is a directory | Check box that determines if the specified file path is a directory. | Remove file or directory |
| Credential tag for source | Specific Credential aliases for Orchestration activities this activity must use to run SFTP commands on the source host. | All |
| Credential tag for target | Specific credential tag this activity must use to run SFTP commands on the target host. | Copy file |
| Excluded files | Names of specific source files to exclude from the management action. The activity acts on all other files found in the source directory or subfolders. Separate the file names in this list with semi-colons. This field supports the use of wild cards. For information about how the values in this field are affected by options in other fields, see the section below the table. | Get file list |
| Include subfolders | Check box to manage the files from subfolders in the source directory. For information about how your selection affects other fields in the form, see the section below the table. | Get file list |
| Target host | Name or IP address of the server to which the files are being transferred. | Copy file |
| Target port | Port number to use to communicate with the target host. The default port number is 22. | Copy file |
| Target file path | Full path to a file on a target host. | - Copy file - Rename file or directory |
| Suffix for temporary file | Temporary suffix to use for a file name if the file already exists on a target host. If this field contains a value, the activity deletes the duplicate target file if it exists, and then copies the source file to a temporary file using targetFilePath + tempFileSuffix as the name. Upon completion, the activity renames the file to the actual target file name. If this field is blank, the activity copies the source file directly to the target file and overwrites it, if it already exists. | Copy file |
| UID | User ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. | Set file attributes |
| GID | Group ID attribute to apply to a file or directory. The UID and GID values must be set together as a pair or they are ignored. | Set file attributes |
| Permissions | File or directory permissions to set for the user and group specified. This value must be expressed as an integer, such as 16877, which defines these permissions: rwxr-xr-x. The permissions number is an internal value returned by the Get File List activity. Typically, you first use the Get File List activity to return a list of files and their attributes. Then you can move a file from a source host to a target host and set the source file attributes on the target file. This is demonstrated in the SFTP File Transfer workflow. | Set file attributes |
| Access timestamp | Override the timestamp when the file or directory was last accessed. The access and modification timestamps must be set together as a pair. | Set file attributes |
| Modification timestamp | Override the timestamp when the file or directory was last modified. The access and modification timestamps must be set together as a pair. | Set file attributes |
| Size in bytes | Size of a file, expressed in bytes. | Set file attributes |
| Required MID Server capabilities | MID Server with the appropriate MID Server capabilities for connecting to the source and target servers. By default, the system selects an SSH MID Server. | All |
The system uses this logic to determine which files to move from the source host:
- If the **Source files** field is empty, the system select all the files in the source directory. Otherwise, it only selects those files whose names match one of the file name patterns given in the field.
- If the **Excluded files** field is empty, the system excludes nothing. Otherwise, it excludes those files whose names match one of the file name patterns given in the field.
- The exclude rule has a higher preference than the include rule. A file whose name matches one of the file name patterns in the **Excluded files** field does not get into the selection, even though it would be selected by the include rule.
- When the **Include subfolders** check box is cleared, the system looks only in the source directory for files to include or exclude. Otherwise, it looks in the source directory and any of its subfolders for files to include or exclude.
Click Save.
Click Continue to advance to the Outputs stage.