Version 22.0.8473

• Overview
• Connector Configuration
  • Settings Tab
    • Connector Settings
    • Receive
    • Caching
  • Automation Tab
    • Automation Settings
    • Performance
  • Advanced Tab
    • Advanced Settings
    • Message
    • Logging
    • Miscellaneous
• Establishing a Connection
• Sending and Receiving Files
  • Sending Files
  • Receiving Files
  • Subfolder Headers
• Macros
  • Example

The File Connector can:

• Pull files from external directories into the CData Arc flow.
• Push files from the Arc flow into to external directories.

Overview

Each File Connector is configured with a path on disk that determines what folder it will read files from or write files to. This can be a local file path or a UNC path to access other file locations on the network.

When a File Connector receives a file from the configured path, it passes the received file off to the next connector in the Arc flow without modifying the file. In this way, File Connectors can be used to poll an external folder location for files to pull into the Arc flow.

When a file in the Arc flow is processed by a File Connector, that file is written to the folder at the configured path. In this way, File Connectors can be used to drop off processed files at a location external to the application.

File Connectors can be configured with username/password credentials to provide access to external filepaths that otherwise would not be accessible to the application. This allows Arc to pull files from protected folder locations without needing to grant the entire application access to the protected location.

The Path configured in the File Connector can be dynamically populated using macros. For more information, please see the Macros section.

Connector Configuration

This section contains all of the configurable connector properties.

Settings Tab

Connector Settings

Settings related to the core operation of the connector.

• Connector Id The static name of the connector. All connector-specific files are held in a folder by the same name within the Data Directory.
• Connector Description An optional field to provide free-form description of the connector and its role in the flow.
• Path The external filepath where the connector will pull files from or push files to. The Path can include macros for dynamic evaluation as described in the Macros section.

Receive

Settings related to pulling files from the external filepath.

• File Mask A glob pattern for filtering files that should be pulled from the configured Path. Only files matching the filemask will be pulled. This setting can be combined with Receive Filter in the Advanced tab to specify multiple filters.
• Delete files (after received) Whether to remove files from the external path after they are pulled into the Arc flow.

Caching

• File Size Comparison When enabled, the connector skips previously-downloaded files unless the size of the files change.
• Timestamp Comparison When enabled, the connector skips previously-downloaded files unless the timestamp of the files change.

Automation Tab

Automation Settings

Settings related to the automatic processing of files by the connector.

• Send A toggle that instructs the connector to automatically send files when they are ready.
• Retry Interval The interval the connector will wait before retrying a failed send.
• Max Attempts The number of attempts the connector will make to send the message. Setting this value to 1 instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by Retry Interval between each attempt.
• Receive A toggle that instructs the connector to automatically process files when they are ready and send them to the Output tab.
• Receive Interval The interval at which the connector will process all pending files and send them to the Output tab. The next field depends on the selection here:
  Hourly — A Minutes Past the Hour dropdown menu allows you to specify the number of minutes past the hour to process receive files.
  Daily — A Time field appears for specifying the time of day (in UTC) to process receive files.
  Weekly — Two fields appear. Day allows you to select the day of the week for processing, and Time allows you to specify the time (in UTC) to process receive files.
  Monthly — Two fields appear. Day allows you to select the day of the month for processing, and Time allows you to specify the time (in UTC) to process receive files.
  Minute — A Minutes field appears for specifying the number of minutes between processing intervals.
  Advanced — A five-position Cron Expression field allows you to specify exact processing intervals. Highlight the field in the connector for more information about these expressions.

Performance

Settings related to the allocation of resources to the connector.

• Max Workers The maximum number of worker threads that will be consumed from the threadpool to process files on this connector. If set, overrides the default setting from the Profile tab.
• Max Files The maximum number of files that will be processed by the connector each time worker threads are assigned to the connector. If set, overrides the default setting from the Profile tab.

Advanced Tab

Advanced Settings

Settings not included in the previous categories.

• Max Receive Files The maximum number of files that will be pulled in a single receive interval. If this value is not a positive integer, no limit will be applied.
• Overwrite Option Specifies how the connector should handle the case when it attempts to write a file to the external path and that file already exists. The connector can rename the current file to a unique filename, overwrite the existing file, append the current file to the existing file, skip the operation without error, or stop the operation and throw an error.
• Processing Delay The amount of time (in seconds) that the connector will wait to process files in the Input folder.
• Receiving Delay The amount of time (in seconds) that the connector will wait before receiving files from the remote path.
• Recurse remote subdirectories Toggles downloading files in subfolders of the target remote path.
• Temp Send Path A directory to use as a staging folder before moving the file to its configured destination path.
• Temp Send Prefix If specified, the connector will upload the file with the temporary prefix, then rename the file to its original filename after the operation is complete.
• Temp Send Extension If specified, the connector will upload the file with the temporary extension, then rename the file to its original filename and extension after the operation is complete.
• Local File Scheme A filemask for determining local file names as they are downloaded by the connector. The following macros may be used to reference contextual information:
  %ConnectorId%, %Filename%, %FilenameNoExt%, %Ext%, %ShortDate%, %LongDate%, %RegexFilename:%, %DateFormat:%.
  As an example: %FilenameNoExt%_%ShortDate%%Ext%
• Receive Filter A glob pattern filter to determine which files should be downloaded from the remote storage (e.g. *.txt). Negative patterns may be used to indicate files that should not be downloaded (e.g. -*.tmp). This setting should be used when multiple File Mask patterns are desired. Multiple patterns may be separated by commas, with later filters taking priority except when an exact match is found.

Message

Settings that determine how the connector will search for messages and handle them after processing.

• Save to Sent Folder A toggle that instructs the connector to keep a copy of sent messages in the Sent folder.
• Sent Folder Scheme Instructs the connector to group files in the Sent folder according to the selected interval. For example, the Weekly option instructs the connector to create a new subfolder each week and store all sent files for the week in that folder. The blank setting tells the connector to save all files directly in the Sent folder. For connectors that process many transactions, using subfolders can help keep files organized and improve performance.

Logging

Settings that govern the creation and storage of logs.

• Log Level Specifies the type of information to log in the connector’s Logs directory:
  • None — Does not create any logs.
  • Error — Creates logs only when the connector encounters an error.
  • Warning — Creates logs only when the connector issues a warning.
  • Info — Logs general information about the workflow, including any errors and warnings (if applicable).
  • Debug — Logs detailed debugging information for both successful and failed workflows.
  • Trace — Logs detailed trace information for both successful and failed workflows.

  Please note that Debug and Trace may log sensitive information including message contents and SSL certificates. Although connection properties (such as passwords) are masked, please review logs of this level for sensitive information before sharing them outside of your organization.

• Log Subfolder Scheme Instructs the connector to group files in the Logs folder according to the selected interval. For example, the Weekly option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance.
• Log Messages A toggle that instructs the connector to save a copy of the most recent message in the Logs directory. Note that the connector only keeps one message per subfolder, and the connector overrides the previously-saved message when it runs again.

Miscellaneous

Settings for specific use cases.

• Other Settings Allows configuration of hidden connector settings in a semicolon-separated list, like setting1=value1;setting2=value2. Normal connector use cases and functionality should not require use of these settings.

Establishing a Connection

File Connectors must have the appropriate permissions to read/write from the configured Path. Permissions issues are primarily a concern when the configured path is a UNC path to another server on the network, but may also arise when pushing or pulling files from a protected folder on the local disk.

If the user running Arc does not already have permission to access the Path, the Username, Password, and Domain fields can be set to a specific user that does have the appropriate permissions.

Sending and Receiving Files

Sending Files

The File Connector sends files from the Input/Send Folder to the external folder specified in Path. If Send Filter is specified in the Advanced tab, only files that match the filter will be sent. Files are automatically sent if Send Automation is enabled in the connector.

Receiving Files

The File Connector receives files from the external folder specified in Path and places them in the Output/Receive Folder of the connector. If the File Connector is connected to another connector in the flow, the file does not remain in this folder and is instead passed along to the next connected connector.

The connector will automatically poll the external folder for files to receive if Receive Automation is enabled in the Automation tab.

The connector will only pull files that match the specified File Mask. If Enable Timestamp Comparison or Enable File Size Comparison are enabled, the connector will cache file names pulled from the remote path and only receive files that have not been received before or have been modified since they were last received.

Subfolder Headers

If Recurse Subdirectories is set to True when receiving a file from the Path in the File connector:

• The message for the file that is received will contain a Subfolder header in its metadata.
• This header contains the subfolder, relative to the Path in the File connector, that the file was received from.
• This subfolder header will be supported by other connectors that support a Subfolder header in Send operations.

When sending files, if a Subfolder header is present on the message that is sent to the Path in the File connector:

• The file will be placed in the subfolder, relative to the Path in the File connector, that is specified in the subfolder header.
• The subfolder will be created if permissions are available.

Macros

The Path configured in the File Connector can be dynamically populated using Macros with the following syntax:

%MacroName%

The following macros are supported:

• ConnectorID
• Ext
• Filename
• FilenameNoExt
• Header:myHeaderName
• LongDate
• RegexFilename
• ShortDate

For these macros, ext refers the file extension for the file currently being processed by the connector.

Example

Some files need to be placed in different folders based on data included within the file, such as a header value promoted on the Arc message. If a file has a header called ‘Customer’ that contains the folder into which the file should be placed, the Path property of the File Connector could be set to the following:

/some/path/%Header:Customer%