Azure Blob Connector
Azure Blob Connector
The Azure Blob Connector supports uploading to and downloading from the Azure Blob cloud storage service.
Each Azure Blob Connector connects to a specified Microsoft Azure storage account. Within the remote storage, a single folder is set as the upload target and a single folder is set as the download target.
Files that reach the Azure Blob Connector in the CData Arc flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the Arc flow. The connector supports upload and download filters to only send and receive specified file names and/or file extensions.
This section contains all of the configurable connector properties.
Settings related to the Azure account used to access the Azure Blob service.
- Connector Id The static name of the connector. All connector-specific files are contained in a folder by the same name within the Data Directory.
- Connector Description An optional field to provide free-form description of the connector and its role in the flow.
- Account The Azure Blob account to access.
- AccessKey The access credential (provided by Microsoft) for the specified account.
Settings related to uploading to the remote storage.
- Container Name Files processed by the connector will be uploaded to this specified container.
- Overwrite remote files Whether files that already exist in the remote folder should be uploaded (overwritten) or skipped.
Settings related to downloading from the remote storage.
- Container Name The container from which files will be downloaded. Multiple containers can be specified in a comma-delimited list.
- File Mask A glob pattern that determines which files within the Remote Folder should be downloaded (e.g. *.txt). If multiple File Masks are required, the Receive Filter advanced field should be used.
- Delete files (after received) Whether files that are successfully downloaded should be deleted from the remote storage afterwards.
Settings related to the automatic processing of files by the connector.
- Send Whether files arriving at the connector will automatically be uploaded.
- Retry Interval The amount of time before a failed upload is retried.
- Max Attempts The maximum number of times the connector will process the input file. Success is measured based on a successful server acknowledgement. If this is set to 0, the connect will retry the file indefinitely.
- Receive Whether the connector should automatically poll the remote download path for files to download.
- Receive Interval The interval between automatic download attempts.
- Minutes The number of minutes to wait before downloading. Only applicable when Receive Interval is set to Minute.
- Minutes Past the Hour The minutes offset for an hourly schedule. Only applicable when Receive Interval is set to Hourly. For example, if this value is set to 5, the automation service will download at 1:05, 2:05, 3:05, etc.
- Time The time within a given day that the download should occur. Only applicable when Receive Interval is set to Daily, or Weekly, or Monthly.
- Day The day on which the download should occur. Only applicable when Receive Interval is set to Weekly or Monthly.
- Cron Expression An arbitrary string representing a cron expression that determines when the download should occur. Only applicable when Receive Interval is set to Advanced.
Settings related to the allocation of resources to the connector.
- Max Workers The maximum number of worker threads that are consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the Profiles page.
- Max Files The maximum number of files that are processed by the connector each time worker threads are assigned to the connector. If set, this overrides the default setting on the Profiles page.
Settings not included in the previous categories.
- Log Level The verbosity of logs generated by the connector. When requesting support, it is recommended to set this to Debug.
- Timeout The duration the connector should wait for a response from the remote server before throwing a Timeout error.
- Parent Connector The connector from which settings should be inherited, unless explicitly overwritten within the existing connector configuration. Must be set to a connector of the same type as the current connector.
- Receive Filter A glob pattern filter to determine which files should be downloaded from the remote storage (e.g. *.txt). Negative patterns may be used to indicate files that should not be downloaded (e.g. -*.tmp). This setting should be used when multiple File Mask patterns are desired. Multiple patterns may be separated by commas, with later filters taking priority except when an exact match is found.
- Send Filter A glob pattern filter to determine which files in the Send folder will be uploaded by the connector (e.g. *.txt). Negative patterns may be used to indicate files that should not be uploaded (e.g. -*.tmp). Multiple patterns may be separated by commas, with later filters taking priority except when an exact match is found.
- Recurse Subdirectories Whether to download files in subfolders of the target remote path.
- Log Subfolder Scheme Instructs the connector to group files in the Logs folder according to the selected interval. For example, the Weekly option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
- Log Messages Whether the log entry for a processed file will include a copy of the file itself.
- Save to Sent Folder Whether files processed by the connector should be copied to the Sent folder for the connector.
Miscellaneous settings are for specific use cases.
- Other Settings Enables you to configure hidden connector settings in a semicolon-separated list (for example,
setting1=value1;setting2=value2). Normal connector use cases and functionality should not require the use of these settings.
Establishing a Connection
Establishing a connection to the Azure Blob service requires Azure account credentials:
The Test Connection button can be used to verify that the account provided is valid.
Uploading and Downloading
To upload files, set the Send -> Remote Folder field to the folder where files should be uploaded to. Each Azure Blob Connector uploads to a single specified folder location.
The Overwrite remote files option can be toggled to determine whether files that already exist in the remote folder should be overwritten or skipped. The Send Filter field in the Advanced tab can be used to determine which files should be uploaded by the connector, based on filename or file extension.
After configuring, any files placed into the Send/Input Folder of the connector will be uploaded to the remote storage. If Send Automation is enabled, the upload will happen automatically; otherwise, individual files can be Sent via the Input tab of the connector settings.
To download files, set the Receive -> Remote Folder field to the folder where files should be downloaded from. Multiple download folders can be specified in a comma-delimited list.
The File Mask field can be set to a glob filter (e.g. *.txt), and only files matching this filter will be downloaded. To specify multiple filters, the Receive Filter setting can be configured in the Advanced tab.
The Delete files (after received) option toggles whether or not successfully downloaded files should be removed from the remote storage afterwards.
After configuring, files will be downloaded according to the Receive Automation settings, or by manually clicking the Receive button within the Output tab.