S3 Connector

Version 21.0.8222


S3 Connector


The S3 Connector integrates with Amazon’s S3 (Simple Storage Service) and other S3-like services (Google Storage, Wasabi, etc).

概要

Each S3 Connector can automatically upload to and download from a single configured S3 Bucket. An Amazon Account (or Google Storage account, Wasabi account, etc) with the appropriate credentials is required. Upload and download paths can be specified within the Bucket, and files can be filtered by filename before download.

Connector Configuration

This section contains all of the configurable connector properties.

Settings Tab

Host Configuration

Settings related to the remote connection target.

  • Connector Id The static name of the connector. All connector-specific files are held in a folder by the same name within the Data Directory.
  • Connector Description An optional field to provide free-form description of the connector and its role in the flow.
  • バケット名 The S3 Bucket that should be polled or uploaded to.

Amazon Account Settings

Settings related to the Amazon Account with permission to access the configured バケット名.

  • アクセスキー The Access Key account credential acquired from Amazon (or Google, Wasabi, etc).
  • シークレットキー The Secret Key account credential acquired from Amazon (or Google, Wasabi, etc).
  • リージョン The Region where the specified バケット名 is stored.

SSL Settings

Settings related to the SSL negotiation with the S3 server.

  • Use SSL when connecting with Amazon servers Whether SSL negotiation is enabled.
  • サーバーパブリック証明書 The public key certificate to trust when connecting to the S3 server. Can be set to Any Certificate to implicitly trust the server.

Upload

Settings related to the path within the specified bucket where files will be uploaded.

  • リモートパス The path within the Bucket where files will be uploaded.
  • リモートファイルを上書き Whether files that already exist in the specified Bucket should be overwritten during upload.

Download

Settings related to the path within the specified bucket where files will be uploaded.

  • リモートパス The path within the Bucket from which files will be downloaded. Multiple paths can be specified in a comma-delimited list.
  • ファイルフィルタ A glob pattern that determines which files within the リモートパス should be downloaded. Multiple patterns may be specified in a comma-delimited list.

Automation Tab

Automation Settings

Settings related to the automatic processing of files by the connector.

  • アップロード Whether files arriving at the connector will automatically be uploaded.
  • 再試行間隔 The amount of time before a failed upload is retried.
  • Max Attempts The maximum number of times the connector will process the input file. Success is measured based on a successful server acknowledgement. If this is set to 0, the connect will retry the file indefinitely.
  • ダウンロード Whether the connector should automatically poll the remote download path for files to download.
  • ダウンロード間隔 The interval between automatic download attempts.
  • The number of minutes to wait before downloading. Only applicable when ダウンロード間隔 is set to Minute.
  • 毎時何分 The minutes offset for an hourly schedule. Only applicable when ダウンロード間隔 is set to Hourly. For example, if this value is set to 5, the automation service will download at 1:05, 2:05, 3:05, etc.
  • The time within a given day that the download should occur. Only applicable when ダウンロード間隔 is set to Daily, or Weekly, or Monthly.
  • The day on which the download should occur. Only applicable when ダウンロード間隔 is set to Weekly or Monthly.
  • Cron 式 An arbitrary string representing a cron expression that determines when the download should occur. Only applicable when ダウンロード間隔 is set to Advanced.

Performance

Settings related to the allocation of resources to the connector.

  • Max Workers The maximum number of worker threads that will be consumed from the threadpool to process files on this connector. If set, overrides the default setting from the Profile tab.
  • Max Files The maximum number of files that will be processed by the connector each time worker threads are assigned to the connector. If set, overrides the default setting from the Profile tab.

Advanced Tab

Other Settings

Settings not included in the previous categories.

  • アクセスポリシー The access policy set on objects after they are uploaded to the S3 server.
  • サイズ比較を有効化 Whether to cache downloaded file names and sizes; if True then files will only be downloaded if they have not been downloaded before or have changed in size.
  • タイムスタンプ比較を有効化 Whether to cache downloaded file names and last-modified timestamps; if True then files will only be downloaded if they have not been downloaded before or have been modified since they were downloaded.
  • 暗号化パスワード If set, object data will be encrypted on the client side before upload, and downloaded objects will be automatically decrypted.
  • UseServerSideEncryption Whether to request the S3 server encrypts object data server-side.
  • 送信フィルタ A glob pattern filter to determine which files in the Send folder will be uploaded by the connector (e.g. *.txt).Negative patterns may be used to indicate files that should not be processed by the connector (e.g. -*.tmp). Multiple patterns may be separated by commas, with later filters taking priority except when an exact match is found.
  • ローカルファイルスキーム A filemask for determining local file names as they are downloaded by the connector. The following macros may be used to reference contextual information:
    %ConnectorId%, %Filename%, %FilenameNoExt%, %Ext%, %ShortDate%, %LongDate%, %RegexFilename:%, %DateFormat:%.
    As an example: %FilenameNoExt%_%ShortDate%%Ext%
  • ログレベル The verbosity of logs generated by the connector. When requesting support, it is recommended to set this to Debug.
  • 親コネクタ The connector from which settings should be inherited, unless explicitly overwritten within the existing connector configuration. Must be set to a connector of the same type as the current connector.
  • サブディレクトリを再帰 Whether to download files in subfolders of the target remote path.
  • バーチャルホスティングを使用 Whether to use hosted-style or path-style when referencing the Bucket endpoint.
  • Log Subfolder Scheme Instructs the connector to group files in the Logs folder according to the selected interval. For example, the Weekly option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance.
  • Log Messages Whether the log entry for a processed file will include a copy of the file itself.
  • Save to Sent Folder Whether files processed by the connector should be copied to the Sent folder for the connector.

Miscellaneous

Settings for specific use cases.

  • Other Settings Allows configuration of hidden connector settings in a semicolon-separated list, like setting1=value1;setting2=value2. Normal connector use cases and functionality should not require use of these settings.

Establishing a Connection

The requirements for establishing an S3 connection are simple:

  • Amazon account credentials (or Google, Wasabi, etc)
    • アクセスキー
    • シークレットキー
  • An S3 Bucket that can be accessed by the above account

For Amazon S3 specifically, this link can be used to obtain アクセスキー and シークレットキー information from Amazon.
Optionally, the connection with S3 servers can be secured by SSL by enabling the Use SSL when connecting with Amazon servers option.

Uploading

Uploading to Remote Folders

The リモートパス setting within the Upload section specifies the path within the Bucket to upload files. This allows for the logical separation of files into virtual folders within the same Bucket.

Note that S3 servers do not maintain a real folder structure, and ArcESB uses application logic to present a pseudo folder structure. Slashes in the リモートパス (/, \\) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from ‘subfolders’ within the Bucket based on the slashes in the path.

Upload Automation

The S3 Connector supports automatic upload via the Automation tab in the connector configuration panel. When アップロード automation is enabled, files that reach the Input folder for the connector will be automatically uploaded to the specified バケット名 at the specified リモートパス.

If a file fails to upload, the application will attempt to send it again after the Retry Interval has elapsed. This process will continue until the Max Attempts has been reached, after which the connector will raise an error.

Downloading

Downloading from Remote Folders

The リモートパス setting within the Download section specifies the path within the Bucket to upload files. This allows for the logical separation of files into virtual folders within the same Bucket.

The ファイルフィルタ setting provides a way to only download specific filenames within the specified path.

Note that S3 servers do not maintain a real folder structure, and Arc uses application logic to present a pseudo folder structure. Slashes in the リモートパス (/, \\) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from ‘subfolders’ within the Bucket based on the slashes in the path.

Download Automation

The S3 Connector supports automatic upload via the Automation tab in the connector configuration panel. When ダウンロード automation is enabled, will be automatically poll the remote Bucket according to the specified ダウンロード間隔.