Version 24.3.9120

• Add the Avro Connector
• Authenticate to Avro
  • Local
  • Amazon S3
  • Azure Blob Storage
  • Azure Data Lake Storage
  • Azure Files
  • Box
  • Dropbox
  • FTP
  • Google Cloud Storage
  • Google Drive
  • Hadoop Distributed File System (HDFS)
  • HTTP
  • IBM Object Storage Source
  • OneDrive
  • Oracle Cloud Storage
  • SFTP
  • SharePoint Online
• Complete Your Connection
• More Information

You can use the Avro connector from the CData Sync application to capture data from Avro and move it to any supported destination. To do so, you need to add the connector, authenticate to the connector, and complete your connection.

Add the Avro Connector

To enable Sync to use data from Avro, you first must add the connector, as follows:

1. Open the Connections page of the Sync dashboard.

2. Click Add Connection to open the Select Connectors page.

3. Click the Sources tab and locate the Avro row.

4. Click the Configure Connection icon at the end of that row to open the New Connection page. If the Configure Connection icon is not available, click the Download Connector icon to install the Avro connector. For more information about installing new connectors, see Connections.

Authenticate to Avro

After you add the connector, you need to set the required properties.

On the New Connection page, enter the connection name of your choice.

CData Sync supports the ability to connect to local and cloud-hosted Avro files. Select your connection type below to proceed to the relevant section that contains the connection details.

• Local
• Amazon S3
• Azure Blob Storage
• Azure Data Lake Storage
• Azure Files
• Box
• Dropbox
• FTP
• Google Cloud Storage
• Google Drive
• HDFS
• HTTP
• IBM Object Storage Source
• OneDrive
• Oracle Cloud Storage
• SFTP
• SharePoint Online

Local

To connect to Avro resources on the local drive, specify the following settings:

• Connection Type - Select Local.

• URI - Enter your local directory path, one file per table (for example, C:/FolderPath).

This connection type does not require any authentication settings. Complete the remaining steps for establishing your connection.

Amazon S3

To connect to Avro resources on Amazon S3, use the Amazon S3 connector instead.

Azure Blob Storage

To connect to Avro resources on Azure Blob Storage, specify the following settings:

• Connection Type - Select Azure Blob Storage.

• URI - Enter the name of your container and blob in the URI text box (for example, azureblob://MyContainer/MyBlob).

• Azure Storage Account - Enter the name of your Azure storage account.

See Connecting to Azure Blob Storage for details about choosing your authentication scheme.

Azure Data Lake Storage

To connect to Avro resources on Azure Data Lake Storage, use the Azure Data Lake Storage connector instead.

Azure Files

To connect to Avro resources on Azure File Storage, specify these properties:

• Connection Type - Select Azure Files.

• URI - Enter the name of your Azure file share and the name of the resource (for example, azurefile://FileShare/RemotePath).

• Azure Storage Account - Enter the name of your Azure storage account.

See Connecting to Azure Files for details about choosing your authentication scheme.

Box

To connect to Avro resources on Box, specify these properties:

• Connection Type - Select Box.

• URI - Enter the pathname for the file system and the name of the folder that contains your CSV files (for example, box://Folder1).

• Auth Scheme - Select OAuth. CData Sync provides an embedded OAuth application with which to connect. No additional properties are required.

Complete the remaining steps for establishing your connection.

Dropbox

To connect to Avro resources on Dropbox, specify these properties:

• Connection Type - Select Dropbox.

• URI - Enter the path to a folder that contains Avro files (for example, dropbox://Folder1).

• Auth Scheme - Select OAuth. provides an embedded OAuth application with which to connect. No additional properties are required.

Complete the remaining steps for establishing your connection.

FTP

To connect to Avro resources on an FTP server, specify these properties:

• Connection Type - Select either FTP or FTPS.

• URI - Enter the address of the server, followed by the path to the folder that you want to use as the root folder, as shown in these examples:

  • FTP: ftp://localhost:990/Folder1

  • FTPS: ftps://localhost:990/Folder1

• Auth Scheme - Use the default scheme, Basic.

• User - Enter the username that you use to authenticate to your FTP or FTPS server.

• Password - Enter the password that you use to authenticate to your FTP or FTPS server.

Complete the remaining steps for establishing your connection.

Google Cloud Storage

To connect to Avro resources on Google Cloud Storage, specify these properties:

• Connection Type - Select Google Cloud Storage.

• URI - Enter the name of the bucket and folder that contains your Avro files (for example, gs://Bucket/RemotePath).

• Project Id - Enter the identifier for the project to which you want to connect.

See Connecting to Google Cloud Storage for details about choosing your authentication scheme.

Google Drive

To connect to Avro resources on Google Drive, specify these properties:

• Connection Type - Select Google Drive.

• URI - Enter the pathname for the file system and the name of the folder that contains your Avro files (for example, gdrive://Folder1).

See Connecting to Google Drive for details about choosing your authentication scheme.

Hadoop Distributed File System (HDFS)

To connect to Avro resources on HDFS, specify these properties:

• Connection Type - Select either HDFS or HDFS Secure.

• URI - Enter the path to a folder that contains your Avro files, as shown in these examples:

  • HDFS: webhdfs://Host:Port/RemotePath

  • HDFS Secure: webhdfss://Host:Port/RemotePath

See Connecting to HDFS for details about choosing your authentication scheme.

HTTP

To connect to Avro resources on HTTP Streams, specify these properties:

• Connection Type - Select either HTTP or HTTPS.

• URI - Enter the path to a folder that contains your Avro files, as shown in these examples:

  • HTTP: http://RemoteStream

  • HTTPS: https://RemoteStream

See Connecting to HTTP or HTTPS for details about setting your authentication scheme.

IBM Object Storage Source

To connect to Avro resources on IBM Object Storage, specify these properties:

• Connection Type - Select IBM Object Storage Source.

• URI - Enter the path to a folder that contains your Avro files (for example, ibmobjectstorage://Bucket1/RemotePath).

• Api Key - Enter the API key for your IBM Object Storage source. To obtain this key:

  1. Log in to your IBM Cloud account.

  2. Navigate to the Platform API Keys page.

  3. Click Create an IBM Cloud API Key (middle right corner ) to create a new API key. This action opens a dialog box.

  4. Specify a name for the API key name. Then, click Create.

  Note: Save the API key somewhere because you cannot access it again from the dashboard.

• Region - Enter the region for your IBM instance (for example, eu-gb).

See Connecting to IBM Object Storage for details about choosing your authentication scheme.

OneDrive

To connect to Avro resources on OneDrive, specify these properties:

• Connection Type - Select OneDrive.

• URI - Enter the path to a folder that contains your Avro files (for example, onedrive://RemotePath).

See Connecting to OneDrive for details about choosing your authentication scheme.

Oracle Cloud Storage

To connect to Avro resources on Oracle Cloud Storage, specify these properties:

• Connection Type - Select OracleCloudStorage.

• URI - Enter the path to a folder that contains your Avro files (for example, os://Bucket/RemotePath).

• Region - Enter the hosting region for your S3-like web services.

• Oracle Namespace - Enter your Oracle cloud namespace.

• Auth Scheme - Use the default scheme, HMAC.

• Access Key - Enter your Oracle Cloud access key.

• Secret Key - Enter your Oracle Cloud secret key.

Complete the remaining steps for establishing your connection.

SFTP

To connect to Avro resources on an SFTP server, specify these properties:

• Connection Type - Select SFTP.

• URI - Enter the path to a folder that contains your Avro files (for example, sftp://Server:Port/RemotePath).

• Auth Scheme - Use the default scheme, SFTP.

See Connecting to SFTP for details about choosing your SSH authentication mode.

SharePoint Online

To connect to Avro resources on SharePoint Online, specify these properties:

• Connection Type - Select either SharePoint Online REST or SharePoint Online SOAP.

• URI - Enter the path to a folder that contains your Avro files, as shown in these examples:

  • SharePoint Online REST: sprest://RemotePath

  • SharePoint Online SOAP: spr://RemotePath

• Storage Base URL - Enter the base URL for the SharePoint site. The following are examples of valid URLs:

  • http://server/SharePoint/

  • http://server/Sites/mysite/

  • http://server:90/

Note: If the domain for this option ends in -my (for example, https://bigcorp-my.sharepoint.com), then you might need to connect through OneDrive.

See Connecting to SharePoint Online for details about choosing your authentication scheme.

Complete Your Connection

To complete your connection:

1. Specify the following properties:

   • Data Model - Select the data model to use for parsing Avro documents and generating the database metadata. The default data model is Document.

   • OAuth Grant Type - Select the grant type for the OAuth flow. The default type is CODE.

   • Aggregate Files - Specify whether you want to aggregate all the files in the URI directory into a single result. The default version is False.

2. Define advanced connection settings on the Advanced tab. (In most cases, though, you should not need these settings.)

3. If you authenticate with AzureAD, OAuth, or OAuthPKCE, click Connect to Avro to connect to your Avro account.

4. Click Create & Test to create your connection.

More Information

For more information about interactions between CData Sync and Avro, see Avro Connector for CData Sync.