Connecting to Azure Data Lake Store Destination

Navigate to the Connections page in the Sync App application and select the corresponding icon in the Add Connections panel. If the Azure Data Lake Store Destination icon is not available, click the Add More icon to download and install the Azure Data Lake Store Destination connector from the CData site.

Required properties are listed under the Settings tab. The Advanced tab lists connection properties that are not typically required.

Connecting to Azure DataLakeStorage Gen 1

To connect to a Gen 1 DataLakeStorage account, you should first set the following properties:

• Generation: Set this to 1.
• AzureStorageAccount: Set this to the name of the account.
• AzureTenant: Set this to the tenant Id. See the property for more information on how to acquire this.
• Path: (Optional) Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

Authenticating to Azure DataLakeStorage Gen 1

Gen 1 supports the following authentication methods: Azure Active Directory OAuth (AzureAD) and Managed Service Identity (AzureMSI).

Azure AD

Gen 1 uses OAuth 2.0 Client Credentials Flow in Azure AD for service-to-service authentication.

For this, an Active Directory web application is required. To create one, follow the stops below:

1. Sign in to your Azure Account through the Azure portal (https://portal.azure.com/).
2. Select Azure Active Directory.
3. Select App registrations.
4. Select New application registration.
5. Provide a name and URL for the application. Select Web app for the type of application you want to create.
6. Click on Required permissions and change the required permissions for this app. At least "Azure Data Lake" and "Windows Azure Service Management API" are needed.
7. Click on Key and generate a new key. Add a description, a duration, and take note of the generated key. You won't be able to see it again.

To authenticate against a Gen 1 DataLakeStore account, the following properties are required:

1. AuthScheme -- Set this to "AzureAD".
2. InitiateOAuth -- Set this to "GETANDREFRESH".
3. OAuthClientId -- this is the application Id of the app you created.
4. OAuthClientSecret -- this is the key generated for the app you created.

Azure Service Principal

Azure Service Principal is a connection type that goes through client credentials OAuth flow. For this authentication scheme, an Active Directory web application is required. You can follow the steps above in the AzureAD authentication section to create a custom OAuth app, then you can follow the steps below to assign the app a role with access right to your Azure storage.

1. Log in to https://portal.azure.com.
2. Open the Subscriptions page by searching and selecting the Subscriptions service from the search bar.
3. Select the particular subscription to assign the application to, then open the Access control (IAM) section, and click on the Add role assignment button.
4. Set the role to assign to your created OAuth app. Select the Owner role for the ADLSGen1 Schema. Alternatively, you can also grant access rights by assigning ACLs (Access control lists) to your service principal.

You are then ready to authenticate against a Gen 1 DataLakeStore account after setting the following properties:

1. AuthScheme -- Set this to "AzureServicePrincipal".
2. InitiateOAuth -- Set this to "GETANDREFRESH".
3. OAuthClientId -- this is the application Id of the app you created.
4. OAuthClientSecret -- this is the key generated for the app you created.

Managed Service Identity (MSI)

If you are running Azure Data Lake Store Destination on an Azure VM and want to leverage MSI to connect, set AuthScheme to AzureMSI.

User-Managed Identities

To obtain a token for a managed identity, use the OAuthClientId property to specify the managed identity's "client_id".

It is also necessary to specify OAuthClientId when your VM has multiple user-assigned managed identities.

Connecting to Azure DataLakeStorage Gen 2

To connect to a Gen 2 DataLakeStorage account, you should first set the following properties:

• Generation: Set this to 2.
• AzureStorageAccount: Set this to the name of the storage account.
• FileSystem: Set this to the file system name which will be used for this account. For example, the name of an Azure Blob Container
• Path: (Optional) Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

Authenticating to Azure DataLakeStorage Gen 2

Gen 2 supports the following authentication methods: using an AccessKey and using a Shared Access Signature.

Access Key

To connect using a Shared Access Signature set the AzureAccessKey property and the AuthScheme to AccessKey.

You can obtain an access key for the ADLS Gen2 storage account using the Azure portal:

1. Go to your ADLS Gen2 Storage Account in the Azure portal.
2. Under Settings, select Access keys.
3. Copy the value for one of the available access keys to the AzureAccessKey connection property.

Shared Access Signature (SAS)

To connect using a Shared Access Signature set the AzureSharedAccessSignature property to a valid signature of a resource to connect to and the AuthScheme to SAS. The AzureSharedAccessSignature may be generated with a tool such as Azure Storage Explorer.

Authenticating using Azure AD

Please follow the instructions from the Gen 1 section on using Azure AD since they apply for Gen 2 as well.

Azure Service Principal

Please follow the instructions from the Gen 1 section on using Azure Service Principal since they apply for Gen 2 as well.

Managed Service Identity (MSI)

If you are running Azure Data Lake Store Destination on an Azure VM and want to leverage MSI to connect, set AuthScheme to AzureMSI.

User-Managed Identities

To obtain a token for a managed identity, use the OAuthClientId property to specify the managed identity's "client_id".

It is also necessary to specify OAuthClientId when your VM has multiple user-assigned managed identities.

The connection string properties are the various options that can be used to establish a connection. This section provides a complete list of the options you can configure in the connection string for this provider. Click the links for further details.

For more information on establishing a connection, see Establishing a Connection.

Authentication

---

Property	Description
AuthScheme	The type of authentication to use when connecting to Azure Data Lake Store Destination.
Generation	The generation of the Azure DataLakeStore account you are authenticating to.
FileSystem	This property specifies the name of the FileSystem which will be used in a Gen 2 account.

Azure Authentication

---

Property	Description
AzureStorageAccount	The name of your Azure storage account.
AzureAccessKey	The storage key associated with your Azure account.
AzureSharedAccessSignature	A shared access key signature that may be used for authentication.
AzureTenant	The Microsoft Online tenant being used to access data. If not specified, your default tenant is used.
AzureEnvironment	The Azure Environment to use when establishing a connection.

OAuth

---

Property	Description
OAuthClientId	The client Id assigned when you register your application with an OAuth authorization server.
OAuthClientSecret	The client secret assigned when you register your application with an OAuth authorization server.
OAuthGrantType	The grant type for the OAuth flow.

JWT OAuth

---

Property	Description
OAuthJWTCert	The JWT Certificate store.
OAuthJWTCertType	The type of key store containing the JWT Certificate.
OAuthJWTCertPassword	The password for the OAuth JWT certificate.
OAuthJWTCertSubject	The subject of the OAuth JWT certificate.

SSL

---

Property	Description
SSLServerCert	The certificate to be accepted from the server when connecting using TLS/SSL.

Firewall

---

Property	Description
FirewallType	The protocol used by a proxy-based firewall.
FirewallServer	The name or IP address of a proxy-based firewall.
FirewallPort	The TCP port for a proxy-based firewall.
FirewallUser	The user name to use to authenticate with a proxy-based firewall.
FirewallPassword	A password used to authenticate to a proxy-based firewall.

Proxy

---

Property	Description
ProxyAutoDetect	This indicates whether to use the system proxy settings or not. This takes precedence over other proxy settings, so you'll need to set ProxyAutoDetect to FALSE in order use custom proxy settings.
ProxyServer	The hostname or IP address of a proxy to route HTTP traffic through.
ProxyPort	The TCP port the ProxyServer proxy is running on.
ProxyAuthScheme	The authentication type to use to authenticate to the ProxyServer proxy.
ProxyUser	A user name to be used to authenticate to the ProxyServer proxy.
ProxyPassword	A password to be used to authenticate to the ProxyServer proxy.
ProxySSLType	The SSL type to use when connecting to the ProxyServer proxy.
ProxyExceptions	A semicolon separated list of destination hostnames or IPs that are exempt from connecting through the ProxyServer .

Logging

---

Property	Description
LogModules	Core modules to be included in the log file.

Schema

---

Property	Description
Location	A path to the directory that contains the schema files defining tables, views, and stored procedures.
BrowsableSchemas	This property restricts the schemas reported to a subset of the available schemas. For example, BrowsableSchemas=SchemaA,SchemaB,SchemaC.
Tables	This property restricts the tables reported to a subset of the available tables. For example, Tables=TableA,TableB,TableC.
Views	Restricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC.

Data Formatting

---

Property	Description
CustomDelimiter	Specifies the character that is used as a delimiter for data in CSV files. The default delimiter is the comma (,).
QuoteEscapeCharacter	Determines the character which will be used to escape quotes.
QuoteCharacter	Determines the character which will be used to quote values in CSV file.
DateTimeFormat	This setting specifies in which format the datetime values will be written to for CSV files.

Miscellaneous

---

Property	Description
Charset	Specifies the session character set for encoding and decoding character data transferred to and from the Azure Data Lake Store Destination file. The default value is UTF-8.
ChunkSize	Specifies the chunk size (in Mb) for multi part upload.
FileFormat	Specifies the file format.
IncludeColumnsHeaders	Whether to add column names in the first line of the replicated CSV file.
InsertMode	Specifies the mode for inserting data into CSV files.
MaxRows	Limits the number of rows returned when no aggregation or GROUP BY is used in the query. This takes precedence over LIMIT clauses.
MaxThreads	Specifies the number of concurrent requests.
Other	These hidden properties are used only in specific use cases.
PseudoColumns	This property indicates whether or not to include pseudo columns as columns to the table.
Timeout	The value in seconds until the timeout error is thrown, canceling the operation.
UserDefinedViews	A filepath pointing to the JSON configuration file containing your custom views.

This section provides a complete list of the Authentication properties you can configure in the connection string for this provider.

---

Property	Description
AuthScheme	The type of authentication to use when connecting to Azure Data Lake Store Destination.
Generation	The generation of the Azure DataLakeStore account you are authenticating to.
FileSystem	This property specifies the name of the FileSystem which will be used in a Gen 2 account.

The type of authentication to use when connecting to Azure Data Lake Store Destination.

Remarks

• Auto: Lets the CData Sync App determine what to do based on other connection properties.
• AzureAD: Set this to perform Azure Active Directory OAuth authentication.
• AzureServicePrincipal: Set this to authenticate as an Azure Service Principal.
• AzureServicePrincipalCert: Set this to authenticate as an Azure Service Principal using a Certificate.
• AzureMSI: Set this to automatically obtain Managed Service Identity credentials when running on an Azure VM.
• AccessKey: Set this to authenticate the calls to the API by setting the AccessKey to your Access Key. This is applicable only when the Generation ist set to 2.
• SAS: Set this to authenticate the calls to the API by setting the SharedAccessSignature to your Shared Access Signature (SAS). This is applicable only when the Generation ist set to 2.

The generation of the Azure DataLakeStore account you are authenticating to.

Remarks

The generation of the Azure DataLakeStore account you are authenticating to.

This property specifies the name of the FileSystem which will be used in a Gen 2 account.

Remarks

This property specifies the name of the FileSystem which will be used in a Gen 2 account.

This section provides a complete list of the Azure Authentication properties you can configure in the connection string for this provider.

---

Property	Description
AzureStorageAccount	The name of your Azure storage account.
AzureAccessKey	The storage key associated with your Azure account.
AzureSharedAccessSignature	A shared access key signature that may be used for authentication.
AzureTenant	The Microsoft Online tenant being used to access data. If not specified, your default tenant is used.
AzureEnvironment	The Azure Environment to use when establishing a connection.

The name of your Azure storage account.

Remarks

The name of your Azure storage account.

The storage key associated with your Azure account.

Remarks

The storage key associated with your Azure Data Lake Store Destination account. You can retrieve it as follows:

1. Sign into the azure portal with the credentials for your root account. (https://portal.azure.com/)
2. Click on storage accounts and select the storage account you want to use.
3. Under settings, click Access keys.
4. Your storage account name and key will be displayed on that page.

A shared access key signature that may be used for authentication.

Remarks

A shared access signature. You can create one by following these steps:

1. Sign into the azure portal with the credentials for your root account. (https://portal.azure.com/)
2. Click on storage accounts and select the storage account you want to use.
3. Under settings, click Shared Access Signature.
4. Set the permissions and when the token will expire
5. Click Generate SAS can copy the token.

The Microsoft Online tenant being used to access data. If not specified, your default tenant is used.

Remarks

The Microsoft Online tenant being used to access data. For instance, contoso.onmicrosoft.com. Alternatively, specify the tenant Id. This value is the directory Id in the Azure Portal > Azure Active Directory > Properties.

Typically it is not necessary to specify the Tenant. This can be automatically determined by Microsoft when using the OAuthGrantType set to CODE (default). However, it may fail in the case that the user belongs to multiple tenants. For instance, if an Admin of domain A invites a user of domain B to be a guest user. The user will now belong to both tenants. It is a good practice to specify the Tenant, although in general things should normally work without having to specify it.

The AzureTenant is required when setting OAuthGrantType to CLIENT. When using client credentials, there is no user context. The credentials are taken from the context of the app itself. While Microsoft still allows client credentials to be obtained without specifying which Tenant, it has a much lower probability of picking the specific tenant you want to work with. For this reason, we require AzureTenant to be explicitly stated for all client credentials connections to ensure you get credentials that are applicable for the domain you intend to connect to.

The Azure Environment to use when establishing a connection.

Remarks

In most cases, leaving the environment set to global will work. However, if your Azure Account has been added to a different environment, the AzureEnvironment may be used to specify which environment. The available values are GLOBAL, CHINA, USGOVT, USGOVTDOD.

This section provides a complete list of the OAuth properties you can configure in the connection string for this provider.

---

Property	Description
OAuthClientId	The client Id assigned when you register your application with an OAuth authorization server.
OAuthClientSecret	The client secret assigned when you register your application with an OAuth authorization server.
OAuthGrantType	The grant type for the OAuth flow.

The client Id assigned when you register your application with an OAuth authorization server.

Remarks

As part of registering an OAuth application, you will receive the OAuthClientId value, sometimes also called a consumer key, and a client secret, the OAuthClientSecret.

The client secret assigned when you register your application with an OAuth authorization server.

Remarks

As part of registering an OAuth application, you will receive the OAuthClientId, also called a consumer key. You will also receive a client secret, also called a consumer secret. Set the client secret in the OAuthClientSecret property.

The grant type for the OAuth flow.

Remarks

The following options are available: CLIENT,CODE

This section provides a complete list of the JWT OAuth properties you can configure in the connection string for this provider.

---

Property	Description
OAuthJWTCert	The JWT Certificate store.
OAuthJWTCertType	The type of key store containing the JWT Certificate.
OAuthJWTCertPassword	The password for the OAuth JWT certificate.
OAuthJWTCertSubject	The subject of the OAuth JWT certificate.

The JWT Certificate store.

Remarks

The name of the certificate store for the client certificate.

The OAuthJWTCertType field specifies the type of the certificate store specified by OAuthJWTCert. If the store is password protected, specify the password in OAuthJWTCertPassword.

OAuthJWTCert is used in conjunction with the OAuthJWTCertSubject field in order to specify client certificates. If OAuthJWTCert has a value, and OAuthJWTCertSubject is set, a search for a certificate is initiated. Please refer to the OAuthJWTCertSubject field for details.

Designations of certificate stores are platform-dependent.

The following are designations of the most common User and Machine certificate stores in Windows:

MY	A certificate store holding personal certificates with their associated private keys.
CA	Certifying authority certificates.
ROOT	Root certificates.
SPC	Software publisher certificates.

In Java, the certificate store normally is a file containing certificates and optional private keys.

When the certificate store type is PFXFile, this property must be set to the name of the file. When the type is PFXBlob, the property must be set to the binary contents of a PFX file (i.e. PKCS12 certificate store).

The type of key store containing the JWT Certificate.

Remarks

This property can take one of the following values:

USER	For Windows, this specifies that the certificate store is a certificate store owned by the current user. Note: This store type is not available in Java.
MACHINE	For Windows, this specifies that the certificate store is a machine store. Note: this store type is not available in Java.
PFXFILE	The certificate store is the name of a PFX (PKCS12) file containing certificates.
PFXBLOB	The certificate store is a string (base-64-encoded) representing a certificate store in PFX (PKCS12) format.
JKSFILE	The certificate store is the name of a Java key store (JKS) file containing certificates. Note: this store type is only available in Java.
JKSBLOB	The certificate store is a string (base-64-encoded) representing a certificate store in Java key store (JKS) format. Note: this store type is only available in Java.
PEMKEY_FILE	The certificate store is the name of a PEM-encoded file that contains a private key and an optional certificate.
PEMKEY_BLOB	The certificate store is a string (base64-encoded) that contains a private key and an optional certificate.
PUBLIC_KEY_FILE	The certificate store is the name of a file that contains a PEM- or DER-encoded public key certificate.
PUBLIC_KEY_BLOB	The certificate store is a string (base-64-encoded) that contains a PEM- or DER-encoded public key certificate.
SSHPUBLIC_KEY_FILE	The certificate store is the name of a file that contains an SSH-style public key.
SSHPUBLIC_KEY_BLOB	The certificate store is a string (base-64-encoded) that contains an SSH-style public key.
P7BFILE	The certificate store is the name of a PKCS7 file containing certificates.
PPKFILE	The certificate store is the name of a file that contains a PPK (PuTTY Private Key).
XMLFILE	The certificate store is the name of a file that contains a certificate in XML format.
XMLBLOB	The certificate store is a string that contains a certificate in XML format.

The password for the OAuth JWT certificate.

Remarks

If the certificate store is of a type that requires a password, this property is used to specify that password in order to open the certificate store.

The subject of the OAuth JWT certificate.

Remarks

When loading a certificate the subject is used to locate the certificate in the store.

If an exact match is not found, the store is searched for subjects containing the value of the property.

If a match is still not found, the property is set to an empty string, and no certificate is selected.

The special value "*" picks the first certificate in the certificate store.

The certificate subject is a comma separated list of distinguished name fields and values. For instance "CN=www.server.com, OU=test, C=US, [email protected]". Common fields and their meanings are displayed below.

Field	Meaning
CN	Common Name. This is commonly a host name like www.server.com.
O	Organization
OU	Organizational Unit
L	Locality
S	State
C	Country
E	Email Address

If a field value contains a comma it must be quoted.

This section provides a complete list of the SSL properties you can configure in the connection string for this provider.

---

Property	Description
SSLServerCert	The certificate to be accepted from the server when connecting using TLS/SSL.

The certificate to be accepted from the server when connecting using TLS/SSL.

Remarks

If using a TLS/SSL connection, this property can be used to specify the TLS/SSL certificate to be accepted from the server. Any other certificate that is not trusted by the machine is rejected.

This property can take the following forms:

Description	Example
A full PEM Certificate (example shortened for brevity)	-----BEGIN CERTIFICATE----- MIIChTCCAe4CAQAwDQYJKoZIhv......Qw== -----END CERTIFICATE-----
A path to a local file containing the certificate	C:\cert.cer
The public key (example shortened for brevity)	-----BEGIN RSA PUBLIC KEY----- MIGfMA0GCSq......AQAB -----END RSA PUBLIC KEY-----
The MD5 Thumbprint (hex values can also be either space or colon separated)	ecadbdda5a1529c58a1e9e09828d70e4
The SHA1 Thumbprint (hex values can also be either space or colon separated)	34a929226ae0819f2ec14b4a3d904f801cbb150d

If not specified, any certificate trusted by the machine is accepted.

Use '*' to signify to accept all certificates. Note that this is not recommended due to security concerns.

This section provides a complete list of the Firewall properties you can configure in the connection string for this provider.

---

Property	Description
FirewallType	The protocol used by a proxy-based firewall.
FirewallServer	The name or IP address of a proxy-based firewall.
FirewallPort	The TCP port for a proxy-based firewall.
FirewallUser	The user name to use to authenticate with a proxy-based firewall.
FirewallPassword	A password used to authenticate to a proxy-based firewall.

The protocol used by a proxy-based firewall.

Remarks

This property specifies the protocol that the Sync App will use to tunnel traffic through the FirewallServer proxy. Note that by default, the Sync App connects to the system proxy; to disable this behavior and connect to one of the following proxy types, set ProxyAutoDetect to false.

Type	Default Port	Description
TUNNEL	80	When this is set, the Sync App opens a connection to Azure Data Lake Store Destination and traffic flows back and forth through the proxy.
SOCKS4	1080	When this is set, the Sync App sends data through the SOCKS 4 proxy specified by FirewallServer and FirewallPort and passes the FirewallUser value to the proxy, which determines if the connection request should be granted.
SOCKS5	1080	When this is set, the Sync App sends data through the SOCKS 5 proxy specified by FirewallServer and FirewallPort. If your proxy requires authentication, set FirewallUser and FirewallPassword to credentials the proxy recognizes.

To connect to HTTP proxies, use ProxyServer and ProxyPort. To authenticate to HTTP proxies, use ProxyAuthScheme, ProxyUser, and ProxyPassword.

The name or IP address of a proxy-based firewall.

Remarks

This property specifies the IP address, DNS name, or host name of a proxy allowing traversal of a firewall. The protocol is specified by FirewallType: Use FirewallServer with this property to connect through SOCKS or do tunneling. Use ProxyServer to connect to an HTTP proxy.

Note that the Sync App uses the system proxy by default. To use a different proxy, set ProxyAutoDetect to false.

The TCP port for a proxy-based firewall.

Remarks

This specifies the TCP port for a proxy allowing traversal of a firewall. Use FirewallServer to specify the name or IP address. Specify the protocol with FirewallType.

The user name to use to authenticate with a proxy-based firewall.

Remarks

The FirewallUser and FirewallPassword properties are used to authenticate against the proxy specified in FirewallServer and FirewallPort, following the authentication method specified in FirewallType.

A password used to authenticate to a proxy-based firewall.

Remarks

This property is passed to the proxy specified by FirewallServer and FirewallPort, following the authentication method specified by FirewallType.

This section provides a complete list of the Proxy properties you can configure in the connection string for this provider.

---

Property	Description
ProxyAutoDetect	This indicates whether to use the system proxy settings or not. This takes precedence over other proxy settings, so you'll need to set ProxyAutoDetect to FALSE in order use custom proxy settings.
ProxyServer	The hostname or IP address of a proxy to route HTTP traffic through.
ProxyPort	The TCP port the ProxyServer proxy is running on.
ProxyAuthScheme	The authentication type to use to authenticate to the ProxyServer proxy.
ProxyUser	A user name to be used to authenticate to the ProxyServer proxy.
ProxyPassword	A password to be used to authenticate to the ProxyServer proxy.
ProxySSLType	The SSL type to use when connecting to the ProxyServer proxy.
ProxyExceptions	A semicolon separated list of destination hostnames or IPs that are exempt from connecting through the ProxyServer .

This indicates whether to use the system proxy settings or not. This takes precedence over other proxy settings, so you'll need to set ProxyAutoDetect to FALSE in order use custom proxy settings.

Remarks

This takes precedence over other proxy settings, so you'll need to set ProxyAutoDetect to FALSE in order use custom proxy settings.

To connect to an HTTP proxy, see ProxyServer. For other proxies, such as SOCKS or tunneling, see FirewallType.

The hostname or IP address of a proxy to route HTTP traffic through.

Remarks

The hostname or IP address of a proxy to route HTTP traffic through. The Sync App can use the HTTP, Windows (NTLM), or Kerberos authentication types to authenticate to an HTTP proxy.

If you need to connect through a SOCKS proxy or tunnel the connection, see FirewallType.

By default, the Sync App uses the system proxy. If you need to use another proxy, set ProxyAutoDetect to false.

The TCP port the ProxyServer proxy is running on.

Remarks

The port the HTTP proxy is running on that you want to redirect HTTP traffic through. Specify the HTTP proxy in ProxyServer. For other proxy types, see FirewallType.

The authentication type to use to authenticate to the ProxyServer proxy.

Remarks

This value specifies the authentication type to use to authenticate to the HTTP proxy specified by ProxyServer and ProxyPort.

Note that the Sync App will use the system proxy settings by default, without further configuration needed; if you want to connect to another proxy, you will need to set ProxyAutoDetect to false, in addition to ProxyServer and ProxyPort. To authenticate, set ProxyAuthScheme and set ProxyUser and ProxyPassword, if needed.

The authentication type can be one of the following:

• BASIC: The Sync App performs HTTP BASIC authentication.
• DIGEST: The Sync App performs HTTP DIGEST authentication.
• NEGOTIATE: The Sync App retrieves an NTLM or Kerberos token based on the applicable protocol for authentication.
• PROPRIETARY: The Sync App does not generate an NTLM or Kerberos token. You must supply this token in the Authorization header of the HTTP request.

If you need to use another authentication type, such as SOCKS 5 authentication, see FirewallType.

A user name to be used to authenticate to the ProxyServer proxy.

Remarks

The ProxyUser and ProxyPassword options are used to connect and authenticate against the HTTP proxy specified in ProxyServer.

You can select one of the available authentication types in ProxyAuthScheme. If you are using HTTP authentication, set this to the user name of a user recognized by the HTTP proxy. If you are using Windows or Kerberos authentication, set this property to a user name in one of the following formats:

user@domain
domain\user

A password to be used to authenticate to the ProxyServer proxy.

Remarks

This property is used to authenticate to an HTTP proxy server that supports NTLM (Windows), Kerberos, or HTTP authentication. To specify the HTTP proxy, you can set ProxyServer and ProxyPort. To specify the authentication type, set ProxyAuthScheme.

If you are using HTTP authentication, additionally set ProxyUser and ProxyPassword to HTTP proxy.

If you are using NTLM authentication, set ProxyUser and ProxyPassword to your Windows password. You may also need these to complete Kerberos authentication.

For SOCKS 5 authentication or tunneling, see FirewallType.

By default, the Sync App uses the system proxy. If you want to connect to another proxy, set ProxyAutoDetect to false.

The SSL type to use when connecting to the ProxyServer proxy.

Remarks

This property determines when to use SSL for the connection to an HTTP proxy specified by ProxyServer. This value can be AUTO, ALWAYS, NEVER, or TUNNEL. The applicable values are the following:

AUTO	Default setting. If the URL is an HTTPS URL, the Sync App will use the TUNNEL option. If the URL is an HTTP URL, the component will use the NEVER option.
ALWAYS	The connection is always SSL enabled.
NEVER	The connection is not SSL enabled.
TUNNEL	The connection is through a tunneling proxy. The proxy server opens a connection to the remote host and traffic flows back and forth through the proxy.

A semicolon separated list of destination hostnames or IPs that are exempt from connecting through the ProxyServer .

Remarks

The ProxyServer is used for all addresses, except for addresses defined in this property. Use semicolons to separate entries.

Note that the Sync App uses the system proxy settings by default, without further configuration needed; if you want to explicitly configure proxy exceptions for this connection, you need to set ProxyAutoDetect = false, and configure ProxyServer and ProxyPort. To authenticate, set ProxyAuthScheme and set ProxyUser and ProxyPassword, if needed.

This section provides a complete list of the Logging properties you can configure in the connection string for this provider.

---

Property	Description
LogModules	Core modules to be included in the log file.

Core modules to be included in the log file.

Remarks

Only the modules specified (separated by ';') will be included in the log file. By default all modules are included.

See the Logging page for an overview.

This section provides a complete list of the Schema properties you can configure in the connection string for this provider.

---

Property	Description
Location	A path to the directory that contains the schema files defining tables, views, and stored procedures.
BrowsableSchemas	This property restricts the schemas reported to a subset of the available schemas. For example, BrowsableSchemas=SchemaA,SchemaB,SchemaC.
Tables	This property restricts the tables reported to a subset of the available tables. For example, Tables=TableA,TableB,TableC.
Views	Restricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC.

A path to the directory that contains the schema files defining tables, views, and stored procedures.

Remarks

The path to a directory which contains the schema files for the Sync App (.rsd files for tables and views, .rsb files for stored procedures). The folder location can be a relative path from the location of the executable. The Location property is only needed if you want to customize definitions (for example, change a column name, ignore a column, and so on) or extend the data model with new tables, views, or stored procedures.

If left unspecified, the default location is "%APPDATA%\\CData\\AZUREDATALAKESTOREDestination Data Provider\\Schema" with %APPDATA% being set to the user's configuration directory:

Platform	%APPDATA%
Windows	The value of the APPDATA environment variable
Linux	~/.config

This property restricts the schemas reported to a subset of the available schemas. For example, BrowsableSchemas=SchemaA,SchemaB,SchemaC.

Remarks

Listing the schemas from databases can be expensive. Providing a list of schemas in the connection string improves the performance.

This property restricts the tables reported to a subset of the available tables. For example, Tables=TableA,TableB,TableC.

Remarks

Listing the tables from some databases can be expensive. Providing a list of tables in the connection string improves the performance of the Sync App.

This property can also be used as an alternative to automatically listing views if you already know which ones you want to work with and there would otherwise be too many to work with.

Specify the tables you want in a comma-separated list. Each table should be a valid SQL identifier with any special characters escaped using square brackets, double-quotes or backticks. For example, Tables=TableA,[TableB/WithSlash],WithCatalog.WithSchema.`TableC With Space`.

Note that when connecting to a data source with multiple schemas or catalogs, you will need to provide the fully qualified name of the table in this property, as in the last example here, to avoid ambiguity between tables that exist in multiple catalogs or schemas.

Restricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC.

Remarks

Listing the views from some databases can be expensive. Providing a list of views in the connection string improves the performance of the Sync App.

This property can also be used as an alternative to automatically listing views if you already know which ones you want to work with and there would otherwise be too many to work with.

Specify the views you want in a comma-separated list. Each view should be a valid SQL identifier with any special characters escaped using square brackets, double-quotes or backticks. For example, Views=ViewA,[ViewB/WithSlash],WithCatalog.WithSchema.`ViewC With Space`.

Note that when connecting to a data source with multiple schemas or catalogs, you will need to provide the fully qualified name of the table in this property, as in the last example here, to avoid ambiguity between tables that exist in multiple catalogs or schemas.

This section provides a complete list of the Data Formatting properties you can configure in the connection string for this provider.

---

Property	Description
CustomDelimiter	Specifies the character that is used as a delimiter for data in CSV files. The default delimiter is the comma (,).
QuoteEscapeCharacter	Determines the character which will be used to escape quotes.
QuoteCharacter	Determines the character which will be used to quote values in CSV file.
DateTimeFormat	This setting specifies in which format the datetime values will be written to for CSV files.

Specifies the character that is used as a delimiter for data in CSV files. The default delimiter is the comma (,).

Remarks

Specifies the character that is used as a delimiter for data in CSV files. The default delimiter is the comma (,).

Determines the character which will be used to escape quotes.

Remarks

Determines the character which will be used to escape quotes.

Determines the character which will be used to quote values in CSV file.

Remarks

Determines the character which will be used to quote values in CSV file.

Note: This property works only for CSV files. Set this property to "NONE" if you want to insert fields in a CSV file without quoting them.

This setting specifies in which format the datetime values will be written to for CSV files.

Remarks

The format should follow a specified pattern:

1. G - Era designator.
2. y - Year.
3. M - Month in year.
4. w - Results in week in year.
5. W - Results in week in month.
6. D - Gives the day count in the year.
7. d - Day of the month.
8. F - Day of the week in month.
9. E - Day name in the week.
10. u - Day number of week where 1 represent Monday, 2 Tuesday and so on.
11. a - AM or PM marker.
12. H - Hour in the day (0-23).
13. h - Hour in am/pm for 12 hour format (1-12).
14. K - Hour in am/pm for 12 hour format (0-11).
15. k - Hour in the day (1-24).
16. m - Minute in the hour.
17. s - Second in the minute.
18. S - Millisecond in the minute.
19. z - Timezone.
20. Z - Timezone offset in hours (RFC pattern).
21. X - Timezone offset in ISO format.

This section provides a complete list of the Miscellaneous properties you can configure in the connection string for this provider.

---

Property	Description
Charset	Specifies the session character set for encoding and decoding character data transferred to and from the Azure Data Lake Store Destination file. The default value is UTF-8.
ChunkSize	Specifies the chunk size (in Mb) for multi part upload.
FileFormat	Specifies the file format.
IncludeColumnsHeaders	Whether to add column names in the first line of the replicated CSV file.
InsertMode	Specifies the mode for inserting data into CSV files.
MaxRows	Limits the number of rows returned when no aggregation or GROUP BY is used in the query. This takes precedence over LIMIT clauses.
MaxThreads	Specifies the number of concurrent requests.
Other	These hidden properties are used only in specific use cases.
PseudoColumns	This property indicates whether or not to include pseudo columns as columns to the table.
Timeout	The value in seconds until the timeout error is thrown, canceling the operation.
UserDefinedViews	A filepath pointing to the JSON configuration file containing your custom views.

Specifies the session character set for encoding and decoding character data transferred to and from the Azure Data Lake Store Destination file. The default value is UTF-8.

Remarks

Specifies the session character set for encoding and decoding character data transferred to and from the Azure Data Lake Store Destination file. The default value is UTF-8.

Specifies the chunk size (in Mb) for multi part upload.

Remarks

Specifies the chunk size (in Mb) for multi part upload. Set the value to -1 if you want to disable the multi part upload.

Specifies the file format.

Remarks

Specifies the file format.

Whether to add column names in the first line of the replicated CSV file.

Remarks

When this property is set to True, the Sync App will add column names to the top for the replicated CSV file.

Specifies the mode for inserting data into CSV files.

Remarks

There are two modes available for inserting data to CSV file:

• FilePerBatch: A new CSV file will be created for every batch. The name of the file will have the format "yyyyMMddhhmmssSSS_batchId" where the date time represents the beginning of the transaction.
• SingleFile: In this mode everything will be inserted in a single CSV file.

Limits the number of rows returned when no aggregation or GROUP BY is used in the query. This takes precedence over LIMIT clauses.

Remarks

Limits the number of rows returned when no aggregation or GROUP BY is used in the query. This takes precedence over LIMIT clauses.

Specifies the number of concurrent requests.

Remarks

This property allows you to issue multiple requests simultaneously, thereby improving performance.

These hidden properties are used only in specific use cases.

Remarks

The properties listed below are available for specific use cases. Normal driver use cases and functionality should not require these properties.

Specify multiple properties in a semicolon-separated list.

Integration and Formatting

DefaultColumnSize	Sets the default length of string fields when the data source does not provide column length in the metadata. The default value is 2000.
ConvertDateTimeToGMT	Determines whether to convert date-time values to GMT, instead of the local time of the machine.
RecordToFile=filename	Records the underlying socket data transfer to the specified file.

This property indicates whether or not to include pseudo columns as columns to the table.

Remarks

This setting is particularly helpful in Entity Framework, which does not allow you to set a value for a pseudo column unless it is a table column. The value of this connection setting is of the format "Table1=Column1, Table1=Column2, Table2=Column3". You can use the "*" character to include all tables and all columns; for example, "*=*".

The value in seconds until the timeout error is thrown, canceling the operation.

Remarks

If Timeout = 0, operations do not time out. The operations run until they complete successfully or until they encounter an error condition.

If Timeout expires and the operation is not yet complete, the Sync App throws an exception.

A filepath pointing to the JSON configuration file containing your custom views.

Remarks

User Defined Views are defined in a JSON-formatted configuration file called UserDefinedViews.json. The Sync App automatically detects the views specified in this file.

You can also have multiple view definitions and control them using the UserDefinedViews connection property. When you use this property, only the specified views are seen by the Sync App.

This User Defined View configuration file is formatted as follows:

• Each root element defines the name of a view.
• Each root element contains a child element, called query, which contains the custom SQL query for the view.

For example:

{
	"MyView": {
		"query": "SELECT * FROM NorthwindOData WHERE MyColumn = 'value'"
	},
	"MyView2": {
		"query": "SELECT * FROM MyTable WHERE Id IN (1,2,3)"
	}
}

Use the UserDefinedViews connection property to specify the location of your JSON configuration file. For example:

"UserDefinedViews", "C:\\Users\\yourusername\\Desktop\\tmp\\UserDefinedViews.json"