The CData Sync App provides a straightforward way to continuously pipeline your Apache Kafka data to any database, data lake, or data warehouse, making it easily available for Analytics, Reporting, AI, and Machine Learning.
The Apache Kafka connector can be used from the CData Sync application to pull data from Apache Kafka and move it to any of the supported destinations.
The Sync App leverages the Apache Kafka client libraries to enable bidirectional access to Kafka topics.
For required properties, see the Settings tab.
For connection properties that are not typically required, see the Advanced tab.
To specify the address of your Apache Kafka server, use the BootstrapServers parameter.
By default, the Sync App communicates with the data source in PLAINTEXT, which means that all data is sent unencrypted. If you want communication to be encrypted:
Note: Proxy settings like ProxyServer and firewall settings like FirewallServer do not affect the connection to the Apache Kafka broker because the Sync App connects to Apache Kafka interally using the official libraries, which do not support proxies. These options are only used when the Sync App connects to the schema registry. For details, see Extracting Metadata From Topics.
To authenticate anonymously, set this property:
Set these properties:
Note: Both your IdP and your Apache Kafka broker can be configured in a wide variety of ways. You should be aware of the requirements on both ends when connecting with the Sync App. For further information, refer to the documentation for your Apache Kafka broker and your IdP.
Set these properties to authenticate:
In addition to the scheme-specific properties covered below, all connections to Azure require you to set the following properties:
Note: Microsoft has rebranded Azure AD as Entra ID. In topics that require the user to interact with the Entra ID Admin site, we use the same names Microsoft does. However, there are still CData connection properties whose names or values reference "Azure AD".
Microsoft Entra ID is a multi-tenant, cloud-based identity and access management platform. It supports OAuth-based authentication flows that enable the driver to access Apache Kafka endpoints securely.
Authentication to Entra ID via a web application always requires that you first create and register a custom OAuth application. This enables your application to define its own redirect URI, manage credential scope, and comply with organization-specific security policies.
For full instructions on how to create and register a custom OAuth application, see Creating an Entra ID (Azure AD) Application.
After setting AuthScheme to AzureAD, the steps to authenticate vary, depending on the environment. For details on how to connect from desktop applications, web-based workflows, or headless systems, see the following sections.
Note: Microsoft has rebranded Azure AD as Entra ID. In topics that require the user to interact with the Entra ID Admin site, we use the same names Microsoft does. However, there are still CData connection properties whose names or values reference "Azure AD".
Azure Service Principal is role-based application-based authentication. This means that authentication is done per application, rather than per user.
All tasks taken on by the application are executed without a default user context, but based on the assigned roles.
The application access to the resources is controlled through the assigned roles' permissions.
For information about how to set up Azure Service Principal authentication, see Creating a Service Principal App in Entra ID (Azure AD).
If you are running Apache Kafka on an Azure VM and want to automatically obtain Managed Service Identity (MSI) credentials to connect, set AuthScheme to AzureMSI.
If your VM has multiple user-assigned managed identities, you must also specify OAuthClientId.
The Sync App supports password-based authentication using shared-access signatures. After you create the shared secret, set these properties:
The Sync App supports connecting to Google Managed Service for Apache Kafka (GCP Kafka). GCP Kafka uses OAuth authentication and supports service accounts, GCP instance accounts, and Workload Identity Federation.
All connections to GCP Kafka must set these properties:
After you set the appropriate scheme-specific properties (described below), you are ready to connect.
GCP Kafka supports authenticating as a Google service account. This service account must have the Managed Kafka Client role.
Provide the service account credentials to the Sync App with these properties:
To connect using a GCP instance account, set this property:
To connect using Workload Identity Federation credentials, set these properties:
This section details a selection of advanced features of the Apache Kafka Sync App.
The Sync App supports the use of user defined views, virtual tables whose contents are decided by a pre-configured user defined query. These views are useful when you cannot directly control queries being issued to the drivers. For an overview of creating and configuring custom views, see User Defined Views .
Use SSL Configuration to adjust how Sync App handles TLS/SSL certificate negotiations. You can choose from various certificate formats;. For further information, see the SSLServerCert property under "Connection String Options" .
Configure the Sync App for compliance with Firewall and Proxy, including Windows proxies and HTTP proxies. You can also set up tunnel connections.
For further information, see Query Processing.
To enable TLS, set UseSSL to True.
With this configuration, the Sync App attempts to negotiate TLS with the server. The server certificate is validated against the default system trusted certificate store. You can override how the certificate gets validated using the SSLServerCert connection property.
To specify another certificate, see the SSLServerCert connection property.
The Apache Kafka Sync App also supports setting client certificates. Set the following to connect using a client certificate.
To authenticate to an HTTP proxy, set the following:
Set the following properties:
SELECTing from a topic returns existing messages on the topic, as well as live messages posted before the number of seconds specified by the ReadDuration have elapsed.
Stored Procedures are function-like interfaces to Apache Kafka. They can be used to create schema files, commit messages, and more.
Connections that the Sync App makes to Apache Kafka are always part of a consumer group. You can control the consumer group by setting a value for the ConsumerGroupId connection property. Using the same consumer group ID across multiple connections puts those connections into the same consumer group. The Sync App generates a random consumer group ID if one is not provided.
All members of a consumer group share an offset that determines what messages are read next within each topic and partition. The Sync App supports two ways of updating the offset:
If there is no existing offset, the Sync App uses the OffsetResetStrategy to determine what the offset should be. This may happen if the broker does not recognize the consumer group or if the consumer group never committed an offset.
The Sync App supports reading bulk messages from topics using the CSV, JSON, or XML SerializationFormat.
When the Sync App reads CSV data like the following block, it splits the CSV and outputs each line as a separate row.
The values of other columns like the partition, timestamp, and key are the same across each row.
"1","alpha" "2","beta" "3","gamma"
Bulk messages are not supported for key values. When MessageKeyType is set to a bulk format, the Sync App reads only the first row of the key and ignore the rest. For example, when the Sync App reads the above CSV data as a message key, the entries on the alpha row are repeated across every bulk row from the message value. The entries on the beta and gamma rows are lost.
Apache Kafka does not natively support bulk messages, which can lead to rows being skipped in some circumstances. For example:
Consider what happens if this procedure is performed on the following topic.
The first connection consumes all rows from the first message and one row from the second.
However, the Sync App has no way to report to Apache Kafka that only part of the second message was read.
This means that step 3 commits the offset 3 and the second connection starts on row 5, skipping row 4.
"row 1" "row 2" /* End of message 1 */ "row 3" "row 4" /* End of message 2 */ "row 5" "row 6" /* End of message 3 */
Stored procedures are function-like interfaces that extend the functionality of the Sync App beyond simple SELECT/INSERT operations with Apache Kafka.
Stored procedures accept a list of parameters, perform their intended function, and then return any relevant response data from Apache Kafka, along with an indication of whether the procedure succeeded or failed.
| Name | Description |
| CommitOffset | Commits the current set of message offsets for a specified consumer group. This stored procedure ensures that consumed messages are marked as processed within Kafka, preventing the same records from being read again when the consumer resumes operation. |
| CreateTopic | Creates a new topic on the Kafka broker. This stored procedure allows you to define key topic parameters (for example, the number of partitions and replication factor) and initialize new message streams for publishing and consumption. |
| DeleteTopic | Deletes an existing topic from the Kafka broker. This stored procedure permanently removes the topic and all of its messages, so it should be used with caution in production environments. |
| GetAdminConsentURL.rsb | Specifies any additional parameters that are required by the authorization request. These parameters are passed directly to the OAuth authorization endpoint. |
| ProduceMessage | Publishes a raw message to a specified Kafka topic. This stored procedure enables data producers to send serialized message payloads (for example, JSON or Avro) directly into the message stream for downstream consumers to process. |
Commits the current set of message offsets for a specified consumer group. This stored procedure ensures that consumed messages are marked as processed within Kafka, preventing the same records from being read again when the consumer resumes operation.
| Name | Type | Description |
| Success | String | Specifies whether the offset commit operation is successful. This output returns a value of 'true' when the stored procedure successfully commits the consumer group offsets to Kafka, and a value of 'false' when the commit fails due to connection issues or broker errors. |
Creates a schema definition that maps a Kafka topic to a relational table structure. This stored procedure is typically used to establish a structured representation of message data so that the topic can be queried or transformed using SQL-like operations.
Creates a local schema file (.rsd) from an existing table or view in the data model.
The schema file is created in the directory set in the Location connection property when this procedure is executed. You can edit the file to include or exclude columns, rename columns, or adjust column datatypes.
The Sync App checks the Location to determine if the names of any .rsd files match a table or view in the data model. If there is a duplicate, the schema file will take precedence over the default instance of this table in the data model. If a schema file is present in Location that does not match an existing table or view, a new table or view entry is added to the data model of the Sync App.
| Name | Type | Required | Description |
| TableName | String | True | Specifies the name of the table for which the schema definition is created. This input determines the table structure that the stored procedure uses to generate the corresponding schema file. |
| FileName | String | False | Specifies the full file path and name of the schema file to generate (for example, 'C:\\Users\\User\\Desktop\\ApacheKafka\\company.rsd'). This input is required when you want the generated schema to be saved directly to a file. |
| Name | Type | Description |
| Result | String | Returns a value of 'true' when the stored procedure successfully generates and writes the schema and a value of 'false' when the operation fails due to missing parameters or write-access issues. |
| FileData | String | Returns the generated schema definition in Base64-encoded format. This output is populated only when the FileName and FileStream inputs are not provided, allowing the schema to be retrieved directly from the stored procedure response. |
Creates a new topic on the Kafka broker. This stored procedure allows you to define key topic parameters (for example, the number of partitions and replication factor) and initialize new message streams for publishing and consumption.
| Name | Type | Required | Description |
| Topic | String | True | Specifies the name of the Kafka topic to create. This input defines the identifier for the new message stream that producers can publish to and consumers can subscribe to after creation. |
| Partitions | String | True | Specifies the number of partitions to assign to the topic. This value determines how message data is distributed across brokers and affects parallel processing and scalability. |
| ReplicationFactor | String | True | Specifies the number of replicas that are maintained for each partition. The value cannot exceed the number of brokers in the Kafka cluster, and it determines the level of fault tolerance and data redundancy. |
| Name | Type | Description |
| Success | String | Returns a value of 'true' when the stored procedure successfully creates the Kafka topic and a value of 'false' when the operation fails because of configuration or broker-level errors. |
Deletes an existing topic from the Kafka broker. This stored procedure permanently removes the topic and all of its messages, so it should be used with caution in production environments.
| Name | Type | Required | Description |
| Topic | String | True | Specifies the name of the Kafka topic to delete. This input identifies the message stream that is permanently removed from the Kafka broker, including all associated partitions and stored data. |
| Name | Type | Description |
| Success | String | Returns a value of 'true' when the stored procedure successfully deletes the specified Kafka topic and a value of 'false' when the deletion fails because of permission restrictions or broker-level errors. |
Specifies any additional parameters that are required by the authorization request. These parameters are passed directly to the OAuth authorization endpoint.
| Name | Type | Required | Description |
| CallbackUrl | String | False | Specifies the URL to which the user is redirected after authorizing the application. This value must match the Reply URL that is defined in the Azure Active Directory application settings. |
| State | String | False | Specifies the same state value that is sent when the application requests the authorization code. This input is used to maintain request integrity and to prevent cross-site request forgery (CSRF) attacks during the OAuth process. |
| Name | Type | Description |
| URL | String | Returns the administrator consent URL that must be entered in a web browser to obtain the verifier token and authorize the application for domain-level access. |
Gets an OAuth authentication token.
| Name | Type | Required | Description |
| AuthMode | String | False | The type of authentication mode to use. Select APP for getting authentication tokens via a desktop app. Select WEB for getting authentication tokens via a Web app.
The allowed values are APP, WEB. The default value is WEB. |
| Verifier | String | False | The verifier returned after the user has authorized your app to have access to their data. This value will be returned as a parameter to the callback URL. |
| CallbackURL | String | False | Determines where the response is sent. |
| State | String | False | Indicates any state which may be useful to your application upon receipt of the response. Your application receives the same value it sent, as this parameter makes a round-trip to the authorization server and back. Uses include redirecting the user to the correct resource in your site, nonces, and cross-site-request-forgery mitigations. |
| Name | Type | Description |
| OAuthAccessToken | String | The access token used for communication with the API. |
| OAuthRefreshToken | String | The refresh access token used to refresh your connection. |
| ExpiresIn | String | The lifetime of the access token, in seconds. |
Gets the authorization URL that must be opened separately by the user to grant access to your application. You will request the OAuthAccessToken from this URL.
| Name | Type | Required | Description |
| CallbackURL | String | False | Determines where the response is sent. The value of this parameter must exactly match one of the values registered in the APIs Console (including the HTTP or HTTPS schemes, capitalization, and trailing '/'). |
| State | String | False | Indicates any state which may be useful to your application upon receipt of the response. Your application receives the same value it sent, as this parameter makes a round-trip to the Google authorization server and back. Uses include redirecting the user to the correct resource in your site, nonces, and cross-site-request-forgery mitigations. |
| Name | Type | Description |
| URL | String | The authorization URL, entered into a Web browser to obtain the verifier token and authorize your app. |
Publishes a raw message to a specified Kafka topic. This stored procedure enables data producers to send serialized message payloads (for example, JSON or Avro) directly into the message stream for downstream consumers to process.
When producing a message, only the Topic input is required:
EXEC ProduceMessage Topic = 'mytopic'
The other properties have the following default behaviors:
Omitting the key or message content may make the message unreadable with certain Sync App settings. For example, setting MessageKeyType to integer may cause read failures with messages that have no keys. TypeDetectionScheme MessageOnly and MessageKeyType Binary are recommended if you need to read messages containing arbitrary content or keys.
The key and message can each be provided either as text or base64-encoded strings. KeyBytes and MessageBytes accept base64-encoded strings. The Sync App decodes these values into bytes before sending the message to Kafka.
KeyText and MessageText accept any text. The Sync App encodes these values as UTF-8 before sending the message to Kafka.
For example, the following statement inserts a message with no key and content that is a single NUL byte:
EXEC ProduceMessage Topic = 'mytopic', MessageBytes = 'AA=='
| Name | Type | Required | Description |
| Topic | String | True | Specifies the Kafka topic that contains the message. This input identifies the message stream to which the data record is published. |
| Partition | String | False | Specifies the partition that the message is assigned to. The partition value must be valid for the specified topic. The native Kafka client automatically assigns a partition when this input is not set. |
| KeyText | String | False | Specifies the message key that is provided as readable text. The KeyText input defines the human-readable key portion of the Kafka message that determines how messages are partitioned. The value is encoded as UTF-8 before it is sent to Kafka. Do not set KeyText when KeyBytes input is provided. |
| KeyBytes | String | False | Specifies the message key that is provided as a Base64-encoded binary string. The KeyBytes input defines the binary representation of the same logical key as the KeyText parameter and is used when the key must preserve exact byte content (for example, hashed or serialized identifiers). Do not set KeyBytes when KeyText input is provided. |
| MessageText | String | False | Specifies the message value that is provided as readable text. The MessageText input defines the human-readable content of the Kafka message that is sent to the specified topic. The value is encoded as UTF-8 before it is sent to Kafka. Do not set MessageText when MessageBytes input is provided. The MessageBytes input represents the same message value in Base64-encoded binary format. |
| MessageBytes | String | False | Specifies the message value that is provided as a Base64-encoded binary string. The MessageBytes input defines the binary representation of the same message value as the MessageText input parameter and is used when the message content must preserve exact byte content (for example, serialized, compressed, or non-text data). Do not set MessageBytes when the MessageText input is provided. |
| Message | String | False | Indicates a deprecated input that functions the same as MessageBytes. This input is used only when MessageBytes is not provided. Both MessageBytes and Message input parameters must be omitted to send a message with no content. |
| Name | Type | Description |
| PartitionWritten | Int | Returns the partition that the message is written to. The value is identical to the Partition input when that parameter is specified. |
| OffsetWritten | Long | Returns the offset position within the partition where the message is written. This value enables message retrieval or replay from a specific point in the stream. |
| TimestampWritten | Long | Returns the UNIX timestamp that represents the exact instant when the message is committed to the partition. |
| KeyWritten | String | Returns the Base64-encoded data of the key that is written. This output is null when neither KeyText nor KeyBytes is provided. |
| MessageWritten | String | Returns the Base64-encoded data of the message value that is written. This output is null when neither MessageText nor MessageBytes is provided. |
| Success | Bool | Returns a value of 'true' when the stored procedure successfully writes the message to Kafka and a value of 'false' when the operation fails due to invalid parameters or broker-level errors. |
Exchanges a access token for a new access token.
| Name | Type | Required | Description |
| OAuthRefreshToken | String | True | The refresh token returned from the original authorization code exchange. |
| Name | Type | Description |
| OAuthAccessToken | String | The access token used for communication with the API. |
| OAuthRefreshToken | String | The refresh access token used to refresh your connection. |
| ExpiresIn | String | The lifetime of the access token, in seconds. |
The connection string properties are the various options that can be used to establish a connection. This section provides a complete list of the options you can configure in the connection string for this provider. Click the links for further details.
For more information on establishing a connection, see Establishing a Connection.
| Property | Description |
| AuthScheme | Specifies the authentication scheme that the provider uses when connecting to the Apache Kafka broker. |
| User | Specifies the username used to authenticate to the Apache Kafka broker. |
| Password | Specifies the password used to authenticate to Apache Kafka for the selected authentication scheme. |
| BootstrapServers | Specifies the Kafka bootstrap servers that the provider uses to establish the initial connection to the Kafka cluster. |
| UseSSL | Specifies whether the provider negotiates SSL/TLS when connecting to the Apache Kafka broker. |
| Property | Description |
| ConsumerGroupId | Specifies the consumer group that the provider uses when reading messages from Apache Kafka. |
| AutoCommit | Specifies whether the Apache Kafka consumer automatically commits read offsets. |
| Property | Description |
| AzureTenant | Identifies the Apache Kafka tenant being used to access data. Accepts either the tenant's domain name (for example, contoso.onmicrosoft.com ) or its directory (tenant) ID. |
| AzureResource | The Azure Active resource to authenticate to (used during Azure OAuth exchange). |
| Property | Description |
| OAuthClientId | Specifies the client ID (also known as the consumer key) assigned to your custom OAuth application. This ID is required to identify the application to the OAuth authorization server during authentication. |
| OAuthClientSecret | Specifies the client secret assigned to your custom OAuth application. This confidential value is used to authenticate the application to the OAuth authorization server. (Custom OAuth applications only.). |
| DelegatedServiceAccounts | Specifies a space-delimited list of service account emails for delegated requests. |
| RequestingServiceAccount | Specifies a service account email to make a delegated request. |
| Scope | Specifies the scopes to request when obtaining an OAuth token from the OIDC token endpoint. |
| OAuthAccessTokenURL | The URL from which the OAuth access token is retrieved. |
| Property | Description |
| OAuthJWTCert | Supplies the name of the client certificate's JWT Certificate store. |
| OAuthJWTCertType | Identifies the type of key store containing the JWT Certificate. |
| OAuthJWTCertPassword | Provides the password for the OAuth JWT certificate used to access a password-protected certificate store. If the certificate store does not require a password, leave this property blank. |
| OAuthJWTCertSubject | Identifies the subject of the OAuth JWT certificate used to locate a matching certificate in the store. Supports partial matches and the wildcard '*' to select the first certificate. |
| Property | Description |
| KerberosKeytabFile | Specifies the path to the keytab file that contains the Kerberos principals and encrypted keys used for authentication. |
| KerberosSPN | Specifies the full Kerberos service principal name (SPN) of the Kafka broker. |
| KerberosServiceName | Specifies the Kerberos service name used when authenticating to the Kafka broker. |
| UseKerberosTicketCache | Specifies whether the provider uses the Kerberos ticket cache for authentication instead of a keytab file. |
| Property | Description |
| SSLServerCert | Specifies the SSL server certificate or certificate store used to verify the identity of the Apache Kafka broker. |
| SSLServerCertType | Specifies the format of the SSL server certificate used to verify the Apache Kafka broker. |
| SSLClientCert | Specifies the SSL client certificate used to authenticate with the Apache Kafka broker. |
| SSLClientCertType | Specifies the format of the SSL client certificate used to connect to the Apache Kafka broker. |
| SSLClientCertPassword | Specifies the password used to decrypt the certificate provided in SSLClientCert . |
| SSLIdentificationAlgorithm | Specifies the endpoint identification algorithm used to validate the server host name during SSL/TLS connections. |
| Property | Description |
| RegistryURL | Specifies the endpoint of the schema registry. When this property is specified, the driver supports reading Avro and JSON schemas from the server. |
| RegistryService | Specifies the schema registry service that the provider uses to retrieve key and value schemas for Apache Kafka topics. |
| RegistryAuthScheme | Specifies the authentication scheme that the provider uses when connecting to the schema registry. |
| RegistryUser | Specifies the user name used when authenticating to the schema registry with the Basic authentication scheme. |
| RegistryPassword | Specifies the password used when authenticating to the schema registry with the Basic authentication scheme. |
| RegistryClientCert | The TLS/SSL client certificate store for SSL Client Authentication (2-way SSL) with the schema registry. |
| RegistryClientCertType | Specifies the type of key store used by the TLS/SSL client certificate given in RegistryClientCert . |
| RegistryClientCertPassword | Specifies the password for the client certificate store defined in RegistryClientCert. |
| RegistryClientCertSubject | Specifies the subject of the client certificate to select from the certificate store defined in RegistryClientCert. |
| RegistryVersion | Specifies which version of a schema the provider retrieves from the schema registry when resolving topic columns. |
| RegistryServerCert | The certificate to be accepted from the schema registry when connecting using TLS/SSL. |
| SchemaMergeMode | Specifies how the provider exposes schemas with multiple versions. |
| Property | Description |
| FirewallType | Specifies the protocol the provider uses to tunnel traffic through a proxy-based firewall. |
| FirewallServer | Identifies the IP address, DNS name, or host name of a proxy used to traverse a firewall and relay user queries to network resources. |
| FirewallPort | Specifies the TCP port to be used for a proxy-based firewall. |
| FirewallUser | Identifies the user ID of the account authenticating to a proxy-based firewall. |
| FirewallPassword | Specifies the password of the user account authenticating to a proxy-based firewall. |
| Property | Description |
| ProxyAutoDetect | Specifies whether the provider checks your system proxy settings for existing proxy server configurations, rather than using a manually specified proxy server. |
| ProxyServer | Identifies the hostname or IP address of the proxy server through which you want to route HTTP traffic. |
| ProxyPort | Identifies the TCP port on your specified proxy server that has been reserved for routing HTTP traffic to and from the client. |
| ProxyAuthScheme | Specifies the authentication method the provider uses when authenticating to the proxy server specified in the ProxyServer connection property. |
| ProxyUser | Provides the username of a user account registered with the proxy server specified in the ProxyServer connection property. |
| ProxyPassword | Specifies the password of the user specified in the ProxyUser connection property. |
| ProxySSLType | Specifies the SSL type to use when connecting to the proxy server specified in the ProxyServer connection property. |
| ProxyExceptions | Specifies a semicolon-separated list of destination hostnames or IPs that are exempt from connecting through the proxy server set in the ProxyServer connection property. |
| Property | Description |
| LogModules | Specifies the core modules to include in the log file. Use a semicolon-separated list of module names. By default, all modules are logged. |
| Property | Description |
| Location | Specifies the location of a directory containing schema files that define tables, views, and stored procedures. Depending on your service's requirements, this may be expressed as either an absolute path or a relative path. |
| BrowsableSchemas | Optional setting that restricts the schemas reported to a subset of all available schemas. For example, BrowsableSchemas=SchemaA,SchemaB,SchemaC . |
| Tables | Optional setting that restricts the tables reported to a subset of all available tables. For example, Tables=TableA,TableB,TableC . |
| Views | Optional setting that restricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC . |
| Property | Description |
| AllowKeyOnlyRegistryTopics | Specifies whether the provider exposes key-only Schema Registry topics as tables. |
| AWSWorkloadIdentityConfig | Configuration properties to provide when using Workload Identity Federation via AWS. |
| AzureWorkloadIdentityConfig | Configuration properties to provide when using Workload Identity Federation via Azure. |
| CompressionType | Specifies the compression algorithm that the provider uses when producing messages to Apache Kafka. |
| ConsumerProperties | Specifies additional Kafka consumer configuration options that the provider passes directly to the underlying Kafka client. |
| CreateTablePartitions | Specifies the number of partitions to assign to a topic created through a CREATE TABLE statement. |
| CreateTableReplicationFactor | Specifies the the number of replicas to assign to a topic created through a CREATE TABLE statement. |
| EnableIdempotence | Specifies whether the provider ensures that produced messages are delivered in order and without duplicates. |
| ExposeQueueMetadataColumns | Specifies whether the Partition, Offset, and Timestamp columns are exposed. |
| FlattenArrays | Specifies how many elements to return from nested arrays when TypeDetectionScheme is set to SchemaRegistry. |
| GenerateSchemaFiles | Indicates the user preference as to when schemas should be generated and saved. |
| HideUnusedColumns | Determines whether to hide key or value colums when the topic has no associated schema information. |
| MaximumBatchSize | Specifies the maximum size, in bytes, of a batch of messages that the provider gathers before sending the batch to Apache Kafka. |
| MaxRows | Specifies the maximum number of rows returned for queries that do not include either aggregation or GROUP BY. |
| MessageKeyColumn | Specifies the name of the column where the provider stores the message key for each record. |
| MessageKeyType | The type of data stored in message keys. |
| NonRegistryTypeDetectionScheme | Specifies the TypeDetectionScheme to use for topics that do not have schemas in the schema registry. |
| OffsetResetStrategy | Specifies how the provider determines the starting offset when no committed offset exists for the consumer group. |
| Other | Specifies advanced connection properties for specialized scenarios. Use this property only under the guidance of our Support team to address specific issues. |
| Pagesize | Specifies the maximum number of rows that the provider retrieves from Apache Kafka in a single read operation. |
| ProducerProperties | Specifies additional Apache Kafka producer configuration options that the provider passes directly to the client. |
| PseudoColumns | Specifies the pseudocolumns to expose as table columns, expressed as a string in the format 'TableName=ColumnName;TableName=ColumnName'. |
| ReadDuration | Specifies how long, in seconds, the provider waits for additional messages after a read operation begins. |
| RowScanDepth | Specifies the maximum number of messages that the provider scans to determine the columns and data types for a topic. |
| SASLOAuthExtensions | Specifies the extension values to send with OAuth auth schemes. |
| SchemaRegistryOnly | Specifies whether the provider connects only to the schema registry. |
| SerializationFormat | Specifies how to serialize/deserialize message contents. |
| Timeout | Specifies the maximum time, in seconds, that the provider waits for a server response before throwing a timeout error. |
| TypeDetectionScheme | Specifies how the provider determines the available fields and data types for each topic. |
| UseConfluentAvroFormat | Specifies how Avro data should be formatted during an INSERT. |
| UserDefinedViews | Specifies a filepath to a JSON configuration file that defines custom views. The provider automatically detects and uses the views specified in this file. |
| ValidateRegistryTopics | Specifies whether or not to validate schema registry topics against the Apache Kafka broker. Only has an effect when TypeDetectionScheme =SchemaRegistry. |
| WorkloadPoolId | The ID of your Workload Identity Federation pool. |
| WorkloadProjectId | The ID of the Google Cloud project that hosts your Workload Identity Federation pool. |
| WorkloadProviderId | The ID of your Workload Identity Federation pool provider. |
This section provides a complete list of the Authentication properties you can configure in the connection string for this provider.
| Property | Description |
| AuthScheme | Specifies the authentication scheme that the provider uses when connecting to the Apache Kafka broker. |
| User | Specifies the username used to authenticate to the Apache Kafka broker. |
| Password | Specifies the password used to authenticate to Apache Kafka for the selected authentication scheme. |
| BootstrapServers | Specifies the Kafka bootstrap servers that the provider uses to establish the initial connection to the Kafka cluster. |
| UseSSL | Specifies whether the provider negotiates SSL/TLS when connecting to the Apache Kafka broker. |
Specifies the authentication scheme that the provider uses when connecting to the Apache Kafka broker.
Authentication requirements vary by Kafka deployment. The Sync App supports several authentication schemes used by self-managed Kafka clusters, cloud-managed Kafka services, and Schema Registry environments. The schemes listed below determine how the Sync App obtains credentials and establishes a secure connection to the broker.
Supported schemes for Apache Kafka:
| Scheme | Description |
| None | Connects without authentication. No credentials are required. |
| Plain | Authenticates using a plain text login module. Requires User and Password. |
| SCRAM | Authenticates using a SCRAM login module with SHA-256 hashing. |
| SCRAM-SHA-512 | Authenticates using a SCRAM login module with SHA-512 hashing. |
| Kerberos | Use Kerberos authentication. Requires a Kerberos configuration file and relevant Kerberos properties. |
| SSLCertificate | Authenticates using an SSL client certificate. |
| KafkaOAuthClient | Authenticates using Kafka's native OpenID Connect (KIP-768) authentication with the client credentials grant type. Note: This does not use the Sync App's built-in support for OAuth like the Azure and OAuthJWT authentication methods do. Only the OAuthClientId, OAuthClientSecret, OAuthAccessTokenURL, and Scope properties have any effect on the behavior of this scheme. Use ConsumerProperties and ProducerProperties to provide advanced configuration. |
| AzureAD | Authenticates using Azure Active Directory OAuth. |
| AzureMSI | Obtains Managed Service Identity credentials automatically when running on an Azure VM. |
| AzureServicePrincipal | Authenticates as an Azure Service Principal using a client secret. |
| AzureServicePrincipalCert | Authenticates as an Azure Service Principal using a certificate. |
| OAuthJWT | Authenticates using an OAuth service account with JWT-based credential flow. |
| GCPInstanceAccount | Authenticates using an access token obtained from a Google Cloud instance. |
| AWSWorkloadIdentity | Authenticates using AWS Workload Identity Federation. |
Schemes for authenticating to Azure Event Hubs:
| AzureAD | Authenticates using Azure Active Directory OAuth. |
| AzureMSI | Obtains Managed Service Identity credentials automatically when running on an Azure VM. |
| AzureServicePrincipal | Authenticates as an Azure Service Principal using a client secret. |
| AzureServicePrincipalCert | Authenticates as an Azure Service Principal using a certificate. |
Schemes for authenticating to GMS Kafka:
| OAuthJWT | Authenticates using an OAuth service account. |
| GCPInstanceAccount | Authenticates using a Google Cloud instance account. |
| AWSWorkloadIdentity | Authenticates using AWS Workload Identity Federation. GMS Kafka does not allow external principals to authenticate directly, so you must delegate authentication to a service account using the RequestingServiceAccount property. |
Specifies the username used to authenticate to the Apache Kafka broker.
When using an authentication scheme that requires credentials, this property provides the username associated with the connection. If this property is left blank, the Sync App attempts an unauthenticated connection, which will fail unless the broker allows anonymous access.
Specifies the password used to authenticate to Apache Kafka for the selected authentication scheme.
This property is required when using the Plain, SCRAM, or SCRAM-SHA-512 authentication schemes. If this property is not set when one of these schemes is selected, the connection will fail during authentication.
Specifies the Kafka bootstrap servers that the provider uses to establish the initial connection to the Kafka cluster.
Specify each server using a hostname or IP address followed by a port number. For example: 10.1.2.3:9092. You may provide multiple comma-separated addresses. The connection succeeds as long as the Sync App can reach at least one of the listed servers.
Kafka bootstrap servers are not responsible for all messaging traffic. They are used only to retrieve metadata, after which the Sync App connects to the appropriate brokers in the cluster.
If you are connecting to Confluent Cloud, the bootstrap server address is available in the Cluster settings.
Specifies whether the provider negotiates SSL/TLS when connecting to the Apache Kafka broker.
This option is enabled automatically when AuthScheme is set to SSL.
This section provides a complete list of the Connection properties you can configure in the connection string for this provider.
| Property | Description |
| ConsumerGroupId | Specifies the consumer group that the provider uses when reading messages from Apache Kafka. |
| AutoCommit | Specifies whether the Apache Kafka consumer automatically commits read offsets. |
Specifies the consumer group that the provider uses when reading messages from Apache Kafka.
A consumer group is a logical identifier that Kafka uses to track committed offsets and coordinate message consumption across multiple consumers. All consumers that share the same group ID use the same committed offsets and divide message processing among themselves.
If this property is not specified, the Sync App generates a unique random group ID for each connection. In this mode, offsets are not shared across connections, and each connection reads messages independently of others. Set this property when you want multiple connections or applications to share the same committed offsets, or when you are using features that depend on offset tracking, such as AutoCommit.
Specifies whether the Apache Kafka consumer automatically commits read offsets.
By default, the Sync App does not commit read offsets unless you invoke CommitOffset. When an offset is committed for a topic, the Sync App starts reading from that offset in future queries until the next call to CommitOffset. If no offsets are committed, each query begins reading from the position determined by OffsetResetStrategy.
When set to true, the Sync App commits offsets periodically and also at the end of each SELECT query. This causes each SELECT query to consume the messages it reads, preventing future queries in the same consumer group from returning those messages. A ConsumerGroupId is required because the committed offset is shared across the consumer group, allowing later connections with the same group ID to use the same offsets.
Consider the following when enabling AutoCommit:
This section provides a complete list of the Azure Authentication properties you can configure in the connection string for this provider.
| Property | Description |
| AzureTenant | Identifies the Apache Kafka tenant being used to access data. Accepts either the tenant's domain name (for example, contoso.onmicrosoft.com ) or its directory (tenant) ID. |
| AzureResource | The Azure Active resource to authenticate to (used during Azure OAuth exchange). |
Identifies the Apache Kafka tenant being used to access data. Accepts either the tenant's domain name (for example, contoso.onmicrosoft.com ) or its directory (tenant) ID.
A tenant is a digital container for your organization's users and resources, managed through Microsoft Entra ID (formerly Azure AD). Each tenant is associated with a unique directory ID, and often with a custom domain (for example, microsoft.com or contoso.onmicrosoft.com).
To find the directory (tenant) ID in the Microsoft Entra Admin Center, navigate to Microsoft Entra ID > Properties and copy the value labeled "Directory (tenant) ID".
This property is required in the following cases:
You can provide the tenant value in one of two formats:
Specifying the tenant explicitly ensures that the authentication request is routed to the correct directory, which is especially important when a user belongs to multiple tenants or when using service principal–based authentication.
If this value is omitted when required, authentication may fail or connect to the wrong tenant. This can result in errors such as unauthorized or resource not found.
A tenant is a digital container for your organization's users and resources, managed through Microsoft Entra ID (formerly Azure AD). Each tenant is associated with a unique directory ID, and often with a custom domain (for example, microsoft.com or contoso.onmicrosoft.com).
To find the directory (tenant) ID in the Microsoft Entra Admin Center, navigate to Microsoft Entra ID > Properties and copy the value labeled "Directory (tenant) ID".
This property is required in the following cases:
You can provide the tenant value in one of two formats:
Specifying the tenant explicitly ensures that the authentication request is routed to the correct directory, which is especially important when a user belongs to multiple tenants or when using service principal–based authentication.
If this value is omitted when required, authentication may fail or connect to the wrong tenant. This can result in errors such as unauthorized or resource not found.
The Azure Active resource to authenticate to (used during Azure OAuth exchange).
The resource must be specified if using Azure OAuth. It should be set to the App Id URI of the web API (secured resource).
This section provides a complete list of the OAuth properties you can configure in the connection string for this provider.
| Property | Description |
| OAuthClientId | Specifies the client ID (also known as the consumer key) assigned to your custom OAuth application. This ID is required to identify the application to the OAuth authorization server during authentication. |
| OAuthClientSecret | Specifies the client secret assigned to your custom OAuth application. This confidential value is used to authenticate the application to the OAuth authorization server. (Custom OAuth applications only.). |
| DelegatedServiceAccounts | Specifies a space-delimited list of service account emails for delegated requests. |
| RequestingServiceAccount | Specifies a service account email to make a delegated request. |
| Scope | Specifies the scopes to request when obtaining an OAuth token from the OIDC token endpoint. |
| OAuthAccessTokenURL | The URL from which the OAuth access token is retrieved. |
Specifies the client ID (also known as the consumer key) assigned to your custom OAuth application. This ID is required to identify the application to the OAuth authorization server during authentication.
This property is required in two cases:
(When the driver provides embedded OAuth credentials, this value may already be provided by the Sync App and thus not require manual entry.)
OAuthClientId is generally used alongside other OAuth-related properties such as OAuthClientSecret and OAuthSettingsLocation when configuring an authenticated connection.
OAuthClientId is one of the key connection parameters that need to be set before users can authenticate via OAuth. You can usually find this value in your identity provider’s application registration settings. Look for a field labeled Client ID, Application ID, or Consumer Key.
While the client ID is not considered a confidential value like a client secret, it is still part of your application's identity and should be handled carefully. Avoid exposing it in public repositories or shared configuration files.
For more information on how this property is used when configuring a connection, see Establishing a Connection.
Specifies the client secret assigned to your custom OAuth application. This confidential value is used to authenticate the application to the OAuth authorization server. (Custom OAuth applications only.).
This property (sometimes called the application secret or consumer secret) is required when using a custom OAuth application in any flow that requires secure client authentication, such as web-based OAuth, service-based connections, or certificate-based authorization flows. It is not required when using an embedded OAuth application.
The client secret is used during the token exchange step of the OAuth flow, when the driver requests an access token from the authorization server. If this value is missing or incorrect, authentication fails with either an invalid_client or an unauthorized_client error.
OAuthClientSecret is one of the key connection parameters that need to be set before users can authenticate via OAuth. You can obtain this value from your identity provider when registering the OAuth application.
Notes:
For more information on how this property is used when configuring a connection, see Establishing a Connection
Specifies a space-delimited list of service account emails for delegated requests.
The service account emails must be specified in a space-delimited list.
Each service account must be granted the roles/iam.serviceAccountTokenCreator role on its next service account in the chain.
The last service account in the chain must be granted the roles/iam.serviceAccountTokenCreator role on the requesting service account. The requesting service account is the one specified in the RequestingServiceAccount property.
Note that for delegated requests, the requesting service account must have the permission iam.serviceAccounts.getAccessToken, which can also be granted through the serviceAccountTokenCreator role.
Specifies a service account email to make a delegated request.
The service account email of the account for which the credentials are requested in a delegated request. With the list of delegated service accounts in DelegatedServiceAccounts, this property is used to make a delegated request.
You must have the IAM permission iam.serviceAccounts.getAccessToken on this service account.
Specifies the scopes to request when obtaining an OAuth token from the OIDC token endpoint.
Scopes are set to define what kind of access the authenticating user will have; for example, read, read and write, restricted access to sensitive information. System administrators can use scopes to selectively enable access by functionality or security clearance.
When InitiateOAuth is set to GETANDREFRESH, you must use this property if you want to change which scopes are requested.
When InitiateOAuth is set to either REFRESH or OFF, you can change which scopes are requested using either this property or the Scope input.
When set to a value, this property is included in the OAuth authorization request sent to your identity provider. Scopes determine the level of access granted in the issued token.
The default server-side Apache Kafka OAuth validator does not require scopes, but your identity provider may enforce them. Set this property to whatever scope string your provider requires for client credential OIDC authentication.
The URL from which the OAuth access token is retrieved.
In OAuth 1.0, the authorized request token is exchanged for the access token at this URL.
This section provides a complete list of the JWT OAuth properties you can configure in the connection string for this provider.
| Property | Description |
| OAuthJWTCert | Supplies the name of the client certificate's JWT Certificate store. |
| OAuthJWTCertType | Identifies the type of key store containing the JWT Certificate. |
| OAuthJWTCertPassword | Provides the password for the OAuth JWT certificate used to access a password-protected certificate store. If the certificate store does not require a password, leave this property blank. |
| OAuthJWTCertSubject | Identifies the subject of the OAuth JWT certificate used to locate a matching certificate in the store. Supports partial matches and the wildcard '*' to select the first certificate. |
Supplies the name of the client certificate's JWT Certificate store.
The OAuthJWTCertType field specifies the type of the certificate store specified in OAuthJWTCert. If the store is password-protected, use OAuthJWTCertPassword to supply the password..
OAuthJWTCert is used in conjunction with the OAuthJWTCertSubject field in order to specify client certificates. If OAuthJWTCert has a value, and OAuthJWTCertSubject is set, the CData Sync App initiates a search for a certificate. For further information, see OAuthJWTCertSubject.
Designations of certificate stores are platform-dependent.
Notes
Identifies the type of key store containing the JWT Certificate.
| Value | Description | Notes |
| USER | A certificate store owned by the current user. | Only available in Windows. |
| MACHINE | A machine store. | Not available in Java or other non-Windows environments. |
| PFXFILE | A PFX (PKCS12) file containing certificates. | |
| PFXBLOB | A string (base-64-encoded) representing a certificate store in PFX (PKCS12) format. | |
| JKSFILE | A Java key store (JKS) file containing certificates. | Only available in Java. |
| JKSBLOB | A string (base-64-encoded) representing a certificate store in Java key store (JKS) format. | Only available in Java. |
| PEMKEY_FILE | A PEM-encoded file that contains a private key and an optional certificate. | |
| PEMKEY_BLOB | A string (base64-encoded) that contains a private key and an optional certificate. | |
| PUBLIC_KEY_FILE | A file that contains a PEM- or DER-encoded public key certificate. | |
| PUBLIC_KEY_BLOB | A string (base-64-encoded) that contains a PEM- or DER-encoded public key certificate. | |
| SSHPUBLIC_KEY_FILE | A file that contains an SSH-style public key. | |
| SSHPUBLIC_KEY_BLOB | A string (base-64-encoded) that contains an SSH-style public key. | |
| P7BFILE | A PKCS7 file containing certificates. | |
| PPKFILE | A file that contains a PPK (PuTTY Private Key). | |
| XMLFILE | A file that contains a certificate in XML format. | |
| XMLBLOB | Astring that contains a certificate in XML format. | |
| BCFKSFILE | A file that contains an Bouncy Castle keystore. | |
| BCFKSBLOB | A string (base-64-encoded) that contains a Bouncy Castle keystore. | |
| GOOGLEJSON | A JSON file containing the service account information. | Only valid when connecting to a Google service. |
| GOOGLEJSONBLOB | A string that contains the service account JSON. | Only valid when connecting to a Google service. |
Provides the password for the OAuth JWT certificate used to access a password-protected certificate store. If the certificate store does not require a password, leave this property blank.
This property specifies the password needed to open a password-protected certificate store. To determine if a password is necessary, refer to the documentation or configuration for your specific certificate store.
This is not required when using the GOOGLEJSON OAuthJWTCertType. Google JSON keys are not encrypted.
Identifies the subject of the OAuth JWT certificate used to locate a matching certificate in the store. Supports partial matches and the wildcard '*' to select the first certificate.
The value of this property is used to locate a matching certificate in the store. The search process works as follows:
You can set the value to '*' to automatically select the first certificate in the store. The certificate subject is a comma-separated list of distinguished name fields and values. For example: CN=www.server.com, OU=test, C=US, [email protected].
Common fields include:
| Field | Meaning |
| CN | Common Name. This is commonly a host name like www.server.com. |
| O | Organization |
| OU | Organizational Unit |
| L | Locality |
| S | State |
| C | Country |
| E | Email Address |
If a field value contains a comma, enclose it in quotes. For example: "O=ACME, Inc.".
This section provides a complete list of the Kerberos properties you can configure in the connection string for this provider.
| Property | Description |
| KerberosKeytabFile | Specifies the path to the keytab file that contains the Kerberos principals and encrypted keys used for authentication. |
| KerberosSPN | Specifies the full Kerberos service principal name (SPN) of the Kafka broker. |
| KerberosServiceName | Specifies the Kerberos service name used when authenticating to the Kafka broker. |
| UseKerberosTicketCache | Specifies whether the provider uses the Kerberos ticket cache for authentication instead of a keytab file. |
Specifies the path to the keytab file that contains the Kerberos principals and encrypted keys used for authentication.
Set this property to the location of a valid keytab file when using Kerberos authentication. The keytab must contain the principal entries required for the Kerberos login configured for the Sync App.
Specifies the full Kerberos service principal name (SPN) of the Kafka broker.
Set this property to the service principal name that identifies the Kafka broker in Kerberos. The SPN typically includes the service name and hostname, such as kafka/host.example.com@REALM. The Sync App uses this value when requesting a service ticket during Kerberos authentication.
Specifies the Kerberos service name used when authenticating to the Kafka broker.
Set this property to the service name portion of the Kerberos principal used by the Kafka broker.
Specifies whether the provider uses the Kerberos ticket cache for authentication instead of a keytab file.
When set to true, the Sync App uses the Kerberos tickets already obtained by the logged-in user. In this mode, KerberosKeytabFile is not required.
When set to false, the Sync App uses the keytab file specified in KerberosKeytabFile to obtain Kerberos credentials.
This option is useful when users authenticate interactively and already have valid Kerberos tickets on the system.
This section provides a complete list of the SSL properties you can configure in the connection string for this provider.
| Property | Description |
| SSLServerCert | Specifies the SSL server certificate or certificate store used to verify the identity of the Apache Kafka broker. |
| SSLServerCertType | Specifies the format of the SSL server certificate used to verify the Apache Kafka broker. |
| SSLClientCert | Specifies the SSL client certificate used to authenticate with the Apache Kafka broker. |
| SSLClientCertType | Specifies the format of the SSL client certificate used to connect to the Apache Kafka broker. |
| SSLClientCertPassword | Specifies the password used to decrypt the certificate provided in SSLClientCert . |
| SSLIdentificationAlgorithm | Specifies the endpoint identification algorithm used to validate the server host name during SSL/TLS connections. |
Specifies the SSL server certificate or certificate store used to verify the identity of the Apache Kafka broker.
The value for this property must match the format described by SSLServerCertType. This determines whether the certificate is supplied as a file path, a base64-encoded blob, or a reference to a system certificate store.
The server certificate is used to confirm that the broker’s SSL certificate is signed by a trusted Certificate Authority (CA). If the provided certificate does not match the server’s certificate chain, the SSL connection will be rejected.
Specifies the format of the SSL server certificate used to verify the Apache Kafka broker.
This property is used to determine what format the SSLServerCert property expects. This property can take one of the following values:
| Store Type | Description |
| PEMKEY_FILE | The certificate store is the name of a PEM-encoded file that contains a the server certificate. |
| PEMKEY_BLOB | The certificate store is a string that contains the server certificate. |
Specifies the SSL client certificate used to authenticate with the Apache Kafka broker.
This property provides the certificate or certificate store that the Sync App uses for SSL client authentication. The expected format depends on the value of SSLClientCertType, which determines whether the certificate is supplied as a file path, a base64-encoded string, or a reference to a system certificate store.
If the certificate store is password protected, set the password in SSLClientCertPassword.
See SSLClientCertType for details about supported certificate formats and how to supply them.
Specifies the format of the SSL client certificate used to connect to the Apache Kafka broker.
This property is used to determine what format the SSLClientCert property expects. This property can take one of the following values:
| Store Type | Description |
| PEMKEY_FILE | The certificate store is the name of a PEM-encoded file that contains a private key and certificate. |
| PEMKEY_BLOB | The certificate store is a string that contains a private key and certificate, optionally encoded in base64. |
Specifies the password used to decrypt the certificate provided in SSLClientCert .
Leave this property blank if the client certificate or keystore is not password protected.
Specifies the endpoint identification algorithm used to validate the server host name during SSL/TLS connections.
By default, this property uses https, which enables host name verification. If you set this property to a blank value, host name verification is disabled.
Host name verification ensures that the certificate presented by the server matches the expected server host name. Disabling this verification can be useful in development or testing environments, but is not recommended for production.
This section provides a complete list of the Schema Registry properties you can configure in the connection string for this provider.
| Property | Description |
| RegistryURL | Specifies the endpoint of the schema registry. When this property is specified, the driver supports reading Avro and JSON schemas from the server. |
| RegistryService | Specifies the schema registry service that the provider uses to retrieve key and value schemas for Apache Kafka topics. |
| RegistryAuthScheme | Specifies the authentication scheme that the provider uses when connecting to the schema registry. |
| RegistryUser | Specifies the user name used when authenticating to the schema registry with the Basic authentication scheme. |
| RegistryPassword | Specifies the password used when authenticating to the schema registry with the Basic authentication scheme. |
| RegistryClientCert | The TLS/SSL client certificate store for SSL Client Authentication (2-way SSL) with the schema registry. |
| RegistryClientCertType | Specifies the type of key store used by the TLS/SSL client certificate given in RegistryClientCert . |
| RegistryClientCertPassword | Specifies the password for the client certificate store defined in RegistryClientCert. |
| RegistryClientCertSubject | Specifies the subject of the client certificate to select from the certificate store defined in RegistryClientCert. |
| RegistryVersion | Specifies which version of a schema the provider retrieves from the schema registry when resolving topic columns. |
| RegistryServerCert | The certificate to be accepted from the schema registry when connecting using TLS/SSL. |
| SchemaMergeMode | Specifies how the provider exposes schemas with multiple versions. |
Specifies the endpoint of the schema registry. When this property is specified, the driver supports reading Avro and JSON schemas from the server.
Set this property to the URL or identifier of the schema registry service you are connecting to.
Specifies the schema registry service that the provider uses to retrieve key and value schemas for Apache Kafka topics.
This property determines which registry implementation the Sync App connects to when using a registry-based TypeDetectionScheme. The supported services are:
| Value | Description |
| Confluent | Use Confluent Schema Registry. Supports both key and value schemas and client certificate authentication. |
| AWSGlue | Use AWS Glue Schema Registry. Supports schema registration and discovery through AWS Glue. |
Specifies the authentication scheme that the provider uses when connecting to the schema registry.
The following authentication schemes are supported. Some schemes are available only for specific registry services, depending on the capabilities of the configured RegistryService.
| Value | Description |
| None | No authentication is used when connecting to the schema registry. |
| Basic | The Sync App uses RegistryUser and RegistryPassword. For Confluent Schema Registry, these values correspond to the API key and secret. For AWS Glue Schema Registry, these are the IAM access key and secret key. |
| SSLCertificate | The Sync App authenticates using RegistryClientCert using SSL client authentication. This option is supported only when connecting to a Confluent registry. |
Specifies the user name used when authenticating to the schema registry with the Basic authentication scheme.
Set this property when RegistryAuthScheme is set to Basic, as Basic authentication requires both a user name and a password.
The meaning of this value depends on the registry service:
This property works together with RegistryPassword, which supplies the corresponding secret or key.
Specifies the password used when authenticating to the schema registry with the Basic authentication scheme.
Set this property when RegistryAuthScheme is set to Basic. For Confluent Schema Registry, this value corresponds to the Secret Key shown under Schemas > Schema Registry > API access.
The TLS/SSL client certificate store for SSL Client Authentication (2-way SSL) with the schema registry.
Set this property to the certificate store that contains the client certificate and its associated private key. The type of the store is determined by RegistryClientCertType, and password-protected stores require a value in RegistryClientCertPassword.
If both RegistryClientCert and RegistryClientCertSubject are set, the Sync App searches the certificate store for a certificate whose subject matches the value of RegistryClientCertSubject.
Designations of certificate stores depend on the platform. On Windows, the following logical store names are commonly used:
| Store Name | Description |
| MY | Personal certificates, including private keys. |
| CA | Intermediate and certifying authority certificates. |
| ROOT | Trusted root certificates. |
| SPC | Software publisher certificates. |
On Java platforms, the certificate store is typically a file containing certificates and optional private keys.
When the certificate store type is PFXFile, set this property to the path to the PFX file. When the type is PFXBlob, set this property to the binary contents of the PFX (PKCS#12) file.
Specifies the type of key store used by the TLS/SSL client certificate given in RegistryClientCert .
This property can take one of the following values:
| Store Type | Description |
| USER - default | For Windows, this specifies that the certificate store is a certificate store owned by the current user. Note that this store type is not available in Java. |
| MACHINE | For Windows, this specifies that the certificate store is a machine store. Note that this store type is not available in Java. |
| PFXFILE | The certificate store is the name of a PFX (PKCS12) file containing certificates. |
| PFXBLOB | The certificate store is a string (base-64-encoded) representing a certificate store in PFX (PKCS12) format. |
| JKSFILE | The certificate store is the name of a Java key store (JKS) file containing certificates. Note that this store type is only available in Java. |
| JKSBLOB | The certificate store is a string (base-64-encoded) representing a certificate store in JKS format. Note that this store type is only available in Java. |
| PEMKEY_FILE | The certificate store is the name of a PEM-encoded file that contains a private key and an optional certificate. |
| PEMKEY_BLOB | The certificate store is a string (base64-encoded) that contains a private key and an optional certificate. |
| PUBLIC_KEY_FILE | The certificate store is the name of a file that contains a PEM- or DER-encoded public key certificate. |
| PUBLIC_KEY_BLOB | The certificate store is a string (base-64-encoded) that contains a PEM- or DER-encoded public key certificate. |
| SSHPUBLIC_KEY_FILE | The certificate store is the name of a file that contains an SSH-style public key. |
| SSHPUBLIC_KEY_BLOB | The certificate store is a string (base-64-encoded) that contains an SSH-style public key. |
| P7BFILE | The certificate store is the name of a PKCS7 file containing certificates. |
| PPKFILE | The certificate store is the name of a file that contains a PuTTY Private Key (PPK). |
| XMLFILE | The certificate store is the name of a file that contains a certificate in XML format. |
| XMLBLOB | The certificate store is a string that contains a certificate in XML format. |
Specifies the password for the client certificate store defined in RegistryClientCert.
Set this property when the certificate store requires a password to access the client certificate or its private key. This applies to password-protected store types such as PFXFile and PFXBlob.
Specifies the subject of the client certificate to select from the certificate store defined in RegistryClientCert.
The Sync App uses this value to locate a certificate within the specified store. If an exact subject match is not found, the Sync App searches for certificates whose subjects contain the value of this property. If no match is found, no certificate is selected and the property is set to an empty string.
The special value * selects the first certificate in the certificate store.
Certificate subjects use distinguished name (DN) format, which consists of comma-separated field/value pairs. For example: CN=www.server.com, OU=Test, C=US. Common DN fields include:
| Field | Meaning |
| CN | Common Name, typically a host name. |
| O | Organization. |
| OU | Organizational Unit. |
| L | Locality. |
| S | State or province. |
| C | Country. |
| E | Email address. |
If a field value contains a comma, it must be quoted.
Specifies which version of a schema the provider retrieves from the schema registry when resolving topic columns.
Set this property to latest to use the most recent version of each schema, or set it to a specific version number. When a version number is provided, that version must exist for every schema subject used by the topics being read.
If SchemaMergeMode is set to Simple, this property is ignored. In that mode, the Sync App merges all available schema versions for each topic to produce a combined set of columns.
The certificate to be accepted from the schema registry when connecting using TLS/SSL.
If you are using a TLS/SSL connection, use this property to specify the TLS/SSL certificate to be accepted from the server. If you specify a value for this property, all other certificates that are not trusted by the machine are rejected.
This property can take the following forms:
| Description | Example |
| A full PEM Certificate (example shortened for brevity) | -----BEGIN CERTIFICATE----- MIIChTCCAe4CAQAwDQYJKoZIhv......Qw== -----END CERTIFICATE----- |
| A path to a local file containing the certificate | C:\cert.cer |
| The public key (example shortened for brevity) | -----BEGIN RSA PUBLIC KEY----- MIGfMA0GCSq......AQAB -----END RSA PUBLIC KEY----- |
| The MD5 Thumbprint (hex values can also be either space- or colon-separated) | ecadbdda5a1529c58a1e9e09828d70e4 |
| The SHA1 Thumbprint (hex values can also be either space- or colon-separated) | 34a929226ae0819f2ec14b4a3d904f801cbb150d |
Note: It is possible to use '*' to signify that all certificates should be accepted, but due to security concerns this is not recommended.
Specifies how the provider exposes schemas with multiple versions.
By default the Sync App sets SchemaMergeMode to None.
This mode only supports one version for schemas in the registry. It is normally the latest version, but you can change RegistryVersion to use a specific version number. The Sync App ignores the content of any message that does not match the schema for its topic. Reading the topic returns the message, but all of its data fields (fields other than Partition, Offset, and Timestamp) are reported as NULL.
This mode supports both SELECT and INSERT queries into each topic. An INSERT always uses the version of the schema specified by RegistryVersion.
This mode supports all options for RegistryService.
For compatibility with previous versions, the Sync App does not enforce the schema ID included on messages when RegistryService is set to Confluent. With SchemaMergeMode set to None this ID is always ignored, but even with SchemaMergeMode set to Simple the Sync App ignores the ID if it cannot find a matching schema. This may cause the Sync App to output field values under unexpected columns.
For example, consider the following two Avro schemas that store names and address details. The schemas are binary compatible: even though the field names differ,
they have the same number of fields with the same types in the same order.
{
"type": "record",
"name": "personname",
"fields": [
{ "name": "PersonID", "type": "int" },
{ "name": "LastName", "type": "string" },
{ "name": "FirstName", "type": "string" }
],
}
{
"type": "record",
"name": "personaddress",
"fields": [
{ "name": "PersonID", "type": "int" },
{ "name": "Address", "type": "string" },
{ "name": "City", "type": "string" }
],
}
If you produce these messages to the topic using the personname schema,
the Sync App may parse these messages using the personaddress
schema. This happens if, for example, personname and personaddress are
two versions of the same registry schema. The Sync App sees that personaddress
is the latest version and uses it for this topic.
{"PersonID": 1, "LastName": "Smithers", "FirstName": "William"}
{"PersonID": 2, "LastName": "McAllister", "FirstName": "Amy"}
In that scenario, the Sync App outputs these results:
| PersonID | Address | City |
| 1 | Smithers | William |
| 2 | McAllister | Amy |
Setting SchemaMergeMode to Simple causes the Sync App to load all versions of each topic schema and merge them according to the following rules. These rules ensure that the Sync App produces NULL or a valid value for each column. If any rule fails, the Sync App fails the schema merge by logging an error and outputting a schema with no data fields.
This mode supports only SELECT queries. The Sync App does not have a way to specify a specific version of a schema to use for INSERT queries. If you need to produce messages in this mode, use the ProduceMessage stored procedure.
This mode only supports RegistryService when set to Confluent. Messages produced with the Confluent libraries include the ID of the schema their data conforms to. The Sync App uses this to determine what schema to parse each message with.
If a message does not have an ID, or if the ID refers to a schema that does not match the topic name, the Sync App defaults to the latest schema. This may cause field values to appear in unexpected columns if the schemas are different but produce compatible output. See the Schema Confusion section above for a more detailed discussion of this issue.
If all versions of the schema are valid according to these rules, the Sync App includes every field from every version of the schema in the table.
During validation, the type of a field is the type that the Sync App uses to report the field, not its original Avro type.
This means that two versions of a schema can have a field which in one version is an aggregate (array, map, ...) and another is a string.
For example, the Sync App considers these two schemas compatible, but there is currently no way to tell whether the address field is JSON or just text.
{
"type": "record"
"name": "person",
"fields": [
{ "name": "address", "type": {"type": "array", "items": "string"}}
]
}
{
"type": "record"
"name": "person",
"fields": [
{ "name": "address", "type": "string" }
]
}
The rules apply transitively across all versions. This means that two versions of the schema may be valid in isolation, but not when considering all versions of the schema. For example, consider a schema where v1 contains an integer amount field, v2 removes it, and v3 adds a decimal amount field. v1 and v2 are valid together because removing fields is allowed, and v2 and v3 are valid together because adding fields is allowed. However, all three versions combined violate the rules because the amount field changed type between v1 and v3.
For best results, enable one of the transitive schema compatibility modes within the schema registry. The Sync App does not check the compatibility mode as part of its validation rules. However, setting a transitive schema compatibility mode prevents you from creating schemas that the Sync App cannot process.
This section provides a complete list of the Firewall properties you can configure in the connection string for this provider.
| Property | Description |
| FirewallType | Specifies the protocol the provider uses to tunnel traffic through a proxy-based firewall. |
| FirewallServer | Identifies the IP address, DNS name, or host name of a proxy used to traverse a firewall and relay user queries to network resources. |
| FirewallPort | Specifies the TCP port to be used for a proxy-based firewall. |
| FirewallUser | Identifies the user ID of the account authenticating to a proxy-based firewall. |
| FirewallPassword | Specifies the password of the user account authenticating to a proxy-based firewall. |
Specifies the protocol the provider uses to tunnel traffic through a proxy-based firewall.
A proxy-based firewall (or proxy firewall) is a network security device that acts as an intermediary between user requests and the resources they access. The proxy accepts the request of an authenticated user, tunnels through the firewall, and transmits the request to the appropriate server.
Because the proxy evaluates and transfers data backets on behalf of the requesting users, the users never connect directly with the servers, only with the proxy.
Note: By default, the Sync App connects to the system proxy. To disable this behavior and connect to one of the following proxy types, set ProxyAutoDetect to false.
The following table provides port number information for each of the supported protocols.
| Protocol | Default Port | Description |
| TUNNEL | 80 | The port where the Sync App opens a connection to Apache Kafka. Traffic flows back and forth via the proxy at this location. |
| SOCKS4 | 1080 | The port where the Sync App opens a connection to Apache Kafka. SOCKS 4 then passes theFirewallUser value to the proxy, which determines whether the connection request should be granted. |
| SOCKS5 | 1080 | The port where the Sync App sends data to Apache Kafka. If the SOCKS 5 proxy requires authentication, set FirewallUser and FirewallPassword to credentials the proxy recognizes. |
To connect to HTTP proxies, use ProxyServer and ProxyPort. To authenticate to HTTP proxies, use ProxyAuthScheme, ProxyUser, and ProxyPassword.
Identifies the IP address, DNS name, or host name of a proxy used to traverse a firewall and relay user queries to network resources.
A proxy-based firewall (or proxy firewall) is a network security device that acts as an intermediary between user requests and the resources they access. The proxy accepts the request of an authenticated user, tunnels through the firewall, and transmits the request to the appropriate server.
Because the proxy evaluates and transfers data backets on behalf of the requesting users, the users never connect directly with the servers, only with the proxy.
Specifies the TCP port to be used for a proxy-based firewall.
A proxy-based firewall (or proxy firewall) is a network security device that acts as an intermediary between user requests and the resources they access. The proxy accepts the request of an authenticated user, tunnels through the firewall, and transmits the request to the appropriate server.
Because the proxy evaluates and transfers data backets on behalf of the requesting users, the users never connect directly with the servers, only with the proxy.
Identifies the user ID of the account authenticating to a proxy-based firewall.
A proxy-based firewall (or proxy firewall) is a network security device that acts as an intermediary between user requests and the resources they access. The proxy accepts the request of an authenticated user, tunnels through the firewall, and transmits the request to the appropriate server.
Because the proxy evaluates and transfers data backets on behalf of the requesting users, the users never connect directly with the servers, only with the proxy.
Specifies the password of the user account authenticating to a proxy-based firewall.
A proxy-based firewall (or proxy firewall) is a network security device that acts as an intermediary between user requests and the resources they access. The proxy accepts the request of an authenticated user, tunnels through the firewall, and transmits the request to the appropriate server.
Because the proxy evaluates and transfers data backets on behalf of the requesting users, the users never connect directly with the servers, only with the proxy.
This section provides a complete list of the Proxy properties you can configure in the connection string for this provider.
| Property | Description |
| ProxyAutoDetect | Specifies whether the provider checks your system proxy settings for existing proxy server configurations, rather than using a manually specified proxy server. |
| ProxyServer | Identifies the hostname or IP address of the proxy server through which you want to route HTTP traffic. |
| ProxyPort | Identifies the TCP port on your specified proxy server that has been reserved for routing HTTP traffic to and from the client. |
| ProxyAuthScheme | Specifies the authentication method the provider uses when authenticating to the proxy server specified in the ProxyServer connection property. |
| ProxyUser | Provides the username of a user account registered with the proxy server specified in the ProxyServer connection property. |
| ProxyPassword | Specifies the password of the user specified in the ProxyUser connection property. |
| ProxySSLType | Specifies the SSL type to use when connecting to the proxy server specified in the ProxyServer connection property. |
| ProxyExceptions | Specifies a semicolon-separated list of destination hostnames or IPs that are exempt from connecting through the proxy server set in the ProxyServer connection property. |
Specifies whether the provider checks your system proxy settings for existing proxy server configurations, rather than using a manually specified proxy server.
When this connection property is set to True, the Sync App checks your system proxy settings for existing proxy server configurations (no need to manually supply proxy server details).
This connection property takes precedence over other proxy settings. If you want to configure the Sync App to connect to a specific proxy server, set ProxyAutoDetect to False.
To connect to an HTTP proxy, see ProxyServer. For other proxies, such as SOCKS or tunneling, see FirewallType.
Identifies the hostname or IP address of the proxy server through which you want to route HTTP traffic.
The Sync App only routes HTTP traffic through the proxy server specified in this connection property when ProxyAutoDetect is set to False.
If ProxyAutoDetect is set to True (the default), the Sync App instead routes HTTP traffic through the proxy server specified in your system proxy settings.
Identifies the TCP port on your specified proxy server that has been reserved for routing HTTP traffic to and from the client.
The Sync App only routes HTTP traffic through the ProxyServer port specified in this connection property when ProxyAutoDetect is set to False.
If ProxyAutoDetect is set to True (the default), the Sync App instead routes HTTP traffic through the proxy server port specified in your system proxy settings.
For other proxy types, see FirewallType.
Specifies the authentication method the provider uses when authenticating to the proxy server specified in the ProxyServer connection property.
Supported authentication types :
For all values other than NONE, you must also set the ProxyUser and ProxyPassword connection properties.
If you need to use another authentication type, such as SOCKS 5 authentication, see FirewallType.
Provides the username of a user account registered with the proxy server specified in the ProxyServer connection property.
The ProxyUser and ProxyPassword connection properties are used to connect and authenticate against the HTTP proxy specified in ProxyServer.
After selecting one of the available authentication types in ProxyAuthScheme, set this property as follows:
| ProxyAuthScheme Value | Value to set for ProxyUser |
| BASIC | The username of a user registered with the proxy server. |
| DIGEST | The username of a user registered with the proxy server. |
| NEGOTIATE | The username of a Windows user who is a valid user in the domain or trusted domain that the proxy server is part of, in the format user@domain or domain\user. |
| NTLM | The username of a Windows user who is a valid user in the domain or trusted domain that the proxy server is part of, in the format user@domain or domain\user. |
| NONE | Do not set the ProxyPassword connection property. |
Note: The Sync App only uses this username if ProxyAutoDetect is set to False. If ProxyAutoDetect is set to True (the default), the Sync App instead uses the username specified in your system proxy settings.
Specifies the password of the user specified in the ProxyUser connection property.
The ProxyUser and ProxyPassword connection properties are used to connect and authenticate against the HTTP proxy specified in ProxyServer.
After selecting one of the available authentication types in ProxyAuthScheme, set this property as follows:
| ProxyAuthScheme Value | Value to set for ProxyPassword |
| BASIC | The password associated with the proxy server user specified in ProxyUser. |
| DIGEST | The password associated with the proxy server user specified in ProxyUser. |
| NEGOTIATE | The password associated with the Windows user account specified in ProxyUser. |
| NTLM | The password associated with the Windows user account specified in ProxyUser. |
| NONE | Do not set the ProxyPassword connection property. |
For SOCKS 5 authentication or tunneling, see FirewallType.
Note: The Sync App only uses this password if ProxyAutoDetect is set to False. If ProxyAutoDetect is set to True (the default), the Sync App instead uses the password specified in your system proxy settings.
Specifies the SSL type to use when connecting to the proxy server specified in the ProxyServer connection property.
This property determines when to use SSL for the connection to the HTTP proxy specified by ProxyServer. You can set this connection property to the following values :
| AUTO | Default setting. If ProxyServer is set to an HTTPS URL, the Sync App uses the TUNNEL option. If ProxyServer is set to an HTTP URL, the component uses the NEVER option. |
| ALWAYS | The connection is always SSL enabled. |
| NEVER | The connection is not SSL enabled. |
| TUNNEL | The connection is made through a tunneling proxy. The proxy server opens a connection to the remote host and traffic flows back and forth through the proxy. |
Specifies a semicolon-separated list of destination hostnames or IPs that are exempt from connecting through the proxy server set in the ProxyServer connection property.
The ProxyServer is used for all addresses, except for addresses defined in this property. Use semicolons to separate entries.
Note: The Sync App uses the system proxy settings by default, without further configuration needed. If you want to explicitly configure proxy exceptions for this connection, set ProxyAutoDetect to False.
This section provides a complete list of the Logging properties you can configure in the connection string for this provider.
| Property | Description |
| LogModules | Specifies the core modules to include in the log file. Use a semicolon-separated list of module names. By default, all modules are logged. |
Specifies the core modules to include in the log file. Use a semicolon-separated list of module names. By default, all modules are logged.
The Sync App writes details about each operation it performs into the logfile specified by the Logfile connection property.
Each of these logged operations are assigned to a themed category called a module, and each module has a corresponding short code used to labels individual Sync App operations as belonging to that module.
When this connection property is set to a semicolon-separated list of module codes, only operations belonging to the specified modules are written to the logfile. Note that this only affects which operations are logged moving forward and doesn't retroactively alter the existing contents of the logfile. For example: INFO;EXEC;SSL;META;
By default, logged operations from all modules are included.
You can explicitly exclude a module by prefixing it with a "-". For example: -HTTP
To apply filters to submodules, identify them with the syntax <module name>.<submodule name>. For example, the following value causes the Sync App to only log actions belonging to the HTTP module, and further refines it to exclude actions belonging to the Res submodule of the HTTP module: HTTP;-HTTP.Res
Note that the logfile filtering triggered by the Verbosity connection property takes precedence over the filtering imposed by this connection property. This means that operations of a higher verbosity level than the level specified in the Verbosity connection property are not printed in the logfile, even if they belong to one of the modules specified in this connection property.
The available modules and submodules are:
| Module Name | Module Description | Submodules |
| INFO | General Information. Includes the connection string, product version (build number), and initial connection messages. |
|
| EXEC | Query Execution. Includes execution messages for user-written SQL queries, parsed SQL queries, and normalized SQL queries. Success/failure messages for queries and query pages appear here as well. |
|
| HTTP | HTTP protocol messages. Includes HTTP requests/responses (including POST messages), as well as Kerberos related messages. |
|
| WSDL | Messages pertaining to the generation of WSDL/XSD files. | — |
| SSL | SSL certificate messages. |
|
| AUTH | Authentication related failure/success messages. |
|
| SQL | Includes SQL transactions, SQL bulk transfer messages, and SQL result set messages. |
|
| META | Metadata cache and schema messages. |
|
| FUNC | Information related to executing SQL functions. |
|
| TCP | Incoming and outgoing raw bytes on TCP transport layer messages. |
|
| FTP | Messages pertaining to the File Transfer Protocol. |
|
| SFTP | Messages pertaining to the Secure File Transfer Protocol. |
|
| POP | Messages pertaining to data transferred via the Post Office Protocol. |
|
| SMTP | Messages pertaining to data transferred via the Simple Mail Transfer Protocol. |
|
| CORE | Messages relating to various internal product operations not covered by other modules. | — |
| DEMN | Messages related to SQL remoting. | — |
| CLJB | Messages about bulk data uploads (cloud job). |
|
| SRCE | Miscellaneous messages produced by the product that don't belong in any other module. | — |
| TRANCE | Advanced messages concerning low-level product operations. | — |
| KFKA | Applies to log messages generated from the Apache Kafka protocol. | — |
This section provides a complete list of the Schema properties you can configure in the connection string for this provider.
| Property | Description |
| Location | Specifies the location of a directory containing schema files that define tables, views, and stored procedures. Depending on your service's requirements, this may be expressed as either an absolute path or a relative path. |
| BrowsableSchemas | Optional setting that restricts the schemas reported to a subset of all available schemas. For example, BrowsableSchemas=SchemaA,SchemaB,SchemaC . |
| Tables | Optional setting that restricts the tables reported to a subset of all available tables. For example, Tables=TableA,TableB,TableC . |
| Views | Optional setting that restricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC . |
Specifies the location of a directory containing schema files that define tables, views, and stored procedures. Depending on your service's requirements, this may be expressed as either an absolute path or a relative path.
The Location property is only needed if you want to either customize definitions (for example, change a column name, ignore a column, etc.) or extend the data model with new tables, views, or stored procedures.
If left unspecified, the default location is %APPDATA%\\CData\\ApacheKafka Data Provider\\Schema, where %APPDATA% is set to the user's configuration directory:
| Platform | %APPDATA% |
| Windows | The value of the APPDATA environment variable |
| Linux | ~/.config |
Optional setting that restricts the schemas reported to a subset of all available schemas. For example, BrowsableSchemas=SchemaA,SchemaB,SchemaC .
Listing all available database schemas can take extra time, thus degrading performance. Providing a list of schemas in the connection string saves time and improves performance.
Optional setting that restricts the tables reported to a subset of all available tables. For example, Tables=TableA,TableB,TableC .
Listing all available tables from some databases can take extra time, thus degrading performance. Providing a list of tables in the connection string saves time and improves performance.
If there are lots of tables available and you already know which ones you want to work with, you can use this property to restrict your viewing to only those tables. To do this, specify the tables you want in a comma-separated list. Each table should be a valid SQL identifier with any special characters escaped using square brackets, double-quotes or backticks. For example, Tables=TableA,[TableB/WithSlash],WithCatalog.WithSchema.`TableC With Space`.
Note: If you are connecting to a data source with multiple schemas or catalogs, you must specify each table you want to view by its fully qualified name. This avoids ambiguity between tables that may exist in multiple catalogs or schemas.
Optional setting that restricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC .
Listing all available views from some databases can take extra time, thus degrading performance. Providing a list of views in the connection string saves time and improves performance.
If there are lots of views available and you already know which ones you want to work with, you can use this property to restrict your viewing to only those views. To do this, specify the views you want in a comma-separated list. Each view should be a valid SQL identifier with any special characters escaped using square brackets, double-quotes or backticks. For example, Views=ViewA,[ViewB/WithSlash],WithCatalog.WithSchema.`ViewC With Space`.
Note: If you are connecting to a data source with multiple schemas or catalogs, you must specify each view you want to examine by its fully qualified name. This avoids ambiguity between views that may exist in multiple catalogs or schemas.
This section provides a complete list of the Miscellaneous properties you can configure in the connection string for this provider.
| Property | Description |
| AllowKeyOnlyRegistryTopics | Specifies whether the provider exposes key-only Schema Registry topics as tables. |
| AWSWorkloadIdentityConfig | Configuration properties to provide when using Workload Identity Federation via AWS. |
| AzureWorkloadIdentityConfig | Configuration properties to provide when using Workload Identity Federation via Azure. |
| CompressionType | Specifies the compression algorithm that the provider uses when producing messages to Apache Kafka. |
| ConsumerProperties | Specifies additional Kafka consumer configuration options that the provider passes directly to the underlying Kafka client. |
| CreateTablePartitions | Specifies the number of partitions to assign to a topic created through a CREATE TABLE statement. |
| CreateTableReplicationFactor | Specifies the the number of replicas to assign to a topic created through a CREATE TABLE statement. |
| EnableIdempotence | Specifies whether the provider ensures that produced messages are delivered in order and without duplicates. |
| ExposeQueueMetadataColumns | Specifies whether the Partition, Offset, and Timestamp columns are exposed. |
| FlattenArrays | Specifies how many elements to return from nested arrays when TypeDetectionScheme is set to SchemaRegistry. |
| GenerateSchemaFiles | Indicates the user preference as to when schemas should be generated and saved. |
| HideUnusedColumns | Determines whether to hide key or value colums when the topic has no associated schema information. |
| MaximumBatchSize | Specifies the maximum size, in bytes, of a batch of messages that the provider gathers before sending the batch to Apache Kafka. |
| MaxRows | Specifies the maximum number of rows returned for queries that do not include either aggregation or GROUP BY. |
| MessageKeyColumn | Specifies the name of the column where the provider stores the message key for each record. |
| MessageKeyType | The type of data stored in message keys. |
| NonRegistryTypeDetectionScheme | Specifies the TypeDetectionScheme to use for topics that do not have schemas in the schema registry. |
| OffsetResetStrategy | Specifies how the provider determines the starting offset when no committed offset exists for the consumer group. |
| Other | Specifies advanced connection properties for specialized scenarios. Use this property only under the guidance of our Support team to address specific issues. |
| Pagesize | Specifies the maximum number of rows that the provider retrieves from Apache Kafka in a single read operation. |
| ProducerProperties | Specifies additional Apache Kafka producer configuration options that the provider passes directly to the client. |
| PseudoColumns | Specifies the pseudocolumns to expose as table columns, expressed as a string in the format 'TableName=ColumnName;TableName=ColumnName'. |
| ReadDuration | Specifies how long, in seconds, the provider waits for additional messages after a read operation begins. |
| RowScanDepth | Specifies the maximum number of messages that the provider scans to determine the columns and data types for a topic. |
| SASLOAuthExtensions | Specifies the extension values to send with OAuth auth schemes. |
| SchemaRegistryOnly | Specifies whether the provider connects only to the schema registry. |
| SerializationFormat | Specifies how to serialize/deserialize message contents. |
| Timeout | Specifies the maximum time, in seconds, that the provider waits for a server response before throwing a timeout error. |
| TypeDetectionScheme | Specifies how the provider determines the available fields and data types for each topic. |
| UseConfluentAvroFormat | Specifies how Avro data should be formatted during an INSERT. |
| UserDefinedViews | Specifies a filepath to a JSON configuration file that defines custom views. The provider automatically detects and uses the views specified in this file. |
| ValidateRegistryTopics | Specifies whether or not to validate schema registry topics against the Apache Kafka broker. Only has an effect when TypeDetectionScheme =SchemaRegistry. |
| WorkloadPoolId | The ID of your Workload Identity Federation pool. |
| WorkloadProjectId | The ID of the Google Cloud project that hosts your Workload Identity Federation pool. |
| WorkloadProviderId | The ID of your Workload Identity Federation pool provider. |
Specifies whether the provider exposes key-only Schema Registry topics as tables.
This property applies only when TypeDetectionScheme is set to SchemaRegistry.
The Sync App supports two types of schemas in the Schema Registry: value schemas and key schemas. By default, the Sync App requires a value schema for a topic to be exposed as a table. Topics that contain only a key schema are not exposed unless this property is enabled. Key schemas are optional and are used only when MessageKeyColumn is set.
When set to true, the Sync App exposes topics that contain either a key schema, a value schema, or both. For topics without a value schema, the message value is returned as a base64 blob because no deserialization can be performed. This behavior is similar to scenarios where SerializationFormat is set to NONE or TypeDetectionScheme is set to MessageOnly.
When set to false, topics without a value schema are not exposed. This is the default behavior and ensures that only topics with complete value schemas are represented as tables.
This behavior is independent of the MessageKeyColumn setting. Even when this property is enabled, you must set MessageKeyColumn if you want the Sync App to return message key data.
Enable this property when you need to read topics defined only with key schemas or when working with registries where not all topics include value schemas. Enabling this property may increase the number of exposed tables and may require additional processing when handling base64-encoded message values.
Configuration properties to provide when using Workload Identity Federation via AWS.
The properties are formatted as a semicolon-separated list of Key=Value properties, where the value is optionally quoted.
For example, this setting authenticates in AWS using a user's root keys:
AWSWorkloadIdentityConfig="AuthScheme=AwsRootKeys;AccessKey='AKIAABCDEF123456';SecretKey=...;Region=us-east-1"
Configuration properties to provide when using Workload Identity Federation via Azure.
The properties are formatted as a semicolon-separated list of Key=Value properties, where the value is optionally quoted.
For example, this setting authenticates in Azure using client credentials:
AzureWorkloadIdentityConfig="AuthScheme=AzureServicePrincipal;AzureTenant=directory (tenant) id;OAuthClientID=application (client) id;OAuthClientSecret=client secret;AzureResource=application id uri;"
Specifies the compression algorithm that the provider uses when producing messages to Apache Kafka.
Select a compression type to reduce the size of produced messages. Compression is applied to batches of messages rather than individual messages.
The following compression types are supported:
| Type | Description |
| NONE | Messages are sent without compression. |
| GZIP | Messages are compressed using gzip. |
| SNAPPY | Messages are compressed using Snappy. |
| LZ4 | Messages are compressed using LZ4. |
Specifies additional Kafka consumer configuration options that the provider passes directly to the underlying Kafka client.
The Sync App exposes several Kafka consumer settings as connection properties and maps them internally to the corresponding Kafka client configuration keys. If the Sync App does not provide a dedicated property for a particular consumer option, you can set it through ConsumerProperties.
This property accepts a semicolon-separated list of Kafka consumer configuration pairs. These options are passed directly to the Kafka client. For example, security.protocol=SASL_SSL;sasl.mechanism=SCRAM-SHA-512 sets the security.protocol and sasl.mechanism Kafka consumer properties.
Be careful what configuration options you set via this property. The Sync App does not consider ConsumerProperties a sensitive property and its value appears in logs. Use this property to configure advanced Kafka consumer settings that are not directly exposed as connection properties, such as fine-tuning fetch behavior, retry intervals, timeouts, and security configuration details.
Specifies the number of partitions to assign to a topic created through a CREATE TABLE statement.
When you execute a CREATE TABLE statement, the Sync App creates a new empty Kafka topic. By default, this topic is created with one partition.
Increase this value to create topics with additional partitions. More partitions allow Kafka to distribute messages across multiple consumers within the same consumer group, enabling greater parallelism and higher throughput. Choose a partition count that aligns with your expected workload and consumer group size.
Specifies the the number of replicas to assign to a topic created through a CREATE TABLE statement.
When you execute a CREATE TABLE statement, the Sync App creates a new empty Kafka topic. By default, the topic is created with a replication factor of 3.
You can adjust this value to create topics with more or fewer replicas. The replication factor must not exceed the number of brokers in your Kafka cluster. For example, a topic with a replication factor of 3 cannot be created on a cluster with only 2 brokers.
Kafka uses replicas to maintain availability and prevent data loss during broker failures. If all replicas for a partition become unavailable, the topic cannot be accessed. Increasing the number of replicas can improve resilience on larger clusters.
Specifies whether the provider ensures that produced messages are delivered in order and without duplicates.
When set to true, the Sync App enables Kafka's idempotent producer mode. Kafka assigns a sequence number to each produced message, allowing the broker to detect and discard duplicates and to preserve message order during retries.
When set to false, the Sync App does not apply these delivery guarantees, and duplicate messages may be produced if retries occur during network interruptions or broker failures.
Specifies whether the Partition, Offset, and Timestamp columns are exposed.
Apache Kafka messages include three pieces of metadata along with every message. These are the timestamp when the message was produced, what partition it was produced to, and the message's offset within that partition. The Sync App exposes these as the Timestamp, Partition, and Offset columns.
When set to true, these metadata columns are included in every table. When set to false, the Sync App hides these columns, but still receives the underlying metadata.
Consider the following points when deciding whether to disable this option:
Specifies how many elements to return from nested arrays when TypeDetectionScheme is set to SchemaRegistry.
When TypeDetectionScheme is set to SchemaRegistry, nested arrays are not exposed by default.
Set this property to expose a specific number of elements from a nested array as individual columns. The Sync App creates one column per element, starting with index 0.
For example, consider the following array:
["FLOW-MATIC","LISP","COBOL"]
If this property is set to 1, the Sync App exposes only the element at index 0:
| Column Name | Column Value |
| languages.0 | FLOW-MATIC |
If this property is set to 0, no elements of nested arrays are exposed as columns.
Indicates the user preference as to when schemas should be generated and saved.
This property outputs schemas to .rsd files in the path specified by Location.
Available settings are the following:
When you set GenerateSchemaFiles to OnUse, the Sync App generates schemas as you execute SELECT queries. Schemas are generated for each table referenced in the query.
When you set GenerateSchemaFiles to OnCreate, schemas are only generated when a CREATE TABLE query is executed.
Another way to use this property is to obtain schemas for every table in your database when you connect. To do so, set GenerateSchemaFiles to OnStart and connect.
Determines whether to hide key or value colums when the topic has no associated schema information.
In some situations the Sync App has no information on the type of data stored in the keys or values of a topic's messages. This can happen for several reasons:
This option determines whether the Sync App exposes fallback columns for parts of the message that have no schema information. In the following descriptions, value columns include both the Message column used for single values as well as columns flattened from the value. Similarly, key columns include both the top-level key column (named by MessageKeyColumn) and columns flattened from the key. Refer to Extracting Metadata From Topics for more information on column flattening.
By default the Sync App follows these legacy rules, which are compatible with versions of the Sync App before this option was introduced:
The other options provide consistent behavior regardless of the TypeDetectionScheme:
Note that there are no cases where TypeDetectionScheme=MessageOnly and schema information is missing. In MessageOnly mode, the Sync App does not use any external information to determine the format of the topic's values. The Sync App always returns a single Message column that is decoded according to SerializationFormat. Key columns behave the same way but using MessageKeyType.
For the same reason, the Sync App does not consider a column unused if it uses a primitive value format. Setting SerializationFormat to Long, Integer, Float, Double, or String prevents the Sync App from accessing the broker or schema registry to determine value schemas. The same applies for MessageKeyType and key schemas.
Specifies the maximum size, in bytes, of a batch of messages that the provider gathers before sending the batch to Apache Kafka.
A batch may contain one or more messages. The Sync App accumulates messages for a single partition until the total size of the batch reaches this value, at which point the batch is sent to Apache Kafka.
If a single message exceeds the batch size, the message is sent on its own. Apache Kafka uses batching to increase throughput, so larger batch sizes may reduce the number of requests sent to the broker, while smaller batch sizes may send messages more quickly, but with additional request overhead.
Specifies the maximum number of rows returned for queries that do not include either aggregation or GROUP BY.
The default value for this property, -1, means that no row limit is enforced unless the query explicitly includes a LIMIT clause. (When a query includes a LIMIT clause, the value specified in the query takes precedence over the MaxRows setting.)
Setting MaxRows to a whole number greater than 0 ensures that queries do not return excessively large result sets by default.
This property is useful for optimizing performance and preventing excessive resource consumption when executing queries that could otherwise return very large datasets.
Specifies the name of the column where the provider stores the message key for each record.
Set this property to expose message key data as a column in the table schema. If this property is not set, message key data is not included in the result set. See MessageKeyType for details on how the Sync App interprets and formats message keys.
The type of data stored in message keys.
By default the Sync App does not report message keys. To enable message keys, this property must be set to a value other than Null and MessageKeyColumn must be set to a valid column name.
See Extracting Metadata From Topics for a description of how this interacts with the TypeDetectionScheme property. SerializationFormat describes how each of these supported formats is encoded. There are three main differences between how that property and this property work:
Specifies the TypeDetectionScheme to use for topics that do not have schemas in the schema registry.
When TypeDetectionScheme is set to a registry-type detection scheme, the Sync App exposes only topics that have schemas registered in the schema registry. Topics without registry entries are skipped and do not appear as tables.
Set this property to a value other than Disabled to expose both registry and non-registry topics. Topics with registry schemas continue to use the registry for column definitions, and topics without registry schemas use the detection scheme specified in this property.
The available values for this property are: Disabled, None, RowScan, and MessageOnly. These values have the same meaning as the corresponding options in TypeDetectionScheme.
See Extracting Metadata From Topics for details on how type detection works.
Specifies how the provider determines the starting offset when no committed offset exists for the consumer group.
This property applies when the consumer group has no previously committed offset for a partition.
Select one of the following strategies:
| Value | Description |
| Earliest | Start reading from the beginning of the partition. |
| Latest | Start reading from the end of the partition, consuming only messages produced after the consumer group begins reading. |
Once a committed offset exists for a consumer group, Apache Kafka resumes reading from the committed offset regardless of this property's value.
Specifies advanced connection properties for specialized scenarios. Use this property only under the guidance of our Support team to address specific issues.
This property allows advanced users to configure hidden properties for specialized situations, with the advice of our Support team. These settings are not required for normal use cases but can address unique requirements or provide additional functionality. To define multiple properties, use a semicolon-separated list.
Note: It is strongly recommended to set these properties only when advised by the Support team to address specific scenarios or issues.
| Property | Description |
| DefaultColumnSize | Sets the default length of string fields when the data source does not provide column length in the metadata. The default value is 2000. |
| ConvertDateTimeToGMT=True | Converts date-time values to GMT, instead of the local time of the machine. The default value is False (use local time). |
| RecordToFile=filename | Records the underlying socket data transfer to the specified file. |
Specifies the maximum number of rows that the provider retrieves from Apache Kafka in a single read operation.
The Sync App reads messages from Apache Kafka in batches and stores them in memory before returning rows to the result set. Only the first row of each batch requires a network request while subsequent rows are returned directly from the in-memory buffer.
This property controls the maximum number of rows the Sync App holds in this buffer at one time. Higher values reduce how often the Sync App must request new data, while lower values reduce memory usage.
Specifies additional Apache Kafka producer configuration options that the provider passes directly to the client.
Use this property to supply Apache Kafka producer settings that are not exposed as dedicated connection properties. This property accepts a semicolon-separated list of key–value pairs, and all options are passed directly to the underlying producer without validation by the Sync App.
For example, you can specify properties such as acks or compression.type by including them in this property.
Be careful what configuration options you set via this property. The Sync App does not consider ConsumerProperties a sensitive property and its value appears in logs.
See ConsumerProperties for a description of how custom client configuration properties are handled.
Specifies the pseudocolumns to expose as table columns, expressed as a string in the format 'TableName=ColumnName;TableName=ColumnName'.
This property allows you to define which pseudocolumns the Sync App exposes as table columns.
To specify individual pseudocolumns, use the following format:
Table1=Column1;Table1=Column2;Table2=Column3
To include all pseudocolumns for all tables use:
*=*
Specifies how long, in seconds, the provider waits for additional messages after a read operation begins.
This property determines the amount of time the Sync App continues polling Apache Kafka for new messages once it has started reading from a topic. If new messages arrive before the duration expires, they are included in the result set. When the duration elapses with no new messages, the read operation completes.
Specifies the maximum number of messages that the provider scans to determine the columns and data types for a topic.
The Sync App inspects message values to infer a topic’s schema when using a type detection scheme that requires scanning. This property limits how many messages are read during that process.
Higher values allow the Sync App to discover more fields and infer more accurate data types, but may increase startup time for the query. Lower values reduce scanning time, but may cause some fields or data types to be missed if they do not appear within the scanned messages.
Specifies the extension values to send with OAuth auth schemes.
When authenticating using OAuth, the Sync App obtains an OAuth access token according to the chosen AuthScheme. The Sync App uses the Kafka native library to authenticates to the broker using SASL OAUTHBEARER. The OAUTHBEARER mechanism sends the access token along with an optional list of extension settings.
This property accepts a semicolon-separated list of key-value pairs. These options are mapped to extension settings during OAUTHBEARER authentication. For example, clusterId=1;computePool=primary sets the clusterId and computePool extension settings.
Extension settings are vendor-defined, you should refer to your provider's documentation since individual managed Kafka providers may allow (or even require) specific extensions.
Providers typically document these settings as JAAS configurations since the Kafka client library is configured with this format.
JAAS configurations with extension settings prefix them with extension_.
To provide the following configuration to the Sync App, set SASLOAuthExtensions to logicalCluster=clust-123;identityPoolId=mypool.
sasl.jaas.config= \
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
clientId='...'
scope='...'
clientSecret='...'
extension_logicalCluster='clust-123';
extension_identityPoolId='mypool';
This option only applies to the following AuthScheme configurations. Other authentication schemes ignore this property because they do not use OAUTHBEARER.
Specifies whether the provider connects only to the schema registry.
When set to true, the Sync App disables all broker connections and operates only against the schema registry. In this mode, the Sync App does not perform any operations that requires access to the Apache Kafka broker. Attempting any of the following operations in registry-only mode results in an error:
Registry-only mode requires the following configuration settings:
Specifies how to serialize/deserialize message contents.
The Sync App uses this property differently based on the value of TypeDetectionScheme. See Extracting Metadata From Topics for details on how these properties interact.
This section applies only when TypeDetectionScheme is set to SchemaRegistry, None, or RowScan modes. MessageOnly always reports the message as a single column regardless of the format. The only difference is the column type.
The Sync App supports two different kinds of formats: primitive formats and complex formats. Primitive formats are reported in a single column called Message. The primitive formats use encodings that are compatible with the kafka-clients and Confluent.Kafka libraries.
Avro, CSV, CSV_WITH_HEADERS, XML, and JSON are all complex formats. The Sync App parses these formats into one or more columns, flattening nested Avro, XML, and JSON values as necessary.
Auto is also a complex format, but the exact data format is determined at runtime. The Sync App determines whether a value is Avro, CSV, XML, or JSON by looking for either a specific header (the Avro OBJ header) or specific characters. If none of these methods succeed the Sync App assumes the value is CSV.
Available formats:
| Format | Description |
| NONE | Message is always BASE64 encoded on both the consume and produce operations. |
| AUTO | Attempts to automatically detect the current topic's serialization format. See Extracting Metadata From Topics for more information. |
| JSON | Message is serialized using the JSON format. |
| CSV | Message is serialized using the CSV format. |
| CSV_WITH_HEADERS | Message is serialized using the CSV format with a separate header line before the data. This option only applies to messages created using INSERT. In SELECT, it behaves the same as CSV. |
| XML | Message is serialized using the XML format. |
| AVRO | Message is serialized using the Avro format. |
| LONG | Message is serialized as a 64-bit big-endian integer. |
| INTEGER | Message is serialized as a 32-bit big-endian integer. |
| FLOAT | Message is serialized as a 32-bit floating-point number. |
| DOUBLE | Message is serialized as a 64-bit floating-point number. |
| STRING | Message is serialized as text. By default the Sync App uses UTF-8, but setting Charset overrides this. |
Specifies the maximum time, in seconds, that the provider waits for a server response before throwing a timeout error.
The timeout applies to each individual communication with the server rather than the entire query or operation. For example, a query could continue running beyond 60 seconds if each paging call completes within the timeout limit.
Timeout is set to 60 seconds by default. To disable timeouts, set this property to 0.
Disabling the timeout allows operations to run indefinitely until they succeed or fail due to other conditions such as server-side timeouts, network interruptions, or resource limits on the server.
Note: Use this property cautiously to avoid long-running operations that could degrade performance or result in unresponsive behavior.
Specifies how the provider determines the available fields and data types for each topic.
This property controls whether the Sync App reads message data, uses schema registry information, or combines multiple registry versions to detect columns. Different options may require access to the schema registry or may scan message data directly. See Extracting Metadata From Topics for more information on how this property interacts with different values of SerializationFormat, RegistryService, and MessageKeyType.
Specifies how Avro data should be formatted during an INSERT.
When set to false, the Sync App writes Avro data using standard Avro file blocks as defined in the Avro specification. This format allows multiple rows to be written into a single Kafka message and is more compact, but it is not compatible with Confluent tools or Confluent schema validation.
When set to true, the Sync App writes each row as a separate message using the Confluent Avro format. Enable this option if you rely on Confluent schema validation or need compatibility with Confluent serialization libraries.
This option cannot be enabled unless RegistryURL is set and points to a Confluent schema registry. AWS Glue registries do not support schema IDs, which are required by the Confluent Avro format.
Specifies a filepath to a JSON configuration file that defines custom views. The provider automatically detects and uses the views specified in this file.
UserDefinedViews allows you to define and manage custom views through a JSON-formatted configuration file called UserDefinedViews.json. These views are automatically recognized by the Sync App and enable you to execute custom SQL queries as if they were standard database views. The JSON file defines each view as a root element with a child element called "query", which contains the SQL query for the view.
For example:
{
"MyView": {
"query": "SELECT * FROM SampleTable_1 WHERE MyColumn = 'value'"
},
"MyView2": {
"query": "SELECT * FROM MyTable WHERE Id IN (1,2,3)"
}
}
You can use this property to define multiple views in a single file and specify the filepath.
For example:
UserDefinedViews=C:\Path\To\UserDefinedViews.jsonWhen you specify a view in UserDefinedViews, the Sync App only sees that view.
For further information, see User Defined Views.
Specifies whether or not to validate schema registry topics against the Apache Kafka broker. Only has an effect when TypeDetectionScheme =SchemaRegistry.
Schema registries can include metadata for topics that cannot be accessed in Kafka. This can happen because the topic doesn't exist on the broker. It is also possible that the principal the connection is authenticated to does not have access to the topic.
By default, the Sync App will get a list of schemas from the registry and then filter out any that the broker does not report. All the remaining valid topics are exposed as tables. You can disable this behavior by setting this option to false. This will report all schemas in the registry as tables regardless of whether they are accessible on the broker.
The ID of your Workload Identity Federation pool.
The ID of your Workload Identity Federation pool.
The ID of the Google Cloud project that hosts your Workload Identity Federation pool.
The ID of the Google Cloud project that hosts your Workload Identity Federation pool.
The ID of your Workload Identity Federation pool provider.
The ID of your Workload Identity Federation pool provider.
LZMA from 7Zip LZMA SDK
LZMA SDK is placed in the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or distribute the original LZMA SDK code, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.
LZMA2 from XZ SDK
Version 1.9 and older are in the public domain.
Xamarin.Forms
Xamarin SDK
The MIT License (MIT)
Copyright (c) .NET Foundation Contributors
All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
NSIS 3.10
Copyright (C) 1999-2025 Contributors THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS COMMON PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT.
1. DEFINITIONS
"Contribution" means:
a) in the case of the initial Contributor, the initial code and documentation distributed under this Agreement, and b) in the case of each subsequent Contributor:
i) changes to the Program, and
ii) additions to the Program;
where such changes and/or additions to the Program originate from and are distributed by that particular Contributor. A Contribution 'originates' from a Contributor if it was added to the Program by such Contributor itself or anyone acting on such Contributor's behalf. Contributions do not include additions to the Program which: (i) are separate modules of software distributed in conjunction with the Program under their own license agreement, and (ii) are not derivative works of the Program.
"Contributor" means any person or entity that distributes the Program.
"Licensed Patents " mean patent claims licensable by a Contributor which are necessarily infringed by the use or sale of its Contribution alone or when combined with the Program.
"Program" means the Contributions distributed in accordance with this Agreement.
"Recipient" means anyone who receives the Program under this Agreement, including all Contributors.
2. GRANT OF RIGHTS
a) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, distribute and sublicense the Contribution of such Contributor, if any, and such derivative works, in source code and object code form.
b) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free patent license under Licensed Patents to make, use, sell, offer to sell, import and otherwise transfer the Contribution of such Contributor, if any, in source code and object code form. This patent license shall apply to the combination of the Contribution and the Program if, at the time the Contribution is added by the Contributor, such addition of the Contribution causes such combination to be covered by the Licensed Patents. The patent license shall not apply to any other combinations which include the Contribution. No hardware per se is licensed hereunder.
c) Recipient understands that although each Contributor grants the licenses to its Contributions set forth herein, no assurances are provided by any Contributor that the Program does not infringe the patent or other intellectual property rights of any other entity. Each Contributor disclaims any liability to Recipient for claims brought by any other entity based on infringement of intellectual property rights or otherwise. As a condition to exercising the rights and licenses granted hereunder, each Recipient hereby assumes sole responsibility to secure any other intellectual property rights needed, if any. For example, if a third party patent license is required to allow Recipient to distribute the Program, it is Recipient's responsibility to acquire that license before distributing the Program.
d) Each Contributor represents that to its knowledge it has sufficient copyright rights in its Contribution, if any, to grant the copyright license set forth in this Agreement.
3. REQUIREMENTS
A Contributor may choose to distribute the Program in object code form under its own license agreement, provided that:
a) it complies with the terms and conditions of this Agreement; and
b) its license agreement:
i) effectively disclaims on behalf of all Contributors all warranties and conditions, express and implied, including warranties or conditions of title and non-infringement, and implied warranties or conditions of merchantability and fitness for a particular purpose;
ii) effectively excludes on behalf of all Contributors all liability for damages, including direct, indirect, special, incidental and consequential damages, such as lost profits;
iii) states that any provisions which differ from this Agreement are offered by that Contributor alone and not by any other party; and
iv) states that source code for the Program is available from such Contributor, and informs licensees how to obtain it in a reasonable manner on or through a medium customarily used for software exchange.
When the Program is made available in source code form:
a) it must be made available under this Agreement; and
b) a copy of this Agreement must be included with each copy of the Program.
Contributors may not remove or alter any copyright notices contained within the Program.
Each Contributor must identify itself as the originator of its Contribution, if any, in a manner that reasonably allows subsequent Recipients to identify the originator of the Contribution.
4. COMMERCIAL DISTRIBUTION
Commercial distributors of software may accept certain responsibilities with respect to end users, business partners and the like. While this license is intended to facilitate the commercial use of the Program, the Contributor who includes the Program in a commercial product offering should do so in a manner which does not create potential liability for other Contributors. Therefore, if a Contributor includes the Program in a commercial product offering, such Contributor ("Commercial Contributor") hereby agrees to defend and indemnify every other Contributor ("Indemnified Contributor") against any losses, damages and costs (collectively "Losses") arising from claims, lawsuits and other legal actions brought by a third party against the Indemnified Contributor to the extent caused by the acts or omissions of such Commercial Contributor in connection with its distribution of the Program in a commercial product offering. The obligations in this section do not apply to any claims or Losses relating to any actual or alleged intellectual property infringement. In order to qualify, an Indemnified Contributor must: a) promptly notify the Commercial Contributor in writing of such claim, and b) allow the Commercial Contributor to control, and cooperate with the Commercial Contributor in, the defense and any related settlement negotiations. The Indemnified Contributor may participate in any such claim at its own expense.
For example, a Contributor might include the Program in a commercial product offering, Product X. That Contributor is then a Commercial Contributor. If that Commercial Contributor then makes performance claims, or offers warranties related to Product X, those performance claims and warranties are such Commercial Contributor's responsibility alone. Under this section, the Commercial Contributor would have to defend claims against the other Contributors related to those performance claims and warranties, and if a court requires any other Contributor to pay any damages as a result, the Commercial Contributor must pay those damages.
5. NO WARRANTY
EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the appropriateness of using and distributing the Program and assumes all risks associated with its exercise of rights under this Agreement, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and unavailability or interruption of operations.
6. DISCLAIMER OF LIABILITY
EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
7. GENERAL
If any provision of this Agreement is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this Agreement, and without further action by the parties hereto, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
If Recipient institutes patent litigation against a Contributor with respect to a patent applicable to software (including a cross-claim or counterclaim in a lawsuit), then any patent licenses granted by that Contributor to such Recipient under this Agreement shall terminate as of the date such litigation is filed. In addition, if Recipient institutes patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Program itself (excluding combinations of the Program with other software or hardware) infringes such Recipient's patent(s), then such Recipient's rights granted under Section 2(b) shall terminate as of the date such litigation is filed.
All Recipient's rights under this Agreement shall terminate if it fails to comply with any of the material terms or conditions of this Agreement and does not cure such failure in a reasonable period of time after becoming aware of such noncompliance. If all Recipient's rights under this Agreement terminate, Recipient agrees to cease use and distribution of the Program as soon as reasonably practicable. However, Recipient's obligations under this Agreement and any licenses granted by Recipient relating to the Program shall continue and survive.
Everyone is permitted to copy and distribute copies of this Agreement, but in order to avoid inconsistency the Agreement is copyrighted and may only be modified in the following manner. The Agreement Steward reserves the right to publish new versions (including revisions) of this Agreement from time to time. No one other than the Agreement Steward has the right to modify this Agreement. IBM is the initial Agreement Steward. IBM may assign the responsibility to serve as the Agreement Steward to a suitable separate entity. Each new version of the Agreement will be given a distinguishing version number. The Program (including Contributions) may always be distributed subject to the version of the Agreement under which it was received. In addition, after a new version of the Agreement is published, Contributor may elect to distribute the Program (including its Contributions) under the new version. Except as expressly stated in Sections 2(a) and 2(b) above, Recipient receives no rights or licenses to the intellectual property of any Contributor under this Agreement, whether expressly, by implication, estoppel or otherwise. All rights in the Program not expressly granted under this Agreement are reserved.
This Agreement is governed by the laws of the State of New York and the intellectual property laws of the United States of America. No party to this Agreement will bring a legal action under this Agreement more than one year after the cause of action arose. Each party waives its rights to a jury trial in any resulting litigation.