Establishing a Connection
Connecting to Apache Kafka
Set BootstrapServers and the Topic properties to specify the address of your Apache Kafka server, as well as the topic you would like to interact with.
By default, the provider communicates with the data source in PLAINTEXT, which means that all data is sent in the clear. To encrypt communication, you should configure the provider to use SSL encryption. To do this, set UseSSL to true and configure SSLServerCert and SSLServerCertType to load the server certificates. PEM files and the Windows certificate store are supported certificate stores.
Note that proxy settings like ProxyServer and firewall settings like FirewallServer do not affect the connection to the Apache Kafka broker. Internally the provider connects to Apache Kafka using the official libraries which do not support proxies. These options are only used when the provider connects to the schema registry as described in Extracting Metadata From Topics.
Authenticating to Apache Kafka
The Apache Kafka data source supports the following authentication methods:
- Anonymous
- Plain
- Scram
- Kerberos
Anonymous
Certain on-premise deployments of Apache Kafka are able to connect to Apache Kafka without setting any authentication connection properties. To do so, simply set the AuthScheme to "None", and you are ready to connect.
SASL Plain
The User and Password properties should be specified. AuthScheme should be set to Plain.
SCRAM login module
The User and Password properties should be specified. The AuthScheme should be set to 'SCRAM' (for SCRAM-SHA-256) or 'SCRAM-SHA-512'.
SSL client certificates
The SSLClientCert and SSLClientCertType properties should be specified and AuthScheme should be set to SSLCertificate. The PEM certificate format is recommended but both PEM and PFX are supported.
Kerberos
To authenticate to Apache Kafka using Kerberos, set the following properties:- AuthScheme: Set this to KERBEROS.
- KerberosServiceName: This should match to the principal name of the Kafka brokers. For example, the principal is "kafka/[email protected]", so: KerberosServiceName=kafka.