Specifies if the Apache Kafka consumer should automatically commit read offsets.

Data Type

bool

Default Value

false

Remarks

By default, the provider does not commit read offsets unless you invoke CommitOffset. If an offset is committed for a topic, the provider starts reading from that offset for future queries (until the next CommitOffset). Otherwise, each query starts reading at the position defined by OffsetResetStrategy.

When AutoCommit is enabled, the provider commits offsets periodically and also at the end of each SELECT query. This means that each SELECT query "consumes" the messages it reads from Kafka, preventing later SELECT queries from returning those messages. You must provide a ConsumerGroupId to enable AutoCommit, as the committed offset is shared across the consumer group, allowing later connections with the same group ID to use the same offsets.

Consider the following when enabling AutoCommit:

• Queries with OFFSET: Some queries may consume more messages than they return. For example, a SELECT query with an OFFSET clause first consumes messages up to the specified offset and then returns rows after that point. Messages consumed to satisfy the OFFSET are discarded and cannot be read again within the same consumer group.
• TypeDetectionScheme Options: The None and RowScan options in TypeDetectionScheme perform internal reads on the topic to identify available columns. This can lead to unpredictable results because the Kafka library may auto-commit after these reads if the AutoCommit timer expires. To prevent this, use SchemaRegistry or MessageOnly modes in TypeDetectionScheme to avoid these reads and ensure consistent offsets.
• Message-Based Offsets: Auto-commit operates in terms of Kafka messages, not rows. Some SerializationFormat options can read multiple rows from a single Kafka message, such as when a Kafka message contains a JSON array. If a SELECT query reads only a subset of rows from that message, Kafka treats the entire message as consumed, meaning any unread rows within that message are discarded.