Determines whether to hide key or value colums when the topic has no associated schema information.

Possible Values

Data Type

string

Default Value

"Legacy"

Remarks

In some situations the provider has no information on the type of data stored in the keys or values of a topic's messages. This can happen for several reasons:

• When TypeDetectionScheme=RowScan or TypeDetectionScheme=None, and none of the messages scanned have key/value content
• When TypeDetectionScheme=SchemaRegistry or TypeDetectionScheme=SchemaRegistryAggregate
  • If message keys are enabled (see MessageKeyType), and no key schema is available for a topic
  • If AllowKeyOnlyRegistryTopics=true, and no value schema is available for a topic
  • If either the key or value schema exisxts, but is in an unsupported format

This option determines whether the provider exposes fallback columns for parts of the message that have no schema information. In the following descriptions, value columns include both the Message column used for single values as well as columns flattened from the value. Similarly, key columns include both the top-level key column (named by MessageKeyColumn) and columns flattened from the key. Refer to Extracting Metadata From Topics for more information on column flattening.

By default the provider follows these legacy rules, which are compatible with versions of the provider before this option was introduced:

• For TypeDetectionScheme=RowScan and TypeDetectionScheme=None, no value columns are exposed if none of the scanned messages have values. The same applies for keys if they are enabled.
• For TypeDetectionScheme=SchemaRegistry or TypeDetectionScheme=SchemaRegistryAggregate, the behavior depends upon AllowKeyOnlyRegistryTopics. By default key-only topics are disabled and topics without value schemas are not exposed as tables. When key-only topics are enabled, topics without value schemas expose a fallback Message column that reports base64 values. In either case, topics without key schemas do not expose key columns.

The other options provide consistent behavior regardless of the TypeDetectionScheme:

• With HideUnusedColumns=Never, the provider pushes fallback columns for topic values, and keys if enabled. The fallback value column is called Message and the fallback key columns is named by MessageKeyColumn. Both fallback columns report base64 values.
• With HideUnusedColumns=Key, the provider does not push a fallback column for keys. It still pushes a fallback column for values. If key columns are not enabled, this option has the same behavior as Never.
• With HideUnusedColumns=Value, the provider does not push a fallback column for values. It still pushes a fallback column for keys if keys are enabled.
• With HideUnusedColumns=KeyAndMessage, the provider never pushes fallback columns.

Note that there are no cases where TypeDetectionScheme=MessageOnly and schema information is missing. In MessageOnly mode, the provider does not use any external information to determine the format of the topic's values. The provider always returns a single Message column that is decoded according to SerializationFormat. Key columns behave the same way but using MessageKeyType.

For the same reason, the provider does not consider a column unused if it uses a primitive value format. Setting SerializationFormat to Long, Integer, Float, Double, or String prevents the provider from accessing the broker or schema registry to determine value schemas. The same applies for MessageKeyType and key schemas.