Tokenize

Split input text into tokens using byte-pair encoding for Cohere models.

Columns

Name	Type	References
Tokens	String
TokenStrings	String
Meta	String
Text	String
Model	String