noun
token
A unit of text treated as a single item in language analysis or computer processing.
The program splits the sentence into tokens.
Etymology
The old word means a sign or identifying mark. Language analysis and computing specialize it to an individual unit treated as a meaningful item.
More examples
- Each token in the file is processed separately.
- A punctuation mark can count as a token.
- The analyzer labels every token with its grammatical role.
Forms
token · tokens
Similar words
- word
- A token may instead be punctuation or part of a word.