Token (parser)
|
|
In computing, a token is a primitive block of a structured text. A token can look like anything: English, gibberish symbols, anything; It just needs to be a useful part of the structured text. Generally, whitespace is ignored, but sometimes it is important and tokenized.
Consider a text describing a calculation: "46 - number_of(cows)". The tokens here might be: "46", "-", "number_of", "(", "cows", and ")". Whitespace was ignored, but could have been included as well.
A token doesn't need to be valid, in order to be recognized as a token. "cows" may be nonsense to the language, "number_of" may be nonsense. But they are tokens none the less, in this example.
Tokens are frequently defined by regular expressions. A lexical analyzer such as lex reads a text, and outputs the tokens. This is called "tokenizing." If the lexer finds an invalid token, it will report an error.
Usually, tokenizing is just the first step of interpreting some text. Following tokenizing is parsing. From there, the interpreted data may be loaded into data structures, for general use, interpretation, or compiling.