Class: ANTLR3::TokenScheme

TokenSchemes exist to handle the problem of defining token types as integer values while maintaining meaningful text names for the types. They are dynamically defined modules that map integer values to constants with token-type names.

Fundamentally, tokens exist to take a chunk of text and identify it as belonging to some category, like "VARIABLE" or "INTEGER". In code, the category is represented by an integer -- some arbitrary value that ANTLR will decide to use as it is creating the recognizer. The purpose of using an integer (instead of say, a ruby symbol) is that ANTLR's decision logic often needs to test whether a token's type falls within a range, which is not possible with symbols.

The downside of token types being represented as integers is that a developer needs to be able to reference the unknown type value by name in action code. Furthermore, code that references the type by name and tokens that can be inspected with names in place of type values are more meaningful to a developer.

Since ANTLR requires token type names to follow capital-letter naming conventions, defining types as named constants of the recognizer class resolves the problem of referencing type values by name. Thus, a token type like "VARIABLE'' can be represented by a number like 5 and referenced within code by VARIABLE. However, when a recognizer creates tokens, the name of the token's type cannot be seen without using the data defined in the recognizer.

Of course, tokens could be defined with a name attribute that could be specified when tokens are created. However, doing so would make tokens take up more space than necessary, as well as making it difficult to change the type of a token while maintaining a correct name value.

TokenSchemes exist as a technique to manage token type referencing and name extraction. They:

1. keep token type references clear and understandable in recognizer code

2. permit access to a token's type-name independently of recognizer objects

3. allow multiple classes to share the same token information

Building Token Schemes

TokenScheme is a subclass of Module. Thus, it has the method TokenScheme.new(tk_class = nil) { ... module-level code ...}, which will evaluate the block in the context of the scheme (module), similarly to Module#module_eval. Before evaluating the block, .new will setup the module with the following actions:

1. define a customized token class (more on that below)

2. add a new constant, TOKEN_NAMES, which is a hash that maps types to names

3. dynamically populate the new scheme module with a couple instance methods

4. include ANTLR3::Constants in the new scheme module

As TokenScheme the class functions as a metaclass, figuring out some of the scoping behavior can be mildly confusing if you're trying to get a handle of the entity for your own purposes. Remember that all of the instance methods of TokenScheme function as module-level methods of TokenScheme instances, ala attr_accessor and friends.

TokenScheme#define_token(name_symbol, int_value) adds a constant definition name_symbol with the value int_value. It is essentially like Module#const_set, except it forbids constant overwriting (which would mess up recognizer code fairly badly) and adds an inverse type-to-name map to its own TOKEN_NAMES table. TokenScheme#define_tokens is a convenience method for defining many types with a hash pairing names to values.

TokenScheme#register_name(value, name_string) specifies a custom type-to-name definition. This is particularly useful for the anonymous tokens that ANTLR generates for literal strings in the grammar specification. For example, if you refer to the literal '=' in some parser rule in your grammar, ANTLR will add a lexer rule for the literal and give the token a name like T__x, where x is the type's integer value. Since this is pretty meaningless to a developer, generated code should add a special name definition for type value x with the string "'='".

Sample TokenScheme Construction

TokenData = ANTLR3::TokenScheme.new do
  define_tokens(
    :INT  => 4,
    :ID   => 6,
    :T__5 => 5,
    :WS   => 7
  )

  # note the self:: scoping below is due to the fact that
  # ruby lexically-scopes constant names instead of
  # looking up in the current scope
  register_name(self::T__5, "'='")
end

TokenData::ID           # => 6
TokenData::T__5         # => 5
TokenData.token_name(4) # => 'INT'
TokenData.token_name(5) # => "'='"

class ARecognizerOrSuch < ANTLR3::Parser
  include TokenData
  ID   # => 6
end

Custom Token Classes and Relationship with Tokens

When a TokenScheme is created, it will define a subclass of ANTLR3::CommonToken and assigned it to the constant name Token. This token class will both include and extend the scheme module. Since token schemes define the private instance method token_name(type), instances of the token class are now able to provide their type names. The Token method name uses the token_name method to provide the type name as if it were a simple attribute without storing the name itself.

When a TokenScheme is included in a recognizer class, the class will now have the token types as named constants, a type-to-name map constant TOKEN_NAMES, and a grammar-specific subclass of ANTLR3::CommonToken assigned to the constant Token. Thus, when recognizers need to manufacture tokens, instead of using the generic CommonToken class, they can create tokens using the customized Token class provided by the token scheme.

If you need to use a token class other than CommonToken, you can pass the class as a parameter to TokenScheme.new, which will be used in place of the dynamically-created CommonToken subclass.