Class: TwitterCldr::Tokenizers::DateTokenizer
- Inherits:
-
Object
- Object
- TwitterCldr::Tokenizers::DateTokenizer
- Defined in:
- lib/twitter_cldr/tokenizers/calendars/date_tokenizer.rb
Instance Attribute Summary collapse
-
#data_reader ⇒ Object
readonly
Returns the value of attribute data_reader.
Class Method Summary collapse
Instance Method Summary collapse
-
#initialize(data_reader) ⇒ DateTokenizer
constructor
A new instance of DateTokenizer.
- #tokenize(pattern) ⇒ Object
- #tokenizer ⇒ Object
Constructor Details
#initialize(data_reader) ⇒ DateTokenizer
Returns a new instance of DateTokenizer.
22 23 24 |
# File 'lib/twitter_cldr/tokenizers/calendars/date_tokenizer.rb', line 22 def initialize(data_reader) @data_reader = data_reader end |
Instance Attribute Details
#data_reader ⇒ Object (readonly)
Returns the value of attribute data_reader.
20 21 22 |
# File 'lib/twitter_cldr/tokenizers/calendars/date_tokenizer.rb', line 20 def data_reader @data_reader end |
Class Method Details
.tokenizer ⇒ Object
11 12 13 14 15 16 17 |
# File 'lib/twitter_cldr/tokenizers/calendars/date_tokenizer.rb', line 11 def tokenizer @tokenizer ||= Tokenizer.new([ TokenRecognizer.new(:composite, /^\#\{[^\}]+\}/, /^\#\{([^\}]+)\}/), TokenRecognizer.new(:pattern, /^(?:G{1,5}|y+|Y+|Q{1,4}|q{1,5}|M{1,5}|L{1,5}|d{1,2}|F{1}|E{1,5}|e{1,5}|c{1,5}|w{1,2}|W{1})/), TokenRecognizer.new(:plaintext, //) ], /(\s*\'[\w\s-]+\'\s*|G{1,5}|y+|Y+|Q{1,4}|q{1,5}|M{1,5}|L{1,5}|d{1,2}|F{1}|E{1,5}|e{1,5}|c{1,5}|w{1,2}|W{1}|\#\{[^\}]+\})/) end |
Instance Method Details
#tokenize(pattern) ⇒ Object
26 27 28 |
# File 'lib/twitter_cldr/tokenizers/calendars/date_tokenizer.rb', line 26 def tokenize(pattern) PatternTokenizer.new(data_reader, tokenizer).tokenize(pattern) end |
#tokenizer ⇒ Object
30 31 32 |
# File 'lib/twitter_cldr/tokenizers/calendars/date_tokenizer.rb', line 30 def tokenizer self.class.tokenizer end |