Method: UV::BufferedTokenizer#extract

Defined in:
lib/uv-rays/buffered_tokenizer.rb

#extract(data) ⇒ Object

Extract takes an arbitrary string of input data and returns an array of tokenized entities, provided there were any available to extract.

Examples:


tokenizer.extract(data).
    map { |entity| Decode(entity) }.each { ... }

Parameters:

  • data (String) 


54
55
56
57
58
59
 
# File 'lib/uv-rays/buffered_tokenizer.rb', line 54

def extract(data)
    data.force_encoding(@encoding)
    @input << data

    @extract_method.call
end