Class: Shuwar::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/shuwar/tokenizer.rb

Overview

The tokenizer

Defined Under Namespace

Classes: CloseParen, OpenParen, Quote, Token

Instance Method Summary collapse

Constructor Details

#initialize(f) ⇒ Tokenizer

Make a new tokenizer that reads input from f

f should respond to each_line



12
13
14
15
 
# File 'lib/shuwar/tokenizer.rb', line 12

def initialize(f)
  @input = f
  @stack = []
end

Instance Method Details

#alter_stack(to, &block) ⇒ Object

Change stack to the specified contents by yielding an open paren and a symbol when push is needed and yielding a close paren when pop is needed.

Make sure you call alter_stack [], block before you go



79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/shuwar/tokenizer.rb', line 79

def alter_stack(to, &block)
  stack_tail, to_tail = @stack.dup, to.dup

  begin
    while stack_tail.fetch(0) == to_tail.fetch(0)
      stack_tail.shift
      to_tail.shift
    end
  rescue
    # No more items in one tail. End it.
  end

  stack_tail.size.times do
    block.call CloseParen.new
    @stack.pop
  end

  to_tail.each do |t|
    @stack.push t
    block.call OpenParen.new
    block.call t.to_sym
  end
end

#each_input_line(&block) ⇒ Object

Just a helper to chomp the line before each_lineing



19
20
21
 
# File 'lib/shuwar/tokenizer.rb', line 19

def each_input_line(&block)
  @input.each_line {|l| block.call l.chomp }
end

#each_token(&block) ⇒ Object

Iterates through each line and yield tokens. You should call this



25
26
27
28
29
30
31
 
# File 'lib/shuwar/tokenizer.rb', line 25

def each_token(&block)
  each_input_line do |line|
    each_token_in_line line, &block
  end

  alter_stack [], &block
end

#each_token_in_line(line, &block) ⇒ Object

Tokenize one line



35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/shuwar/tokenizer.rb', line 35

def each_token_in_line(line, &block)
  sp = line.split "|", 2
  if sp.size != 2
    alter_stack [], &block
    for c in %w{[ ] #}
      line.replace line.gsub c, " #{c} "
    end
    tags = line.split
    tags.each &:strip!
    tags.reject! &:empty?
    tags.each do |t|
      case
        when t == "["
          block.call OpenParen.new
        when t == "]"
          block.call CloseParen.new
        when t == "#"
          block.call Quote.new
        when t == "nil"
          block.call nil
        when t.chars.all? {|c| ('1'..'9') === c }
          block.call t.to_i
        when t.chars.all? {|c| ('1'..'9') === c or c == '.' }
          block.call t.to_f
        else
          block.call t.to_sym
      end
    end
  else
    ts, txt = sp
    tags = ts.split ":"
    tags.each &:strip!
    tags.reject! &:empty?
    alter_stack tags, &block
    block.call txt.strip
  end
end