Class: Sass::Script::Lexer
- Inherits:
-
Object
- Object
- Sass::Script::Lexer
- Includes:
- Sass::SCSS::RX
- Defined in:
- lib/sass/script/lexer.rb
Overview
The lexical analyzer for SassScript. It takes a raw string and converts it to individual tokens that are easier to parse.
Direct Known Subclasses
Defined Under Namespace
Classes: Token
Constant Summary collapse
- OPERATORS =
A hash from operator strings to the corresponding token types.
{ '+' => :plus, '-' => :minus, '*' => :times, '/' => :div, '%' => :mod, '=' => :single_eq, ':' => :colon, '(' => :lparen, ')' => :rparen, ',' => :comma, 'and' => :and, 'or' => :or, 'not' => :not, '==' => :eq, '!=' => :neq, '>=' => :gte, '<=' => :lte, '>' => :gt, '<' => :lt, '#{' => :begin_interpolation, '}' => :end_interpolation, ';' => :semicolon, '{' => :lcurly, }
- OPERATORS_REVERSE =
Sass::Util.map_hash(OPERATORS) {|k, v| [v, k]}
- TOKEN_NAMES =
Sass::Util.map_hash(OPERATORS_REVERSE) {|k, v| [k, v.inspect]}.merge({ :const => "variable (e.g. $foo)", :ident => "identifier (e.g. middle)", :bool => "boolean (e.g. true, false)", })
- OP_NAMES =
A list of operator strings ordered with longer names first so that
>
and<
don't clobber>=
and<=
. OPERATORS.keys.sort_by {|o| -o.size}
- IDENT_OP_NAMES =
A sub-list of OP_NAMES that only includes operators with identifier names.
OP_NAMES.select {|k, v| k =~ /^\w+/}
- REGULAR_EXPRESSIONS =
A hash of regular expressions that are used for tokenizing.
{ :whitespace => /\s+/, :comment => COMMENT, :single_line_comment => SINGLE_LINE_COMMENT, :variable => /(\$)(#{IDENT})/, :ident => /(#{IDENT})(\()?/, :number => /(-)?(?:(\d*\.\d+)|(\d+))([a-zA-Z%]+)?/, :color => HEXCOLOR, :bool => /(true|false)\b/, :ident_op => %r{(#{Regexp.union(*IDENT_OP_NAMES.map{|s| Regexp.new(Regexp.escape(s) + "(?!#{NMCHAR}|\Z)")})})}, :op => %r{(#{Regexp.union(*OP_NAMES)})}, }
- STRING_REGULAR_EXPRESSIONS =
A hash of regular expressions that are used for tokenizing strings.
The key is a
[Symbol, Boolean]
pair. The symbol represents which style of quotation to use, while the boolean represents whether or not the string is following an interpolated segment. { [:double, false] => string_re('"', '"'), [:single, false] => string_re("'", "'"), [:double, true] => string_re('', '"'), [:single, true] => string_re('', "'"), [:uri, false] => /url\(#{W}(#{URLCHAR}*?)(#{W}\)|#\{)/, [:uri, true] => /(#{URLCHAR}*?)(#{W}\)|#\{)/, }
Constants included from Sass::SCSS::RX
Sass::SCSS::RX::CDC, Sass::SCSS::RX::CDO, Sass::SCSS::RX::COMMENT, Sass::SCSS::RX::DASHMATCH, Sass::SCSS::RX::DEFAULT, Sass::SCSS::RX::ESCAPE, Sass::SCSS::RX::FUNCTION, Sass::SCSS::RX::GREATER, Sass::SCSS::RX::H, Sass::SCSS::RX::HASH, Sass::SCSS::RX::HEXCOLOR, Sass::SCSS::RX::IDENT, Sass::SCSS::RX::IDENT_HYPHEN_INTERP, Sass::SCSS::RX::IMPORTANT, Sass::SCSS::RX::INCLUDES, Sass::SCSS::RX::INTERP_START, Sass::SCSS::RX::MOZ_ANY, Sass::SCSS::RX::NAME, Sass::SCSS::RX::NL, Sass::SCSS::RX::NMCHAR, Sass::SCSS::RX::NMSTART, Sass::SCSS::RX::NONASCII, Sass::SCSS::RX::NOT, Sass::SCSS::RX::NUM, Sass::SCSS::RX::NUMBER, Sass::SCSS::RX::PLUS, Sass::SCSS::RX::PREFIXMATCH, Sass::SCSS::RX::RANGE, Sass::SCSS::RX::S, Sass::SCSS::RX::SINGLE_LINE_COMMENT, Sass::SCSS::RX::STATIC_SELECTOR, Sass::SCSS::RX::STATIC_VALUE, Sass::SCSS::RX::STRING, Sass::SCSS::RX::STRING1, Sass::SCSS::RX::STRING1_NOINTERP, Sass::SCSS::RX::STRING2, Sass::SCSS::RX::STRING2_NOINTERP, Sass::SCSS::RX::STRING_NOINTERP, Sass::SCSS::RX::SUBSTRINGMATCH, Sass::SCSS::RX::SUFFIXMATCH, Sass::SCSS::RX::TILDE, Sass::SCSS::RX::UNICODE, Sass::SCSS::RX::UNICODERANGE, Sass::SCSS::RX::URI, Sass::SCSS::RX::URL, Sass::SCSS::RX::URLCHAR, Sass::SCSS::RX::VARIABLE, Sass::SCSS::RX::W
Instance Attribute Summary collapse
-
#line ⇒ Fixnum
readonly
The line number of the lexer's current position.
-
#offset ⇒ Fixnum
readonly
The number of bytes into the current line of the lexer's current position.
Instance Method Summary collapse
-
#after_interpolation? ⇒ Boolean
Whether or not the last token lexed was
:end_interpolation
. -
#done? ⇒ Boolean
Whether or not there's more source text to lex.
-
#expected!(name)
Raise an error to the effect that
name
was expected in the input stream and wasn't found. -
#initialize(str, line, offset, options) ⇒ Lexer
constructor
A new instance of Lexer.
-
#next ⇒ Token
Moves the lexer forward one token.
-
#peek ⇒ Token
Returns the next token without moving the lexer forward.
-
#str { ... } ⇒ String
Records all non-comment text the lexer consumes within the block and returns it as a string.
-
#unpeek!
Rewinds the underlying StringScanner to before the token returned by #peek.
-
#whitespace?(tok = @tok) ⇒ Boolean
Returns whether or not there's whitespace before the next token.
Methods included from Sass::SCSS::RX
Constructor Details
#initialize(str, line, offset, options) ⇒ Lexer
Returns a new instance of Lexer.
128 129 130 131 132 133 134 135 |
# File 'lib/sass/script/lexer.rb', line 128
def initialize(str, line, offset, options)
@scanner = str.is_a?(StringScanner) ? str : StringScanner.new(str)
@line = line
@offset = offset
@options = options
@interpolation_stack = []
@prev = nil
end
|
Instance Attribute Details
#line ⇒ Fixnum (readonly)
The line number of the lexer's current position.
34 35 36 |
# File 'lib/sass/script/lexer.rb', line 34
def line
@line
end
|
#offset ⇒ Fixnum (readonly)
The number of bytes into the current line of the lexer's current position.
40 41 42 |
# File 'lib/sass/script/lexer.rb', line 40
def offset
@offset
end
|
Instance Method Details
#after_interpolation? ⇒ Boolean
Returns Whether or not the last token lexed was :end_interpolation
.
179 180 181 |
# File 'lib/sass/script/lexer.rb', line 179
def after_interpolation?
@prev && @prev.type == :end_interpolation
end
|
#done? ⇒ Boolean
Returns Whether or not there's more source text to lex.
173 174 175 176 |
# File 'lib/sass/script/lexer.rb', line 173
def done?
whitespace unless after_interpolation? && @interpolation_stack.last
@scanner.eos? && @tok.nil?
end
|
#expected!(name)
Raise an error to the effect that name
was expected in the input stream
and wasn't found.
This calls #unpeek! to rewind the scanner to immediately after the last returned token.
191 192 193 194 |
# File 'lib/sass/script/lexer.rb', line 191
def expected!(name)
unpeek!
Sass::SCSS::Parser.expected(@scanner, name, @line)
end
|
#next ⇒ Token
Moves the lexer forward one token.
140 141 142 143 144 145 |
# File 'lib/sass/script/lexer.rb', line 140
def next
@tok ||= read_token
@tok, tok = nil, @tok
@prev = tok
return tok
end
|
#peek ⇒ Token
Returns the next token without moving the lexer forward.
162 163 164 |
# File 'lib/sass/script/lexer.rb', line 162
def peek
@tok ||= read_token
end
|
#str { ... } ⇒ String
Records all non-comment text the lexer consumes within the block and returns it as a string.
201 202 203 204 205 206 |
# File 'lib/sass/script/lexer.rb', line 201
def str
old_pos = @tok ? @tok.pos : @scanner.pos
yield
new_pos = @tok ? @tok.pos : @scanner.pos
@scanner.string[old_pos...new_pos]
end
|
#unpeek!
Rewinds the underlying StringScanner to before the token returned by #peek.
168 169 170 |
# File 'lib/sass/script/lexer.rb', line 168
def unpeek!
@scanner.pos = @tok.pos if @tok
end
|
#whitespace?(tok = @tok) ⇒ Boolean
Returns whether or not there's whitespace before the next token.
150 151 152 153 154 155 156 157 |
# File 'lib/sass/script/lexer.rb', line 150
def whitespace?(tok = @tok)
if tok
@scanner.string[0...tok.pos] =~ /\s\Z/
else
@scanner.string[@scanner.pos, 1] =~ /^\s/ ||
@scanner.string[@scanner.pos - 1, 1] =~ /\s\Z/
end
end
|