Class: UnicodeMultibyte::Multibyte::Handlers::UTF8Handler

Inherits:
Object
  • Object
show all
Defined in:
lib/unicodechars/multibyte/handlers/utf8_handler.rb

Overview

UTF8Handler implements Unicode aware operations for strings, these operations will be used by the Chars proxy when $MULTIBYTE_CODE is set to ‘UTF8’.

Direct Known Subclasses

UTF8HandlerProc

Constant Summary collapse

HANGUL_SBASE =

Hangul character boundaries and properties

0xAC00
HANGUL_LBASE = 
0x1100
HANGUL_VBASE = 
0x1161
HANGUL_TBASE = 
0x11A7
HANGUL_LCOUNT = 
19
HANGUL_VCOUNT = 
21
HANGUL_TCOUNT = 
28
HANGUL_NCOUNT = 
HANGUL_VCOUNT * HANGUL_TCOUNT
HANGUL_SCOUNT = 
11172
HANGUL_SLAST = 
HANGUL_SBASE + HANGUL_SCOUNT
HANGUL_JAMO_FIRST = 
0x1100
HANGUL_JAMO_LAST = 
0x11FF
UNICODE_WHITESPACE =

All the unicode whitespace

[
  (0x0009..0x000D).to_a,  # White_Space # Cc   [5] <control-0009>..<control-000D>
  0x0020,          # White_Space # Zs       SPACE
  0x0085,          # White_Space # Cc       <control-0085>
  0x00A0,          # White_Space # Zs       NO-BREAK SPACE
  0x1680,          # White_Space # Zs       OGHAM SPACE MARK
  0x180E,          # White_Space # Zs       MONGOLIAN VOWEL SEPARATOR
  (0x2000..0x200A).to_a, # White_Space # Zs  [11] EN QUAD..HAIR SPACE
  0x2028,          # White_Space # Zl       LINE SEPARATOR
  0x2029,          # White_Space # Zp       PARAGRAPH SEPARATOR
  0x202F,          # White_Space # Zs       NARROW NO-BREAK SPACE
  0x205F,          # White_Space # Zs       MEDIUM MATHEMATICAL SPACE
  0x3000,          # White_Space # Zs       IDEOGRAPHIC SPACE
].flatten.freeze
UNICODE_LEADERS_AND_TRAILERS =

BOM (byte order mark) can also be seen as whitespace, it’s a non-rendering character used to distinguish between little and big endian. This is not an issue in utf-8, so it must be ignored.

UNICODE_WHITESPACE + [65279]
UTF8_PAT =

Borrowed from the Kconv library by Shinji KONO - (also as seen on the W3C site)

/\A(?:
 [\x00-\x7f]                                     |
 [\xc2-\xdf] [\x80-\xbf]                         |
 \xe0        [\xa0-\xbf] [\x80-\xbf]             |
 [\xe1-\xef] [\x80-\xbf] [\x80-\xbf]             |
 \xf0        [\x90-\xbf] [\x80-\xbf] [\x80-\xbf] |
 [\xf1-\xf3] [\x80-\xbf] [\x80-\xbf] [\x80-\xbf] |
 \xf4        [\x80-\x8f] [\x80-\xbf] [\x80-\xbf]
)*\z/xn
UNICODE_TRAILERS_PAT = 
/(#{codepoints_to_pattern(UNICODE_LEADERS_AND_TRAILERS)})+\Z/
UNICODE_LEADERS_PAT = 
/\A(#{codepoints_to_pattern(UNICODE_LEADERS_AND_TRAILERS)})+/

Class Method Summary collapse

Class Method Details

.capitalize(str) ⇒ Object

Returns a copy of str with the first character converted to uppercase and the remainder to lowercase



180
181
182
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 180

def capitalize(str)
  upcase(slice(str, 0..0)) + downcase(slice(str, 1..-1) || '')
end

.codepoints_to_pattern(array_of_codepoints) ⇒ Object

Returns a regular expression pattern that matches the passed Unicode codepoints



103
104
105
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 103

def self.codepoints_to_pattern(array_of_codepoints) #:nodoc:
  array_of_codepoints.collect{ |e| [e].pack 'U*' }.join('|') 
end

.compose(str) ⇒ Object

Perform composition on the characters in the string



216
217
218
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 216

def compose(str)
  compose_codepoints u_unpack(str).pack('U*')
end

.consumes?(str) ⇒ Boolean

Checks if the string is valid UTF8.

Returns:

  • (Boolean) 


246
247
248
249
250
251
252
253
254
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 246

def consumes?(str)
  # Unpack is a little bit faster than regular expressions
  begin
    str.unpack('U*')
    true
  rescue ArgumentError
    false
  end
end

.decompose(str) ⇒ Object

Perform decomposition on the characters in the string



211
212
213
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 211

def decompose(str)
  decompose_codepoints(:canonical, u_unpack(str)).pack('U*')
end

.downcase(str) ⇒ Object

Convert characters in the string to lowercase



177
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 177

def downcase(str); to_case :lowercase_mapping, str; end

.g_length(str) ⇒ Object

Returns the number of grapheme clusters in the string. This method is very likely to be moved or renamed in future versions.



258
259
260
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 258

def g_length(str)
  g_unpack(str).length
end

.index(str, *args) ⇒ Object

Returns the position of the passed argument in the string, counting in codepoints



126
127
128
129
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 126

def index(str, *args)
  bidx = str.index(*args)
  bidx ? (u_unpack(str.slice(0...bidx)).size) : nil
end

.insert(str, offset, fragment) ⇒ Object

Inserts the passed string at specified codepoint offsets



116
117
118
119
120
121
122
123
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 116

def insert(str, offset, fragment)
  str.replace(
    u_unpack(str).insert(
      offset,
      u_unpack(fragment)
    ).flatten.pack('U*')
  )
end

.lstrip(str) ⇒ Object

Does Unicode-aware lstrip



137
138
139
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 137

def lstrip(str)
  str.gsub(UNICODE_LEADERS_PAT, '')
end

.normalize(str, form = UnicodeMultibyte::Multibyte::DEFAULT_NORMALIZATION_FORM) ⇒ Object

Returns the KC normalization of the string by default. NFKC is considered the best normalization form for passing strings to databases and validations.

  • str: The string to perform normalization on.

  • form: The form you want to normalize in. Should be one of the following: :c, :kc, :d or :kd.



193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 193

def normalize(str, form=UnicodeMultibyte::Multibyte::DEFAULT_NORMALIZATION_FORM)
  # See http://www.unicode.org/reports/tr15, Table 1
  codepoints = u_unpack(str)
  case form
    when :d
      reorder_characters(decompose_codepoints(:canonical, codepoints))
    when :c
      compose_codepoints reorder_characters(decompose_codepoints(:canonical, codepoints))
    when :kd
      reorder_characters(decompose_codepoints(:compatability, codepoints))
    when :kc
      compose_codepoints reorder_characters(decompose_codepoints(:compatability, codepoints))
    else
      raise ArgumentError, "#{form} is not a valid normalization variant", caller
  end.pack('U*')
end

.reverse(str) ⇒ Object

Reverses codepoints in the string.



153
154
155
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 153

def reverse(str)
  u_unpack(str).reverse.pack('U*')
end

.rstrip(str) ⇒ Object

Does Unicode-aware rstrip



132
133
134
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 132

def rstrip(str)
  str.gsub(UNICODE_TRAILERS_PAT, '')
end

.size(str) ⇒ Object Also known as: length

Returns the number of codepoints in the string



147
148
149
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 147

def size(str)
  u_unpack(str).size
end

.slice(str, *args) ⇒ Object Also known as: []

Implements Unicode-aware slice with codepoints. Slicing on one point returns the codepoints for that character.



159
160
161
162
163
164
165
166
167
168
169
170
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 159

def slice(str, *args)
  if (args.size == 2 && args.first.is_a?(Range))
    raise TypeError, 'cannot convert Range into Integer' # Do as if we were native
  elsif args[0].kind_of? Range
    cps = u_unpack(str).slice(*args)
    cps.nil? ? nil : cps.pack('U*')
  elsif args.size == 1 && args[0].kind_of?(Numeric)
    u_unpack(str)[args[0]]
  else
    u_unpack(str).slice(*args).pack('U*')
  end
end

.strip(str) ⇒ Object

Removed leading and trailing whitespace



142
143
144
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 142

def strip(str)
  str.gsub(UNICODE_LEADERS_PAT, '').gsub(UNICODE_TRAILERS_PAT, '')
end

.tidy_bytes(str) ⇒ Object

Replaces all the non-utf-8 bytes by their iso-8859-1 or cp1252 equivalent resulting in a valid utf-8 string



263
264
265
266
267
268
269
270
271
272
273
274
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 263

def tidy_bytes(str)
  str.split(//u).map do |c|
    if !UTF8_PAT.match(c)
      n = c.unpack('C')[0]
      n < 128 ? n.chr :
      n < 160 ? [UCD.cp1252[n] || n].pack('U') :
      n < 192 ? "\xC2" + n.chr : "\xC3" + (n-64).chr
    else
      c
    end
  end.join
end

.translate_offset(str, byte_offset) ⇒ Object

Used to translate an offset from bytes to characters, for instance one received from a regular expression match



225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 225

def translate_offset(str, byte_offset)
  return 0 if str == ''
  return nil if byte_offset.nil?
  chunk = str[0..byte_offset]
  begin
    begin
      chunk.unpack('U*').length - 1
    rescue ArgumentError => e
      chunk = str[0..(byte_offset+=1)]
      # Stop retrying at the end of the string
      raise e unless byte_offset < chunk.length 
      # We damaged a character, retry
      retry
    end
  # Catch the ArgumentError so we can throw our own
  rescue ArgumentError 
    raise EncodingError.new('malformed UTF-8 character')
  end
end

.upcase(str) ⇒ Object

Convert characters in the string to uppercase



174
 
# File 'lib/unicodechars/multibyte/handlers/utf8_handler.rb', line 174

def upcase(str); to_case :uppercase_mapping, str; end