Class: Twitter::Regex

Inherits:

Object

• Object
• Twitter::Regex

Defined in:

lib/twitter-text/regex.rb

Overview

A collection of regular expressions for parsing Tweet text. The regular expression list is frozen at load time to ensure immutability. These regular expressions are used throughout the Twitter classes. Special care has been taken to make sure these reular expressions work with Tweets in all languages.

Constant Summary

REGEXEN =

:nodoc:

{}

TLDS =

YAML.load_file(
  File.join(
    File.expand_path('../../..', __FILE__), # project root
    'lib', 'assets', 'tld_lib.yml'
  )
)

UNICODE_SPACES =

Space is more than %20, U+3000 for example is the full-width space used with Kanji. Provide a short-hand to access both the list of characters and a pattern suitible for use with String#split

Taken from: ActiveSupport::Multibyte::Handlers::UTF8Handler::UNICODE_WHITESPACE

[
      (0x0009..0x000D).to_a,  # White_Space # Cc   [5] <control-0009>..<control-000D>
      0x0020,          # White_Space # Zs       SPACE
      0x0085,          # White_Space # Cc       <control-0085>
      0x00A0,          # White_Space # Zs       NO-BREAK SPACE
      0x1680,          # White_Space # Zs       OGHAM SPACE MARK
      0x180E,          # White_Space # Zs       MONGOLIAN VOWEL SEPARATOR
      (0x2000..0x200A).to_a, # White_Space # Zs  [11] EN QUAD..HAIR SPACE
      0x2028,          # White_Space # Zl       LINE SEPARATOR
      0x2029,          # White_Space # Zp       PARAGRAPH SEPARATOR
      0x202F,          # White_Space # Zs       NARROW NO-BREAK SPACE
      0x205F,          # White_Space # Zs       MEDIUM MATHEMATICAL SPACE
      0x3000,          # White_Space # Zs       IDEOGRAPHIC SPACE
].flatten.map{|c| [c].pack('U*')}.freeze

INVALID_CHARACTERS =

Character not allowed in Tweets

[
  0xFFFE, 0xFEFF, # BOM
  0xFFFF,         # Special
  0x202A, 0x202B, 0x202C, 0x202D, 0x202E # Directional change
].map{|cp| [cp].pack('U') }.freeze

LATIN_ACCENTS =

Latin accented characters Excludes 0xd7 from the range (the multiplication sign, confusable with “x”). Also excludes 0xf7, the division sign

[
      regex_range(0xc0, 0xd6),
      regex_range(0xd8, 0xf6),
      regex_range(0xf8, 0xff),
      regex_range(0x0100, 0x024f),
      regex_range(0x0253, 0x0254),
      regex_range(0x0256, 0x0257),
      regex_range(0x0259),
      regex_range(0x025b),
      regex_range(0x0263),
      regex_range(0x0268),
      regex_range(0x026f),
      regex_range(0x0272),
      regex_range(0x0289),
      regex_range(0x028b),
      regex_range(0x02bb),
      regex_range(0x0300, 0x036f),
      regex_range(0x1e00, 0x1eff)
].join('').freeze

RTL_CHARACTERS =

[
  regex_range(0x0600,0x06FF),
  regex_range(0x0750,0x077F),
  regex_range(0x0590,0x05FF),
  regex_range(0xFE70,0xFEFF)
].join('').freeze

PUNCTUATION_CHARS =

'!"#$%&\'()*+,-./:;<=>?@\[\]^_\`{|}~'

SPACE_CHARS =

" \t\n\x0B\f\r"

CTRL_CHARS =

"\x00-\x1F\x7F"

HASHTAG_ALPHA =

A hashtag must contain at least one unicode letter or mark, as well as numbers, underscores, and select special characters.

/[\p{L}\p{M}]/

HASHTAG_ALPHANUMERIC =

/[\p{L}\p{M}\p{Nd}_\u200c\u200d\u0482\ua673\ua67e\u05be\u05f3\u05f4\u309b\u309c\u30a0\u30fb\u3003\u0f0b\u0f0c\u00b7]/

HASHTAG_BOUNDARY =

/\A|\z|[^&\p{L}\p{M}\p{Nd}_\u200c\u200d\u0482\ua673\ua67e\u05be\u05f3\u05f4\u309b\u309c\u30a0\u30fb\u3003\u0f0b\u0f0c\u00b7]/

HASHTAG =

/(#{HASHTAG_BOUNDARY})(#|＃)(#{HASHTAG_ALPHANUMERIC}*#{HASHTAG_ALPHA}#{HASHTAG_ALPHANUMERIC}*)/io

DOMAIN_VALID_CHARS =

"[^#{PUNCTUATION_CHARS}#{SPACE_CHARS}#{CTRL_CHARS}#{INVALID_CHARACTERS.join('')}#{UNICODE_SPACES.join('')}]"

Class Method Summary

• .[](key) ⇒ Object

  Return the regular expression for a given key.

• .regex_range(from, to = nil) ⇒ Object

  :nodoc:.

Class Method Details

.[](key) ⇒ Object

Return the regular expression for a given key. If the key is not a known symbol a nil will be returned.

# File 'lib/twitter-text/regex.rb', line 329

def self.[](key)
  REGEXEN[key]
end

.regex_range(from, to = nil) ⇒ Object

:nodoc:

# File 'lib/twitter-text/regex.rb', line 13

def self.regex_range(from, to = nil) # :nodoc:
  if $RUBY_1_9
    if to
      "\\u{#{from.to_s(16).rjust(4, '0')}}-\\u{#{to.to_s(16).rjust(4, '0')}}"
    else
      "\\u{#{from.to_s(16).rjust(4, '0')}}"
    end
  else
    if to
      [from].pack('U') + '-' + [to].pack('U')
    else
      [from].pack('U')
    end
  end
end