Class: Metrocot

Inherits:

Object

• Object
• Metrocot

Defined in:

lib/metrocot.rb

Overview

Purpose

This class implements the main Metrocot HTML scanner and a number of handy input scanners (for grabbing time, numbers, or text from HTML). The purpose of the Metrocot is to scan a XML dom for the patterns specified in the Metrocot pattern language.

Pattern Language

The Metrocot pattern language allows for the following types of patterns:

...

matches anything

+“some string”

matches that string

+/(some|pattern)/ matches that regexp pattern

./HPRICOT_PATH

matches a certain type of dom subtree

SPACE

matches zero or more white spaces

+(PATTERN_A PATTERN_B)

matches PATTERN_A followed by PATTERN_B

PATTERN+

matches one or more occurrences of PATTERN

Usage

0) create a Metricot and define the types of fields you want to extract (and their names). 1) use Hpricot to get the doc’s dom 2) use descend(xpath) to create a NodeScraper rooted at the Hpricot node(s) matching the xpath 3) use collect(pattern) to collect all entries found in the HTML which match the Metricot pattern

Defined Under Namespace

Modules: Scanners Classes: AnythingPattern, BasePattern, CompositePattern, MatchRange, NodeScraper, OneOrMorePattern, OptSpacePattern, PathPattern, TextPattern

Constant Summary

VERSION =

'1.0.3'

Instance Attribute Summary

• #verbose ⇒ Object

  Returns the value of attribute verbose.

Instance Method Summary

• #compile_pattern(pattern_s, node_scraper) ⇒ Object
• #initialize(scanners) ⇒ Metrocot constructor

  A new instance of Metrocot.

• #log(s) ⇒ Object
• #scanner_by_name(name) ⇒ Object

  given a Symbol of a scanner (name), return a handle for that scanner declared when the metrocot was created.

• #scrape(doc) ⇒ Object

Constructor Details

#initialize(scanners) ⇒ Metrocot

Returns a new instance of Metrocot.

# File 'lib/metrocot.rb', line 1154

def initialize( scanners )

	@scanners = {}
	@compiled_patterns = {}
	
	scanners.each { |name, value|
		if value.is_a? Class
			@scanners[name] = value.new
		else
			@scanners[name] = value
		end
	}

	@verbose = false

	log("scanners: #{@scanners.inspect}")

end

Instance Attribute Details

#verbose ⇒ Object

Returns the value of attribute verbose.

# File 'lib/metrocot.rb', line 1151

def verbose
  @verbose
end

Instance Method Details

#compile_pattern(pattern_s, node_scraper) ⇒ Object

# File 'lib/metrocot.rb', line 1181

def compile_pattern( pattern_s, node_scraper )	

#		if @compiled_patterns.key? pattern_s 
#			return @compiled_patterns[ pattern_s ] 
#		end

	s = pattern_s
	patterns = []

	log("compiling: #{s}")
	
	while !s.empty? 

		log("left: #{s}")

		if s.index(")") == 0
			break
		end

		name = nil
		s.scan( /^(\w+)=/ ) { |match| 
			log "got name #{match.class} (#{match.size})"
			name = match[0]
		}
		s = s[(name.length + 1) .. -1] if name

		log("after name #{name}: #{s}")

		pattern = nil

		[PathPattern, TextPattern, AnythingPattern, OptSpacePattern].each { |pattern_class|
			pattern = pattern_class.parse(s)
			if pattern 
				pattern.metrocot = self
				pattern.node_scraper = node_scraper
				break 
			end
			log "not a #{pattern_class}"
		}

		if pattern 
			s = s[pattern.source.size .. -1]
			patterns << pattern
			log("found: #{pattern.description}")
			if name
				log("scanned as: #{name}")
				pattern.name = name.to_sym
			end
			next
		end
	
		if s[0..0] == "+" 
			raise "+ must follow pattern" unless patterns.size > 0
			raise "+ applied twice does not make sense" if patterns[-1].is_a? OneOrMorePattern
			pattern = OneOrMorePattern.new( patterns[-1] ) 
			pattern.metrocot = self
			pattern.node_scraper = node_scraper
			patterns[-1] = pattern
			log("now one or more: #{pattern.repeatee}")
			s = s[1 .. -1]
			next
		end

		if s[0..0] == "(" 
			pattern = compile_pattern( s[1 .. -1], node_scraper )
			close_par_index = pattern.source.size + 1
			raise "expected ')' found '#{s[0..0]}'" unless s[close_par_index..close_par_index] == ")"
			s = s[close_par_index + 1 .. -1]
			log("found nested: #{pattern.description} \"#{pattern.source}\"")
			patterns << pattern
			if name
				pattern.name = name.to_sym
			end
			next
		end

		raise "unrecognizable pattern: \"#{s[0..10]}...\""
			
	end
	
	pattern = if patterns.size > 1
		CompositePattern.new( patterns )
	elsif patterns.size == 1
		patterns[0]
	else 
		nil
	end

	if pattern
		pattern.metrocot = self
		pattern.node_scraper = node_scraper
		pattern.source = pattern_s[0 .. (0 - (1 + s.size))] 
	end

	return @compiled_patterns[ pattern_s ] = pattern
	
end

#log(s) ⇒ Object

# File 'lib/metrocot.rb', line 1137

def log( s )
	puts( s ) if @verbose
end

#scanner_by_name(name) ⇒ Object

given a Symbol of a scanner (name), return a handle for that scanner declared when the metrocot was created.

# File 'lib/metrocot.rb', line 1145

def scanner_by_name( name )
	@scanners[name]
end

#scrape(doc) ⇒ Object

# File 'lib/metrocot.rb', line 1176

def scrape(doc) 
	NodeScraper.new( self, nil, nil, doc )
end