Class: TinyGrabber

Inherits:
Object
  • Object
show all
Defined in:
lib/tiny_grabber.rb,
lib/tiny_grabber/version.rb

Overview

Main class for TinyGrabber

Constant Summary collapse

VERSION =

Version number

"0.0.7"

Class Method Summary collapse

Class Method Details

.get(url, params = {}) ⇒ Object

Get Net::HTTP object for content from remote single page

url (string)

Link on content

params (hash)

Addition setting

  • proxy (hash)

    Configuration of remote proxy server

    • ip address of remote proxy server

    • port of remote proxy server

    • proxy_type of remote proxy server

      • http for http(s) proxy servers (by default)

      • socks for socks4/5 proxy servers

  • headers (hash)

    Headers

  • auth (hash)

    Basic Authentication

    • username Authenticate username

    • password Authenticate password

  • post (hash)

    POST data

Example

TinyGrabber.get url, headers: { 'Content-Type' => 'application/json' }, proxy: { ip: 'xx.xx.xx.xx', port: xx, proxy_type: :socks }



33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/tiny_grabber.rb', line 33

def self.get url, params = {}
  uri = URI(URI.escape(url))

  params = convert_params_to_sym params

  # Use proxy for request
  if params[:proxy]
    if ['socks', :socks].include?(params[:proxy][:proxy_type])
      http = Net::HTTP.SOCKSProxy(params[:proxy][:ip], params[:proxy][:port])
    else
      http = Net::HTTP::Proxy(params[:proxy][:ip], params[:proxy][:port])
    end
  else
    http = Net::HTTP
  end

  # Set HTTP request type
  if params[:post] or params.key?(:post)
    request = Net::HTTP::Post.new(uri.request_uri)
    request.set_form_data(params[:post])
  else
    request = Net::HTTP::Get.new(uri.request_uri)
  end

  # Set Basic Auth
  if params[:auth]
    request.basic_auth params[:auth][:username], params[:auth][:password]
  end

  # Set headers
  params[:headers].each { |k, v| request.add_field(String(k), v) } if params[:headers]

  # Get response
  http.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') { |http| http.request(request) }
end