Class: GoogleBuilder

Inherits:
Object
  • Object
show all
Defined in:
lib/scrapedo/google_builder.rb

Constant Summary collapse

CONFIG_PATH = 
Pathname.new(__dir__ || ".").join("config/google_builder").expand_path
FILE_MAP = 
{
  time_period: "time_period.yml",
  hl: "host_language.yml",
  lr: "language_restrict.yml",
  gl: "geo_location.yml",
  cr: "country_restrict.yml",
  google_domain: "domain.yml"
}.freeze

Instance Method Summary collapse

Constructor Details

#initialize(token, all_params: false) ⇒ GoogleBuilder

Initializes a new instance of GoogleBuilder

Parameters:

  • token (String)
  • all_params (Boolean) (defaults to: false) —

    If true, all available parameters will be included, otherwise only common parameters will be included.



22
23
24
25
26
27
 
# File 'lib/scrapedo/google_builder.rb', line 22

def initialize(token, all_params: false)
  @has_next = false
  @data = { token: token }
  @params_cache = {}
  @all_params = all_params
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args, &block) ⇒ Object 



254
255
256
257
258
259
260
261
262
263
264
 
# File 'lib/scrapedo/google_builder.rb', line 254

def method_missing(name, *args, &block)
  source_type, config_data = find_config_source(name)
  if source_type
    define_singleton_method(name) do
      @data[source_type] = config_data[name]
      self
    end
    return send(name)
  end
  super
end

Instance Method Details

#cr(country) ⇒ Object Also known as: country_restrict

Country Restrict applies strict country filtering. Only results originating from the specified country are returned. Supports 240+ countries. See Country Restrict Parameter.

This can also use GoogleBuilder#cr_country. For example:

  • GoogleBuilder#cr("countryUS") is same as GoogleBuilder#cr_us and GoogleBuilder#cr_united_states.
  • GoogleBuilder#cr("countryAL") is same as GoogleBuilder#cr_al and GoogleBuilder#cr_albania. (For this one, all_params must be true)

GoogleBuilder#country_restrict is an alias for GoogleBuilder#cr.

Parameters:

  • country (String) 


182
183
184
185
 
# File 'lib/scrapedo/google_builder.rb', line 182

def cr(country)
  @data[:cr] = country
  self
end

#device(device) ⇒ Object

Device type for SERP layout. See General Parameters. Accepted values: desktop, mobile, default is desktop.

This can also use GoogleBuilder##device. For example:

  • GoogleBuilder#device("desktop") is same as GoogleBuilder#desktop.
  • GoogleBuilder#device("mobile") is same as GoogleBuilder#mobile.

Parameters:

  • device (String) 


56
57
58
59
 
# File 'lib/scrapedo/google_builder.rb', line 56

def device(device)
  @data[:device] = device if %w[desktop mobile].include?(device)
  self
end

#disable_filter ⇒ Object

Disables "Similar Results" and "Omitted Results" filters.



201
202
203
204
 
# File 'lib/scrapedo/google_builder.rb', line 201

def disable_filter
  @data[:filter] = false
  self
end

#domain(domain) ⇒ Object Also known as: google_domain

Google domain to query. Prefixes [https://], [http://], and [www.] are automatically stripped. Default is google.com. Supports 84 regional domains. See Supported Google Domains.

This can also use GoogleBuilder#domain_#location. For example:

  • GoogleBuilder#domain("google.com") is same as GoogleBuilder#domain_united_states.

GoogleBuilder#google_domain is an alias for GoogleBuilder#domain.

Parameters:

  • domain (String) 


125
126
127
128
 
# File 'lib/scrapedo/google_builder.rb', line 125

def domain(domain)
  @data[:google_domain] = domain
  self
end

#enable_nfpr ⇒ Object

Enables Google's automatic spelling correction.



195
196
197
198
 
# File 'lib/scrapedo/google_builder.rb', line 195

def enable_nfpr
  @data[:nfpr] = true
  self
end

#gl(location) ⇒ Object Also known as: geo_location

Geo Location (datacenter country). Determines from which country's perspective results are ranked and returned. Default is us. Supports 240+ countries. ISO 3166-1 alpha-2 codes. See Country Parameter.

This can also use GoogleBuilder#gl_#location. For example:

  • GoogleBuilder#gl("us") is same as GoogleBuilder#gl_us and GoogleBuilder#gl_united_states.
  • GoogleBuilder#gl("cl") is same as GoogleBuilder#gl_cl and GoogleBuilder#gl_chile. (For this one, all_params must be true)

GoogleBuilder#geo_location is an alias for GoogleBuilder#gl.

Parameters:

  • location (String) 


110
111
112
113
 
# File 'lib/scrapedo/google_builder.rb', line 110

def gl(location)
  @data[:gl] = location
  self
end

#hl(language) ⇒ Object Also known as: host_language

Host language of the Google UI. Default is en. Supports 150+ languages. ISO 639-1 codes. See Language Parameter.

This can also use GoogleBuilder#hl_language. For example:

  • GoogleBuilder#hl("en") is same as GoogleBuilder#hl_en and GoogleBuilder#hl_english.
  • GoogleBuilder#hl("ach") is same as GoogleBuilder#hl_ach and GoogleBuilder#hl_luo. (For this one, all_params must be true)

GoogleBuilder#host_language is an alias for GoogleBuilder#hl.

Parameters:

  • language (String) 


94
95
96
97
 
# File 'lib/scrapedo/google_builder.rb', line 94

def hl(language)
  @data[:hl] = language
  self
end

#include_html ⇒ Object

Result includes the raw Google HTML. Useful for debugging and custom parsing. See General Parameters.



79
80
81
82
 
# File 'lib/scrapedo/google_builder.rb', line 79

def include_html
  @data[:include_html] = true
  self
end

#inspect ⇒ Object 



270
271
272
273
274
275
276
 
# File 'lib/scrapedo/google_builder.rb', line 270

def inspect
  vars = instance_variables.map do |var|
    value = instance_variable_get(var)
    "#{var}=#{value.inspect}" unless var == :@params_cache
  end.join(", ")
  "#<#{self.class}:0x#{object_id.to_s(16)} #{vars}>"
end

#location(location) ⇒ Object

Location name in Google's canonical format. Automatically encoded to UULE internally. See Location.

Format: City,State/Region,Country

Examples:

[Istanbul,Istanbul,Turkey]
[New York,New York,United States]

Parameters:

  • location (String) 


141
142
143
144
 
# File 'lib/scrapedo/google_builder.rb', line 141

def location(location)
  @data[:location] = location
  self
end

#lr(language) ⇒ Object Also known as: language_restrict

Language Restrict applies strict language filtering. Only results written in the specified language are returned. Supports 35 languages. See Language Restrict Parameter.

This can also use GoogleBuilder#lr_language. For example:

  • GoogleBuilder#lr("en") is same as GoogleBuilder#lr_en and GoogleBuilder#lr_english.

GoogleBuilder#language_restrict is an alias for GoogleBuilder#lr.

Parameters:

  • language (String) 


166
167
168
169
 
# File 'lib/scrapedo/google_builder.rb', line 166

def lr(language)
  @data[:lr] = language
  self
end

#next ⇒ JSON

Gets next page if result has next page.

Returns:

  • (JSON) —

    The next page of results.



225
226
227
 
# File 'lib/scrapedo/google_builder.rb', line 225

def next
  start(@data[:start] + 10) if next?
end

#next? ⇒ Boolean

Returns true if result have more pages; otherwise false.

Returns:

  • (Boolean) 


30
31
32
 
# File 'lib/scrapedo/google_builder.rb', line 30

def next?
  @has_next
end

#params(*params) ⇒ Object

Set any parameter from Google Search API.

Parameters:

  • params (Hash) —

    A hash of parameters to set.



239
240
241
242
243
244
245
 
# File 'lib/scrapedo/google_builder.rb', line 239

def params(*params)
  params.first.each do |key, value|
    key_sym = key.to_sym
    @data[key_sym] = value if key_sym != :token && key_sym != :start && !value.nil?
  end
  self
end

#respond_to_missing?(name, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


266
267
268
 
# File 'lib/scrapedo/google_builder.rb', line 266

def respond_to_missing?(name, include_private = false)
  find_config_source(name)&.present? || super
end

#safe_search ⇒ Object

Send active to filter adult content from results.



189
190
191
192
 
# File 'lib/scrapedo/google_builder.rb', line 189

def safe_search
  @data[:safe] = "active"
  self
end

#scrapedo_url ⇒ Object

Returns the URL to be used for the request.



248
249
250
251
252
 
# File 'lib/scrapedo/google_builder.rb', line 248

def scrapedo_url
  url = URI("https://api.scrape.do/plugin/google/search")
  url.query = URI.encode_www_form(@data)
  url
end

#search(query) ⇒ Object

Note:

This is required!

Search query. See Required Parameters.

Parameters:

  • query (String)

Raises:

  • ArgumentError if query is nil or empty.



39
40
41
42
43
44
45
46
 
# File 'lib/scrapedo/google_builder.rb', line 39

def search(query)
  raise ArgumentError, "Query is required" if blank? query

  @data.select! { |key| key == :token }
  @has_next = false
  @data[:q] = query
  self
end

#start(start = 0) ⇒ JSON

Start scraping from a specific offset. See General Parameters.

Parameters:

  • start (Integer) (defaults to: 0) —

    The offset to start from. Default is 0.

Returns:

  • (JSON) —

    The search results starting from the specified offset.

Raises:

  • ArgumentError If the search query is not set.



212
213
214
215
216
217
218
219
220
 
# File 'lib/scrapedo/google_builder.rb', line 212

def start(start = 0)
  raise ArgumentError, "Query is required" if blank? @data[:q]

  @data[:start] = start.positive? ? start / 10 * 10 : 0
  response = Net::HTTP.get(scrapedo_url)
  result = JSON.parse(response)
  @has_next = !result["pagination"]["next"].nil?
  result
end

#time(time_period) ⇒ Object

Time Period limits results to a specific recency window. See Time-Based Filtering. Accepted values: last_hour, last_day, last_week, last_month, last_year

This can also use GoogleBuilder#time_period. For example:

  • GoogleBuilder#time("last_hour") is same as GoogleBuilder#hour and GoogleBuilder#last_hour.
  • GoogleBuilder#time("last_day") is same as GoogleBuilder#day and GoogleBuilder#last_day and GoogleBuilder#today.
  • GoogleBuilder#time("last_week") is same as GoogleBuilder#week and GoogleBuilder#last_week.
  • GoogleBuilder#time("last_month") is same as GoogleBuilder#month and GoogleBuilder#last_month.
  • GoogleBuilder#time("last_year") is same as GoogleBuilder#year and GoogleBuilder#last_year.

Parameters:

  • time_period (String) 


72
73
74
75
 
# File 'lib/scrapedo/google_builder.rb', line 72

def time(time_period)
  @data[:time_period] = time_period
  self
end

#uule(uule) ⇒ Object

Google UULE-encoded location string. Auto-generated from location when not provided. If both location and uule are sent, uule takes priority. location is sufficient for most use cases. See uule.

Parameters:

  • uule (String) 


152
153
154
155
 
# File 'lib/scrapedo/google_builder.rb', line 152

def uule(uule)
  @data[:uule] = uule
  self
end