Class: WIKK::WebBrowser

Inherits:
Object
  • Object
show all
Defined in:
lib/wikk_webbrowser.rb

Overview

WIKK WebBrowser class under MIT Lic. github.com/wikarekare. Wrapper around ruby’s http classes

WIKK_WebBrowser.new.https_session(host: 'www.blah.com') do |session|
  response = get_page(query: ,'/')
end

Defined Under Namespace

Classes: Error

Constant Summary collapse

VERSION = 
'0.9.4'

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false) ⇒ WIKK_WebBrowser

Create a WIKK_WebBrowser instance

Parameters:

  • host (String) —

    the host we want to connect to

  • port (Fixnum) (defaults to: nil) —

    Optional http server port

  • use_ssl (Boolean) (defaults to: false) —

    Use https, if true

  • verify_cert (Boolean) (defaults to: true) —

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)



48
49
50
51
52
53
54
55
56
 
# File 'lib/wikk_webbrowser.rb', line 48

def initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false)
  @host = host  #Need to do this, as passing nil is different to passing nothing to initialize!
  @cookies = cookies == nil ? {} : cookies
  @debug = debug
  @use_ssl = use_ssl
  @port = port != nil ? port : ( use_ssl ? 443 : 80 )
  @verify_cert = verify_cert
  @response = nil
end

Instance Attribute Details

#cookies ⇒ Object

Returns the value of attribute cookies.



30
31
32
 
# File 'lib/wikk_webbrowser.rb', line 30

def cookies
  @cookies
end

#debug ⇒ Object

Returns the value of attribute debug.



33
34
35
 
# File 'lib/wikk_webbrowser.rb', line 33

def debug
  @debug
end

#host ⇒ Object (readonly)

Returns the value of attribute host.



28
29
30
 
# File 'lib/wikk_webbrowser.rb', line 28

def host
  @host
end

#page ⇒ Object (readonly)

Returns the value of attribute page.



31
32
33
 
# File 'lib/wikk_webbrowser.rb', line 31

def page
  @page
end

#port ⇒ Object

Returns the value of attribute port.



35
36
37
 
# File 'lib/wikk_webbrowser.rb', line 35

def port
  @port
end

#referer ⇒ Object

Returns the value of attribute referer.



32
33
34
 
# File 'lib/wikk_webbrowser.rb', line 32

def referer
  @referer
end

#response ⇒ Object

Returns the value of attribute response.



37
38
39
 
# File 'lib/wikk_webbrowser.rb', line 37

def response
  @response
end

#session ⇒ Object

Returns the value of attribute session.



29
30
31
 
# File 'lib/wikk_webbrowser.rb', line 29

def session
  @session
end

#use_ssl ⇒ Object

Returns the value of attribute use_ssl.



36
37
38
 
# File 'lib/wikk_webbrowser.rb', line 36

def use_ssl
  @use_ssl
end

#verify_cert ⇒ Object

Returns the value of attribute verify_cert.



34
35
36
 
# File 'lib/wikk_webbrowser.rb', line 34

def verify_cert
  @verify_cert
end

Class Method Details

.http_session(host:, port: nil, debug: false, cookies: {}) {|the| ... } ⇒ Object

Create a WIKK_WebBrowser instance, connect to the host via http, and yield the WIKK_WebBrowser instance. Automatically closes the http session on returning from the block passed to it.

Parameters:

  • host (String) —

    the host we want to connect to

  • port (Fixnum) (defaults to: nil) —

    (80) the port the remote web server is running on

  • block (Proc)

Yield Parameters:

  • the (WIKK_WebBrowser) —

    session descriptor for further calls.



69
70
71
72
73
74
 
# File 'lib/wikk_webbrowser.rb', line 69

def self.http_session(host:, port: nil, debug: false, cookies: {})
  wb = self.new(host: host, port: port, debug: debug, use_ssl: false, cookies: cookies)
  wb.http_session do
    yield wb
  end
end

.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false) {|the| ... } ⇒ Object

Create a WIKK_WebBrowser instance, connect to the host via https, and yield the WIKK_WebBrowser instance.

Automatically closes the http session on returning from the block passed to it.

Parameters:

  • host (String) —

    the host we want to connect to

  • port (Fixnum) (defaults to: nil) —

    (443) the port the remote web server is running on

  • verify_cert (Boolean) (defaults to: true) —

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)

  • block (Proc)

Yield Parameters:

  • the (WIKK_WebBrowser) —

    session descriptor for further calls.



83
84
85
86
87
88
 
# File 'lib/wikk_webbrowser.rb', line 83

def self.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false)
  wb = self.new(host: host, port: port, cookies: cookies, use_ssl: true, verify_cert: verify_cert, debug: debug)
  wb.http_session do
    yield wb
  end
end

Instance Method Details

#add_cookies(cookies) ⇒ Object

Add additional cookies

Parameters:

  • cookies (Hash) —

    cookie_name => cookie_value



125
126
127
 
# File 'lib/wikk_webbrowser.rb', line 125

def add_cookies(cookies)
  cookies.each { |cookie_name, cookie_value| @cookies[cookie_name] = cookie_value }
end

#basic_authorization(user:, password:) ⇒ String

Web basic authentication (not exactly secure)

Parameters:

  • user (String) —

    Account name

  • password (String) —

    Accounts password

Returns:

  • (String) —

    Base64 encoded concatentation of user + ‘:’ + password



111
112
113
114
 
# File 'lib/wikk_webbrowser.rb', line 111

def basic_authorization(user:, password:)
  #req.basic_auth( user, password) if  user != nil
  'Basic ' + Base64.encode64( "#{user}:#{password}" )
end

#bearer_authorization(token:) ⇒ String

Dropbox style token authentication

Parameters:

  • token (String) —

    Token, as issued by dropbox

Returns:

  • (String) —

    Concatenation of ‘Bearer ’ + token



119
120
121
 
# File 'lib/wikk_webbrowser.rb', line 119

def bearer_authorization(token:)
  "Bearer " + token
end

#cookies_to_s ⇒ Object

Convert @cookies to ; separated strings

Returns:

  • cookies string



143
144
145
 
# File 'lib/wikk_webbrowser.rb', line 143

def cookies_to_s
  @cookies.to_a.map { |v| v.join('=') }.join('; ')
end

#delete_req(query:, authorization: nil, extra_headers: {}, extra_cookies: {}) ⇒ String

send a DELETE query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
 
# File 'lib/wikk_webbrowser.rb', line 278

def delete_req(query:, authorization: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? "https" : "http"}://#{@host}/#{query.gsub(/^\//,'')}")
  req = Net::HTTP::Delete.new(query)

  header = { 'HOST' => @host }
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k,v| 
    header[k] = v
  end
  req.initialize_http_header( header )

  begin
    @response = @session.request(req)
    save_cookies(@response)
  
    if(@response.code.to_i >= 300)
      raise "#{url} : #{@response.code} #{@response.message}"
    end
    return @response.body
  rescue StandardError => e
    puts "#{e}"
    return nil
  end
end

#extract_input_fields(body) ⇒ Hash

Extract form field values from the html body.

Parameters:

  • body (String) —

    The html response body

Returns:

  • (Hash) —

    Keys are the field names, values are the field values



369
370
371
372
373
374
375
376
 
# File 'lib/wikk_webbrowser.rb', line 369

def extract_input_fields(body)
  entry = true
  @inputs = {}
  doc = Nokogiri::HTML(body)
  doc.xpath("//form/input").each do |f|
    @inputs[f.get_attribute('name')] = f.get_attribute('value')
  end
end

#extract_link_fields(body) ⇒ Hash

Extract links from the html body.

Parameters:

  • body (String) —

    The html response body

Returns:

  • (Hash) —

    Keys are the link text, values are the html links



381
382
383
384
385
386
387
388
389
 
# File 'lib/wikk_webbrowser.rb', line 381

def extract_link_fields(body)
  entry = true
  @inputs = {}
  doc = Nokogiri::HTML(body)
  doc.xpath("//a").each do |f|
    return URI.parse( f.get_attribute('href') ).path if(f.get_attribute('name') == 'URL$1')
  end
  return nil
end

#form_values_to_s(form_values = nil, has_q = false) ⇒ String

Take a hash of the params to the post and generate a safe URL string.

Parameters:

  • form_values (Hash) (defaults to: nil) —

    Keys are the field names, values are the field values

  • has_q (Boolean) (defaults to: false) —

    We have a leading ? for the html get, so don’t need to add one.

Returns:

  • (String) —

    The ‘safe’ text for fields the get or post query to the web server



395
396
397
398
399
400
401
402
403
404
405
 
# File 'lib/wikk_webbrowser.rb', line 395

def form_values_to_s(form_values=nil, has_q = false)
  return "" if form_values == nil || form_values.length == 0
  s = (has_q == true ? "" : "?")
  first = true
  form_values.each do |key,value|
    s += "&" if !first
    s += "#{CGI::escape(key.to_s)}=#{CGI::escape(value.to_s)}"
    first = false
  end
  return s
end

#get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {}) ⇒ String

send a GET query to the web server using an http get, and returns the response.

Cookies in the response get preserved in @cookies, so they will be sent along with subsequent calls
We are currently ignoring redirects from the PDU's we are querying.

Parameters:

  • query (String) —

    The URL after the host/ bit and not usually not including parameters, if form_values are passed in

  • form_values (Hash{String=>Object-with-to_s}) (defaults to: nil) —

    The parameter passed to the web server eg. ?key1=value1&key2=value2…

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
 
# File 'lib/wikk_webbrowser.rb', line 164

def get_page(query: ,form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {})
  $stderr.puts "Debugging On" if @debug
  query += form_values_to_s(form_values, query.index('?') != nil) #Should be using req.set_form_data, but it seems to by stripping the leading / and then the query fails.
  url = URI.parse("#{@use_ssl ? "https" : "http"}://#{@host}/#{query.gsub(/^\//,'')}")
  $stderr.puts url if @debug

  req = Net::HTTP::Get.new(url.request_uri)  

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = 'application/x-www-form-urlencoded'   
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = "1"
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k,v| 
    header[k] = v
  end

  req.initialize_http_header( header )

  @response = @session.request(req)      
  save_cookies(@response)

  $stderr.puts @response.code.to_i if @debug

  if(@response.code.to_i >= 300)
    if(@response.code.to_i == 302)
        #ignore the redirects.
        #$stderr.puts "302"
        #@response.each {|key, val| $stderr.printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
        #$stderr.puts "Redirect to #{@response['location']}"   #Location seems to have cgi params removed. End up with .../cginame?&
        #$stderr.puts
      return
    elsif @response.code.to_i >= 400 && @response.code.to_i < 500
      return @response.body
    end
    raise Error.new(web_return_code: @response.code.to_i, message: "#{@response.code} #{@response.message} #{query} #{form_values} #{@response.body}")
  end

  return @response.body
end

#header_value(key:) ⇒ String

Get a header value, from the last response

Parameters:

  • key (String) —

    header key

Returns:

  • (String) —

    header value, for the given key.



151
152
153
 
# File 'lib/wikk_webbrowser.rb', line 151

def header_value(key:)
  @response.header[key]
end

#http_session ⇒ Object

Creating a session for http connection

attached block would then call get or post NET::HTTP calls

Parameters:

  • port (Fixnum) —

    Optional http server port

  • use_ssl (Boolean) —

    Use https, if true

  • verify_cert (Boolean) —

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)

  • block (Proc) 


96
97
98
99
100
101
102
103
104
105
 
# File 'lib/wikk_webbrowser.rb', line 96

def http_session
  @http = Net::HTTP.new(@host, @port)   
  @http.set_debug_output($stdout) if @debug
  @http.use_ssl = @use_ssl      
  @http.verify_mode = OpenSSL::SSL::VERIFY_NONE  if ! @use_ssl || ! @verify_cert 
  @http.start do |session| #ensure we close the session after the block
    @session = session 
    yield 
  end
end

#post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String

send a POST query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: 'application/x-www-form-urlencoded') —

    Posted content type

  • data (String) (defaults to: nil) —

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
 
# File 'lib/wikk_webbrowser.rb', line 220

def post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? "https" : "http"}://#{@host}/#{query}")
  req = Net::HTTP::Post.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = "1"
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k,v| 
    header[k] = v
  end
  req.initialize_http_header( header )

  if data != nil
    if data.class == Hash
      if content_type =~ /application\/octet-stream/
        req.set_form_data(data, '&') 
      else
        req.set_form_data.to_j
      end
    else
      req.body = data #If json as a string or raw string
    end
  else
    req.body = ''
  end
  
  @response = @session.request(req)
  save_cookies(@response)

  if(@response.code.to_i >= 300)
    if(@response.code.to_i == 302)
        #ignore the redirects. 
        #puts "302"
        #@response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
        #puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end
    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end
  
  return @response.body
end

#put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String

send a PUT query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: '"application/octet-stream"') —

    Posted content type

  • data (String) (defaults to: nil) —

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
 
# File 'lib/wikk_webbrowser.rb', line 314

def put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? "https" : "http"}://#{@host}/#{query}")
  req = Net::HTTP::Put.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = "1"
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k,v| 
    header[k] = v
  end
  req.initialize_http_header( header )

  if data != nil
    if data.class == Hash
      if content_type =~ /application\/octet-stream/
        req.set_form_data(data, '&') 
      else
        req.set_form_data.to_j
      end
    else
      req.body = data #If json as a string or raw string
    end
  else
    req.body = ''
  end
  
  @response = @session.request(req)
  save_cookies(@response)

  if(@response.code.to_i >= 300)
    if(@response.code.to_i == 302)
        #ignore the redirects. 
        #puts "302"
        #@response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
        #puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end
    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end
  
  return @response.body
end

#save_cookies(response) ⇒ Object

Save cookies returned by last html get/post. Removes previous cookies.

Parameters:

  • response (Net::HTTPResponse) —

    result from HTTP calls



132
133
134
135
136
137
138
139
 
# File 'lib/wikk_webbrowser.rb', line 132

def save_cookies(response)
  if(cookie_lines = response.get_fields('set-cookie')) != nil
    cookie_lines.each do | cookie_line |
      cookies = cookie_line.split('; ').map { |v| v.split('=')}
      cookies.each { |c| @cookies[c[0]] = c[1] }
    end
  end
end