Class: Rightscale::HttpConnection

Inherits:
Object
  • Object
show all
Defined in:
lib/helene/rightscale/right_http_connection.rb

Overview

HttpConnection maintains a persistent HTTP connection to a remote server. Each instance maintains its own unique connection to the HTTP server. HttpConnection makes a best effort to receive a proper HTTP response from the server, although it does not guarantee that this response contains a HTTP Success code.

On low-level errors (TCP/IP errors) HttpConnection invokes a reconnect and retry algorithm. Note that although each HttpConnection object has its own connection to the HTTP server, error handling is shared across all connections to a server. For example, if there are three connections to www.somehttpserver.com, a timeout error on one of those connections will cause all three connections to break and reconnect. A connection will not break and reconnect, however, unless a request becomes active on it within a certain amount of time after the error (as specified by HTTP_CONNECTION_RETRY_DELAY). An idle connection will not break even if other connections to the same server experience errors.

A HttpConnection will retry a request a certain number of times (as defined by HTTP_CONNNECTION_RETRY_COUNT). If all the retries fail, an exception is thrown and all HttpConnections associated with a server enter a probationary period defined by HTTP_CONNECTION_RETRY_DELAY. If the user makes a new request subsequent to entering probation, the request will fail immediately with the same exception thrown on probation entry. This is so that if the HTTP server has gone down, not every subsequent request must wait for a connect timeout before failing. After the probation period expires, the internal state of the HttpConnection is reset and subsequent requests have the full number of potential reconnects and retries available to them.

Defined Under Namespace

Modules: NullLogger

Constant Summary collapse

HTTP_CONNECTION_RETRY_COUNT =

up the 3

42
HTTP_CONNECTION_OPEN_TIMEOUT = 
5
HTTP_CONNECTION_READ_TIMEOUT = 
120
HTTP_CONNECTION_RETRY_DELAY = 
15
@@params =

class methods

{}

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(params = {}) ⇒ HttpConnection

Params hash:

:user_agent => 'www.HostName.com'    # String to report as HTTP User agent
:ca_file    => 'path_to_file'        # A path of a CA certification file in PEM format. The file can contain several CA certificates.
:logger     => Logger object         # If omitted, HttpConnection logs to STDOUT
:exception  => Exception to raise    # The type of exception to raise if a request repeatedly fails. RuntimeError is raised if this parameter is omitted.
:http_connection_retry_count         # by default == Rightscale::HttpConnection.params[:http_connection_retry_count]
:http_connection_open_timeout        # by default == Rightscale::HttpConnection.params[:http_connection_open_timeout]
:http_connection_read_timeout        # by default == Rightscale::HttpConnection.params[:http_connection_read_timeout]
:http_connection_retry_delay         # by default == Rightscale::HttpConnection.params[:http_connection_retry_delay]



156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 156

def initialize(params={})
  @thread = Thread.current
  @caller = caller
  @params = params
  @params[:http_connection_retry_count]  ||= @@params[:http_connection_retry_count]
  @params[:http_connection_open_timeout] ||= @@params[:http_connection_open_timeout]
  @params[:http_connection_read_timeout] ||= @@params[:http_connection_read_timeout]
  @params[:http_connection_retry_delay]  ||= @@params[:http_connection_retry_delay]
  @http   = nil
  @server = nil
  @logger = get_param(:logger)
  @logger ||= RAILS_DEFAULT_LOGGER if defined?(RAILS_DEFAULT_LOGGER)
  @logger ||= NullLogger
  @ca_file = get_param(:ca_file)
  @state = {}
  @eof = {}
end

Instance Attribute Details

#eof ⇒ Object

Returns the value of attribute eof.



144
145
146
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 144

def eof
  @eof
end

#http ⇒ Object

instance methods



138
139
140
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 138

def http
  @http
end

#logger ⇒ Object

Returns the value of attribute logger.



141
142
143
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 141

def logger
  @logger
end

#params ⇒ Object

see @@params



140
141
142
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 140

def params
  @params
end

#server ⇒ Object

Returns the value of attribute server.



139
140
141
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 139

def server
  @server
end

#state ⇒ Object

Returns the value of attribute state.



143
144
145
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 143

def state
  @state
end

#thread ⇒ Object

Returns the value of attribute thread.



142
143
144
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 142

def thread
  @thread
end

Class Method Details

.params ⇒ Object

Query the global (class-level) parameters:

:user_agent => 'www.HostName.com'    # String to report as HTTP User agent
:ca_file    => 'path_to_file'        # Path to a CA certification file in PEM format. The file can contain several CA certificates.  If this parameter isn't set, HTTPS certs won't be verified.
:logger     => Logger object         # If omitted, HttpConnection logs to STDOUT
:exception  => Exception to raise    # The type of exception to raise
                                     # if a request repeatedly fails. RuntimeError is raised if this parameter is omitted.
:http_connection_retry_count         # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_COUNT
:http_connection_open_timeout        # by default == Rightscale::HttpConnection::HTTP_CONNECTION_OPEN_TIMEOUT
:http_connection_read_timeout        # by default == Rightscale::HttpConnection::HTTP_CONNECTION_READ_TIMEOUT
:http_connection_retry_delay         # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_DELAY



120
121
122
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 120

def self.params
  @@params
end

.params=(params) ⇒ Object

Set the global (class-level) parameters



125
126
127
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 125

def self.params=(params)
  @@params = params
end

Instance Method Details

#finish(reason = '') ⇒ Object 



486
487
488
489
490
491
492
493
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 486

def finish(reason = '')
  # prevent_mt_use!
  if @http && @http.started?
    reason = ", reason: '#{reason}'" unless reason.blank?
    log(:info, "Closing #{@http.use_ssl? ? 'HTTPS' : 'HTTP'} connection to #{@http.address}:#{@http.port}#{reason}")
    @http.finish rescue nil  # it's possible for a Thread to reset this before we can finish it (so it wouldn't be started)
  end
end

#get_param(name) ⇒ Object 



194
195
196
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 194

def get_param(name)
  @params[name] || @@params[name]
end

#local_read_size=(newsize) ⇒ Object

Set the maximum size (in bytes) of a single read from local data sources like files. This can be used to tune the performance of, for example, a streaming PUT of a large buffer.



222
223
224
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 222

def local_read_size=(newsize)
  Net::HTTPGenericRequest.local_read_size=(newsize)
end

#local_read_size? ⇒ Boolean

Query for the maximum size (in bytes) of a single read from local data sources like files. This is important, for example, in a streaming PUT of a large buffer.

Returns:

  • (Boolean) 


215
216
217
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 215

def local_read_size?
  Net::HTTPGenericRequest.local_read_size?
end

#log(*args, &block) ⇒ Object 



174
175
176
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 174

def log(*args, &block)
  @logger.send(*args, &block) if @logger
end

#log_request(request_params) ⇒ Object 



457
458
459
460
461
462
463
464
465
466
467
468
469
470
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 457

def log_request(request_params)
  return if logger.nil?
  return unless logger.debug?
  logger.debug{ "HttpConnection - request_params=#{ request_params.inspect }" }
  if((request = request_params[:request]))
    method = request.method
    path = request.path
    #body = (request.body.size > 42 ? (request.body[0,42] + '...') : request.body) if request.body
    body = request.body
    logger.debug{ "HttpConnection - request.method=#{ method.inspect }" }
    logger.debug{ "HttpConnection - request.path=#{ path.inspect }" }
    logger.debug{ "HttpConnection - request.body=#{ body.inspect }" }
  end
end

#log_response(response) ⇒ Object 



472
473
474
475
476
477
478
479
480
481
482
483
484
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 472

def log_response(response)
  return if logger.nil?
  return unless logger.debug?
  if response
    code = response.code
    message = response.message
    #body = (response.body.size > 42 ? (response.body[0,42] + '...') : response.body) if response.body
    body = response.body
    logger.debug{ "HttpConnection - response.code=#{ code.inspect }" }
    logger.debug{ "HttpConnection - response.message=#{ message.inspect }" }
    logger.debug{ "HttpConnection - response.body=#{ body.inspect }" }
  end
end

#prevent_mt_use! ⇒ Object 



178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 178

def prevent_mt_use!
  unless Thread.current==@thread
    msg = <<-__
    @thread : #{ @thread.inspect }
    
    @caller : #{ @caller.inspect }
    
    thread  : #{ Thread.current.inspect }
    
    caller  : #{ caller.inspect }
    __
    
    raise ThreadError.new, msg
  end
end

#request(request_params, &block) ⇒ Object

Send HTTP request to server

 request_params hash:
 :server   => 'www.HostName.com'   # Hostname or IP address of HTTP server
 :port     => '80'                 # Port of HTTP server
 :protocol => 'https'              # http and https are supported on any port
 :request  => 'requeststring'      # Fully-formed HTTP request to make

Raises RuntimeError, Interrupt, and params[:exception] (if specified in new).



366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 366

def request(request_params, &block)
  # prevent_mt_use!
  log_request(request_params) rescue nil
  # We save the offset here so that if we need to retry, we can return the file pointer to its initial position
  mypos = get_fileptr_offset(request_params)
  loop do
    # if we are inside a delay between retries: no requests this time!
    if error_count > @params[:http_connection_retry_count] &&
       error_time + @params[:http_connection_retry_delay] > Time.now
      # store the message (otherwise it will be lost after error_reset and
      # we will raise an exception with an empty text)
      banana_message_text = banana_message
      log(:warn, "#{err_header} re-raising same error: #{banana_message_text} " +
                  "-- error count: #{error_count}, error age: #{Time.now.to_i - error_time.to_i}")
      exception = get_param(:exception) || RuntimeError
      raise exception.new(banana_message_text)
    end

    # try to connect server(if connection does not exist) and get response data
    begin
      request_params[:protocol] ||= (request_params[:port] == 443 ? 'https' : 'http')

      request = request_params[:request]
      request['User-Agent'] = get_param(:user_agent) || ''

      # (re)open connection to server if none exists or params has changed
      unless @http          &&
             @http.started? &&
             @server   == request_params[:server] &&
             @port     == request_params[:port]   &&
             @protocol == request_params[:protocol]
        start(request_params)
      end

      # Detect if the body is a streamable object like a file or socket.  If so, stream that
      # bad boy.
      setup_streaming(request)
      response = @http.request(request, &block)
      log_response(response) rescue nil

      error_reset
      eof_reset
      return response

    # We treat EOF errors and the timeout/network errors differently.  Both
    # are tracked in different statistics blocks.  Note below that EOF
    # errors will sleep for a certain (exponentially increasing) period.
    # Other errors don't sleep because there is already an inherent delay
    # in them; connect and read timeouts (for example) have already
    # 'slept'.  It is still not clear which way we should treat errors
    # like RST and resolution failures.  For now, there is no additional
    # delay for these errors although this may change in the future.

    # EOFError means the server closed the connection on us.
    rescue EOFError => e
      log(:debug, "#{err_header} server #{@server} closed connection")
      @http = nil

        # if we have waited long enough - raise an exception...
      if raise_on_eof_exception?
        exception = get_param(:exception) || RuntimeError
        log(:warn, "#{err_header} raising #{exception} due to permanent EOF being received from #{@server}, error age: #{Time.now.to_i - eof_time.to_i}")
        raise exception.new("Permanent EOF is being received from #{@server}.")
      else
          # ... else just sleep a bit before new retry
        sleep(add_eof)
        # We will be retrying the request, so reset the file pointer
        reset_fileptr_offset(request, mypos)
      end
    rescue Exception => e  # See comment at bottom for the list of errors seen...
      @http = nil
      # if ctrl+c is pressed - we have to reraise exception to terminate proggy
      if e.is_a?(Interrupt) && !( e.is_a?(Errno::ETIMEDOUT) || e.is_a?(Timeout::Error))
        log(:debug, "#{err_header} request to server #{@server} interrupted by ctrl-c")
        raise
      elsif e.is_a?(ArgumentError) && e.message.include?('wrong number of arguments (5 for 4)')
        # seems our net_fix patch was overriden...
        exception = get_param(:exception) || RuntimeError
        raise exception.new('incompatible Net::HTTP monkey-patch')
      end
      # oops - we got a banana: log it
      error_add(e.message)
      log(:warn, "#{err_header} request failure count: #{error_count}, exception: #{e.inspect}")

      # We will be retrying the request, so reset the file pointer
      reset_fileptr_offset(request, mypos)

    end
  end
end

#socket_read_size=(newsize) ⇒ Object

Set the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.



208
209
210
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 208

def socket_read_size=(newsize)
  Net::BufferedIO.socket_read_size=(newsize)
end

#socket_read_size? ⇒ Boolean

Query for the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.

Returns:

  • (Boolean) 


201
202
203
 
# File 'lib/helene/rightscale/right_http_connection.rb', line 201

def socket_read_size?
  Net::BufferedIO.socket_read_size?
end