Class: Rightscale::HttpConnection
- Defined in:
- lib/right_http_connection.rb
Overview
HttpConnection maintains a persistent HTTP connection to a remote server. Each instance maintains its own unique connection to the HTTP server. HttpConnection makes a best effort to receive a proper HTTP response from the server, although it does not guarantee that this response contains a HTTP Success code.
On low-level errors (TCP/IP errors) HttpConnection invokes a reconnect and retry algorithm. Note that although each HttpConnection object has its own connection to the HTTP server, error handling is shared across all connections to a server. For example, if there are three connections to www.somehttpserver.com, a timeout error on one of those connections will cause all three connections to break and reconnect. A connection will not break and reconnect, however, unless a request becomes active on it within a certain amount of time after the error (as specified by HTTP_CONNECTION_RETRY_DELAY). An idle connection will not break even if other connections to the same server experience errors.
A HttpConnection will retry a request a certain number of times (as defined by HTTP_CONNNECTION_RETRY_COUNT). If all the retries fail, an exception is thrown and all HttpConnections associated with a server enter a probationary period defined by HTTP_CONNECTION_RETRY_DELAY. If the user makes a new request subsequent to entering probation, the request will fail immediately with the same exception thrown on probation entry. This is so that if the HTTP server has gone down, not every subsequent request must wait for a connect timeout before failing. After the probation period expires, the internal state of the HttpConnection is reset and subsequent requests have the full number of potential reconnects and retries available to them.
Constant Summary collapse
- HTTP_CONNECTION_RETRY_COUNT =
Number of times to retry the request after encountering the first error
3
- HTTP_CONNECTION_OPEN_TIMEOUT =
Throw a Timeout::Error if a connection isn’t established within this number of seconds
5
- HTTP_CONNECTION_READ_TIMEOUT =
Throw a Timeout::Error if no data have been read on this connnection within this number of seconds
120
- HTTP_CONNECTION_RETRY_DELAY =
Length of the post-error probationary period during which all requests will fail
15
- @@params =
class methods
{}
Instance Attribute Summary collapse
-
#http ⇒ Object
—————— instance methods ——————.
-
#logger ⇒ Object
Returns the value of attribute logger.
-
#params ⇒ Object
see @@params.
-
#server ⇒ Object
Returns the value of attribute server.
Class Method Summary collapse
-
.params ⇒ Object
Query the global (class-level) parameters:.
-
.params=(params) ⇒ Object
Set the global (class-level) parameters.
Instance Method Summary collapse
- #finish(reason = '') ⇒ Object
- #get_param(name, custom_options = {}) ⇒ Object
-
#initialize(params = {}) ⇒ HttpConnection
constructor
Params hash: :user_agent => ‘www.HostName.com’ # String to report as HTTP User agent :ca_file => ‘path_to_file’ # A path of a CA certification file in PEM format.
-
#local_read_size=(newsize) ⇒ Object
Set the maximum size (in bytes) of a single read from local data sources like files.
-
#local_read_size? ⇒ Boolean
Query for the maximum size (in bytes) of a single read from local data sources like files.
-
#request(request_params, &block) ⇒ Object
Send HTTP request to server.
-
#socket_read_size=(newsize) ⇒ Object
Set the maximum size (in bytes) of a single read from the underlying socket.
-
#socket_read_size? ⇒ Boolean
Query for the maximum size (in bytes) of a single read from the underlying socket.
Constructor Details
#initialize(params = {}) ⇒ HttpConnection
Params hash:
:user_agent => 'www.HostName.com' # String to report as HTTP User agent
:ca_file => 'path_to_file' # A path of a CA certification file in PEM format. The file can contain several CA certificates.
:fail_if_ca_mismatch => Boolean # If ca_file is set and the server certificate doesn't verify, a log line is generated regardless, but normally right_http_connection continues on past the failure. If this is set, fail to connect in that case. Defaults to false.
:logger => Logger object # If omitted, HttpConnection logs to STDOUT
:exception => Exception to raise # The type of exception to raise if a request repeatedly fails. RuntimeError is raised if this parameter is omitted.
:proxy_host => 'hostname' # hostname of HTTP proxy host to use, default none.
:proxy_port => port # port of HTTP proxy host to use, default none.
:proxy_username => 'username' # username to use for proxy authentication, default none.
:proxy_password => 'password' # password to use for proxy authentication, default none.
:http_connection_retry_count # by default == Rightscale::HttpConnection.params[:http_connection_retry_count]
:http_connection_open_timeout # by default == Rightscale::HttpConnection.params[:http_connection_open_timeout]
:http_connection_read_timeout # by default == Rightscale::HttpConnection.params[:http_connection_read_timeout]
:http_connection_retry_delay # by default == Rightscale::HttpConnection.params[:http_connection_retry_delay]
:raise_on_timeout # do not perform a retry if timeout is received (false by default)
138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 |
# File 'lib/right_http_connection.rb', line 138 def initialize(params={}) @params = params #set up logging first @logger = get_param(:logger) || (RAILS_DEFAULT_LOGGER if defined?(RAILS_DEFAULT_LOGGER)) || Logger.new(STDOUT) env_proxy_host, env_proxy_port, env_proxy_username, env_proxy_password = get_proxy_info_for_env if ENV['HTTP_PROXY'] @params[:http_connection_retry_count] ||= @@params[:http_connection_retry_count] @params[:http_connection_open_timeout] ||= @@params[:http_connection_open_timeout] @params[:http_connection_read_timeout] ||= @@params[:http_connection_read_timeout] @params[:http_connection_retry_delay] ||= @@params[:http_connection_retry_delay] @params[:proxy_host] ||= @@params[:proxy_host] || env_proxy_host @params[:proxy_port] ||= @@params[:proxy_port] || env_proxy_port @params[:proxy_username] ||= @@params[:proxy_username] || env_proxy_username @params[:proxy_password] ||= @@params[:proxy_password] || env_proxy_password @http = nil @server = nil #-------------- # Retry state - Keep track of errors on a per-server basis #-------------- @state = {} # retry state indexed by server: consecutive error count, error time, and error @eof = {} end |
Instance Attribute Details
#http ⇒ Object
instance methods
118 119 120 |
# File 'lib/right_http_connection.rb', line 118 def http @http end |
#logger ⇒ Object
Returns the value of attribute logger.
121 122 123 |
# File 'lib/right_http_connection.rb', line 121 def logger @logger end |
#params ⇒ Object
see @@params
120 121 122 |
# File 'lib/right_http_connection.rb', line 120 def params @params end |
#server ⇒ Object
Returns the value of attribute server.
119 120 121 |
# File 'lib/right_http_connection.rb', line 119 def server @server end |
Class Method Details
.params ⇒ Object
Query the global (class-level) parameters:
:user_agent => 'www.HostName.com' # String to report as HTTP User agent
:ca_file => 'path_to_file' # Path to a CA certification file in PEM format. The file can contain several CA certificates. If this parameter isn't set, HTTPS certs won't be verified.
:fail_if_ca_mismatch => Boolean # If ca_file is set and the server certificate doesn't verify, a log line is generated regardless, but normally right_http_connection continues on past the failure. If this is set, fail to connect in that case. Defaults to false.
:logger => Logger object # If omitted, HttpConnection logs to STDOUT
:exception => Exception to raise # The type of exception to raise
# if a request repeatedly fails. RuntimeError is raised if this parameter is omitted.
:proxy_host => 'hostname' # hostname of HTTP proxy host to use, default none.
:proxy_port => port # port of HTTP proxy host to use, default none.
:proxy_username => 'username' # username to use for proxy authentication, default none.
:proxy_password => 'password' # password to use for proxy authentication, default none.
:http_connection_retry_count # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_COUNT
:http_connection_open_timeout # by default == Rightscale::HttpConnection::HTTP_CONNECTION_OPEN_TIMEOUT
:http_connection_read_timeout # by default == Rightscale::HttpConnection::HTTP_CONNECTION_READ_TIMEOUT
:http_connection_retry_delay # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_DELAY
:raise_on_timeout # do not perform a retry if timeout is received (false by default)
106 107 108 |
# File 'lib/right_http_connection.rb', line 106 def self.params @@params end |
.params=(params) ⇒ Object
Set the global (class-level) parameters
111 112 113 |
# File 'lib/right_http_connection.rb', line 111 def self.params=(params) @@params = params end |
Instance Method Details
#finish(reason = '') ⇒ Object
531 532 533 534 535 536 537 538 539 |
# File 'lib/right_http_connection.rb', line 531 def finish(reason = '') if @http && @http.started? reason = ", reason: '#{reason}'" unless reason.empty? @logger.info("Closing #{@http.use_ssl? ? 'HTTPS' : 'HTTP'} connection to #{@http.address}:#{@http.port}#{reason}") @http.finish end ensure @http = nil end |
#get_param(name, custom_options = {}) ⇒ Object
181 182 183 |
# File 'lib/right_http_connection.rb', line 181 def get_param(name, ={}) [name] || @params[name] || @@params[name] end |
#local_read_size=(newsize) ⇒ Object
Set the maximum size (in bytes) of a single read from local data sources like files. This can be used to tune the performance of, for example, a streaming PUT of a large buffer.
209 210 211 |
# File 'lib/right_http_connection.rb', line 209 def local_read_size=(newsize) Net::HTTPGenericRequest.local_read_size=(newsize) end |
#local_read_size? ⇒ Boolean
Query for the maximum size (in bytes) of a single read from local data sources like files. This is important, for example, in a streaming PUT of a large buffer.
202 203 204 |
# File 'lib/right_http_connection.rb', line 202 def local_read_size? Net::HTTPGenericRequest.local_read_size? end |
#request(request_params, &block) ⇒ Object
Send HTTP request to server
request_params hash:
:server => 'www.HostName.com' # Hostname or IP address of HTTP server
:port => '80' # Port of HTTP server
:protocol => 'https' # http and https are supported on any port
:request => 'requeststring' # Fully-formed HTTP request to make
:proxy_host => 'hostname' # hostname of HTTP proxy host to use, default none.
:proxy_port => port # port of HTTP proxy host to use, default none.
:proxy_username => 'username' # username to use for proxy authentication, default none.
:proxy_password => 'password' # password to use for proxy authentication, default none.
:raise_on_timeout # do not perform a retry if timeout is received (false by default)
:http_connection_retry_count
:http_connection_open_timeout
:http_connection_read_timeout
:http_connection_retry_delay
:user_agent
:exception
Raises RuntimeError, Interrupt, and params[:exception] (if specified in new).
417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 |
# File 'lib/right_http_connection.rb', line 417 def request(request_params, &block) current_params = @params.merge(request_params) exception = get_param(:exception, current_params) || RuntimeError # Re-establish the connection if any of auth params has changed same_auth_params_as_before = SECURITY_PARAMS.select do |param| request_params[param] != get_param(param) end.empty? # We save the offset here so that if we need to retry, we can return the file pointer to its initial position mypos = get_fileptr_offset(current_params) loop do current_params[:protocol] ||= (current_params[:port] == 443 ? 'https' : 'http') # (re)open connection to server if none exists or params has changed same_server_as_before = @server == current_params[:server] && @port == current_params[:port] && @protocol == current_params[:protocol] && same_auth_params_as_before # if we are inside a delay between retries: no requests this time! # (skip this step if the endpoint has changed) if error_count > current_params[:http_connection_retry_count] && error_time + current_params[:http_connection_retry_delay] > Time.now && same_server_as_before # store the message (otherwise it will be lost after error_reset and # we will raise an exception with an empty text) = @logger.warn("#{err_header} re-raising same error: #{} " + "-- error count: #{error_count}, error age: #{Time.now.to_i - error_time.to_i}") raise exception.new() end # try to connect server(if connection does not exist) and get response data begin request = current_params[:request] request['User-Agent'] = get_param(:user_agent, current_params) || '' unless @http && @http.started? && same_server_as_before same_auth_params_as_before = true start(current_params) end # Detect if the body is a streamable object like a file or socket. If so, stream that # bad boy. setup_streaming(request) # update READ_TIMEOUT value (it can be passed with request_params hash) @http.read_timeout = get_param(:http_connection_read_timeout, current_params) response = @http.request(request, &block) error_reset eof_reset return response # We treat EOF errors and the timeout/network errors differently. Both # are tracked in different statistics blocks. Note below that EOF # errors will sleep for a certain (exponentially increasing) period. # Other errors don't sleep because there is already an inherent delay # in them; connect and read timeouts (for example) have already # 'slept'. It is still not clear which way we should treat errors # like RST and resolution failures. For now, there is no additional # delay for these errors although this may change in the future. # EOFError means the server closed the connection on us. rescue EOFError => e finish(e.) @logger.debug("#{err_header} server #{@server} closed connection") # if we have waited long enough - raise an exception... if raise_on_eof_exception? @logger.warn("#{err_header} raising #{exception} due to permanent EOF being received from #{@server}, error age: #{Time.now.to_i - eof_time.to_i}") raise exception.new("Permanent EOF is being received from #{@server}.") else # ... else just sleep a bit before new retry sleep(add_eof) # We will be retrying the request, so reset the file pointer reset_fileptr_offset(request, mypos) end rescue ArgumentError => e finish(e.) if e..include?('wrong number of arguments (5 for 4)') # seems our net_fix patch was overriden... raise exception.new('incompatible Net::HTTP monkey-patch') else raise e end rescue Timeout::Error, SocketError, SystemCallError, Interrupt => e # See comment at bottom for the list of errors seen... finish(e.) if e.is_a?(Errno::ETIMEDOUT) || e.is_a?(Timeout::Error) # Omit retries if it was explicitly requested # #6481: # ... When creating a resource in EC2 (instance, volume, snapshot, etc) it is undetermined what happened if the call times out. # The resource may or may not have been created in EC2. Retrying the call may cause multiple resources to be created... raise exception.new("#{e.class.name}: #{e.}") if current_params[:raise_on_timeout] elsif e.is_a?(Interrupt) # if ctrl+c is pressed - we have to reraise exception to terminate proggy @logger.debug( "#{err_header} request to server #{@server} interrupted by ctrl-c") raise e end # oops - we got a banana: log it error_add(e) @logger.warn("#{err_header} request failure count: #{error_count}, exception: #{e.inspect}") # We will be retrying the request, so reset the file pointer reset_fileptr_offset(request, mypos) end end end |
#socket_read_size=(newsize) ⇒ Object
Set the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.
195 196 197 |
# File 'lib/right_http_connection.rb', line 195 def socket_read_size=(newsize) Net::BufferedIO.socket_read_size=(newsize) end |
#socket_read_size? ⇒ Boolean
Query for the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.
188 189 190 |
# File 'lib/right_http_connection.rb', line 188 def socket_read_size? Net::BufferedIO.socket_read_size? end |