Class: SSRFProxy::HTTP

Inherits:

Object

• Object
• SSRFProxy::HTTP

Defined in:

lib/ssrf_proxy/http.rb

Overview

SSRFProxy::HTTP object takes information required to connect to a HTTP(S) server vulnerable to Server-Side Request Forgery (SSRF) and issue arbitrary HTTP requests via the vulnerable server.

Once configured, the #send_uri and #send_request methods can be used to tunnel HTTP requests through the vulnerable server.

Several request modification options can be used to format the HTTP request appropriately for the SSRF vector and the destination web server accessed via the SSRF.

Several response modification options can be used to infer information about the response from the destination server and format the response such that the vulnerable intermediary server is mostly transparent to the client initiating the HTTP request.

Refer to the wiki for more information about configuring the SSRF, requestion modification, response modification, and example configurations: github.com/bcoles/ssrf_proxy/wiki/Configuration

Defined Under Namespace

Modules: Error

Instance Attribute Summary

• #headers ⇒ Hash readonly

  SSRF request HTTP headers.

• #logger ⇒ Logger readonly

  Logger.

• #method ⇒ String readonly

  SSRF request HTTP method.

• #post_data ⇒ String readonly

  SSRF request HTTP body.

• #proxy ⇒ URI readonly

  Upstream proxy.

• #url ⇒ URI readonly

  SSRF URL.

Instance Method Summary

• #initialize(url: nil, file: nil, proxy: nil, ssl: false, method: 'GET', placeholder: 'xxURLxx', post_data: nil, rules: nil, no_urlencode: false, ip_encoding: nil, match: '\A(.*)\z', strip: nil, decode_html: false, unescape: false, guess_mime: false, sniff_mime: false, guess_status: false, cors: false, timeout_ok: false, detect_headers: false, fail_no_content: false, forward_method: false, forward_headers: false, forward_body: false, forward_cookies: false, body_to_uri: false, auth_to_uri: false, cookies_to_uri: false, cache_buster: false, cookie: nil, user: nil, timeout: 10, user_agent: nil, insecure: false) ⇒ HTTP constructor

  SSRFProxy::HTTP accepts SSRF connection information, and configuration options for request modification and response modification.

• #send_request(request, use_ssl: false) ⇒ Hash

  Parse a raw HTTP request as a string, then send the requested URL and HTTP headers to #send_uri.

• #send_uri(uri, method: 'GET', headers: {}, body: '') ⇒ Hash

  Fetch a URI via SSRF.

Constructor Details

#initialize(url: nil, file: nil, proxy: nil, ssl: false, method: 'GET', placeholder: 'xxURLxx', post_data: nil, rules: nil, no_urlencode: false, ip_encoding: nil, match: '\A(.*)\z', strip: nil, decode_html: false, unescape: false, guess_mime: false, sniff_mime: false, guess_status: false, cors: false, timeout_ok: false, detect_headers: false, fail_no_content: false, forward_method: false, forward_headers: false, forward_body: false, forward_cookies: false, body_to_uri: false, auth_to_uri: false, cookies_to_uri: false, cache_buster: false, cookie: nil, user: nil, timeout: 10, user_agent: nil, insecure: false) ⇒ HTTP

SSRFProxy::HTTP accepts SSRF connection information, and configuration options for request modification and response modification.

Examples:

Configure SSRF with URL, GET method

SSRFProxy::HTTP.new(url: 'http://example.local/?url=xxURLxx')

Configure SSRF with URL, POST method

SSRFProxy::HTTP.new(url: 'http://example.local/',
                    method: 'POST',
                    post_data: 'url=xxURLxx')

Configure SSRF with raw HTTP request file

SSRFProxy::HTTP.new(file: 'ssrf.txt')

Configure SSRF with raw HTTP request file and force SSL/TLS

SSRFProxy::HTTP.new(file: 'ssrf.txt', ssl: true)

Configure SSRF with raw HTTP request StringIO

SSRFProxy::HTTP.new(file: StringIO.new("GET http://example.local/?url=xxURLxx HTTP/1.1\n\n"))

Parameters:

• url (String) (defaults to: nil) —

  Target URL vulnerable to SSRF

• file (String) (defaults to: nil) —

  Load HTTP request from a file

• proxy (String) (defaults to: nil) —

  Use a proxy to connect to the server. (Supported proxies: http, https, socks)

• ssl (Boolean) (defaults to: false) —

  Connect using SSL/TLS

• method (String) (defaults to: 'GET') —

  HTTP method (GET/HEAD/DELETE/POST/PUT/OPTIONS) (Default: GET)

• post_data (String) (defaults to: nil) —

  HTTP post data

• user (String) (defaults to: nil) —

  HTTP basic authentication credentials

• rules (String) (defaults to: nil) —

  Rules for parsing client request (separated by ‘,’) (Default: none)

• no_urlencode (Boolean) (defaults to: false) —

  Do not URL encode client request

• ip_encoding (String) (defaults to: nil) —

  Encode client request host IP address. (Modes: int, ipv6, oct, hex, dotted_hex)

• match (String) (defaults to: '\A(.*)\z') —

  Regex to match response body content. (Default: A(.*)z)

• strip (String) (defaults to: nil) —

  Headers to remove from the response. (separated by ‘,’) (Default: none)

• decode_html (Boolean) (defaults to: false) —

  Decode HTML entities in response body

• unescape (Boolean) (defaults to: false) —

  Unescape special characters in response body

• guess_status (Boolean) (defaults to: false) —

  Replaces response status code and message headers (determined by common strings in the response body, such as 404 Not Found.)

• guess_mime (Boolean) (defaults to: false) —

  Replaces response content-type header with the appropriate mime type (determined by the file extension of the requested resource.)

• sniff_mime (Boolean) (defaults to: false) —

  Replaces response content-type header with the appropriate mime type (determined by magic bytes in the response body.)

• timeout_ok (Boolean) (defaults to: false) —

  Replaces timeout HTTP status code 504 with 200.

• detect_headers (Boolean) (defaults to: false) —

  Replaces response headers if response headers are identified in the response body.

• fail_no_content (Boolean) (defaults to: false) —

  Return HTTP status 502 if response body is empty.

• forward_method (Boolean) (defaults to: false) —

  Forward client request method

• forward_headers (Boolean) (defaults to: false) —

  Forward all client request headers

• forward_body (Boolean) (defaults to: false) —

  Forward client request body

• forward_cookies (Boolean) (defaults to: false) —

  Forward client request cookies

• body_to_uri (Boolean) (defaults to: false) —

  Add client request body to URI query string

• auth_to_uri (Boolean) (defaults to: false) —

  Use client request basic authentication credentials in request URI.

• cookies_to_uri (Boolean) (defaults to: false) —

  Add client request cookies to URI query string

• cache_buster (Boolean) (defaults to: false) —

  Append a random value to the client request query string

• timeout (Integer) (defaults to: 10) —

  Connection timeout in seconds (Default: 10)

• user_agent (String) (defaults to: nil) —

  HTTP user-agent (Default: none)

• insecure (Boolean) (defaults to: false) —

  Skip server SSL certificate validation

Raises:

• (SSRFProxy::HTTP::Error::InvalidSsrfRequest) —

  Invalid SSRF request specified.

• (SSRFProxy::HTTP::Error::InvalidUpstreamProxy) —

  Invalid upstream proxy specified.

• (SSRFProxy::HTTP::Error::InvalidSsrfRequestMethod) —

  Invalid SSRF request method specified. Supported methods: GET, HEAD, DELETE, POST, PUT, OPTIONS.

• (SSRFProxy::HTTP::Error::NoUrlPlaceholder) —

  ‘xxURLxx’ URL placeholder must be specified in the

  SSRF request URL or body.
  

• (SSRFProxy::HTTP::Error::InvalidIpEncoding) —

  Invalid IP encoding method specified.

# File 'lib/ssrf_proxy/http.rb', line 183

def initialize(url: nil,
               file: nil,
               proxy: nil,
               ssl: false,
               method: 'GET',
               placeholder: 'xxURLxx',
               post_data: nil,
               rules: nil,
               no_urlencode: false,
               ip_encoding: nil,
               match: '\A(.*)\z',
               strip: nil,
               decode_html: false,
               unescape: false,
               guess_mime: false,
               sniff_mime: false,
               guess_status: false,
               cors: false,
               timeout_ok: false,
               detect_headers: false,
               fail_no_content: false,
               forward_method: false,
               forward_headers: false,
               forward_body: false,
               forward_cookies: false,
               body_to_uri: false,
               auth_to_uri: false,
               cookies_to_uri: false,
               cache_buster: false,
               cookie: nil,
               user: nil,
               timeout: 10,
               user_agent: nil,
               insecure: false)

  @SUPPORTED_METHODS = %w[GET HEAD DELETE POST PUT OPTIONS].freeze
  @SUPPORTED_IP_ENCODINGS = %w[int ipv6 oct hex dotted_hex].freeze

  @logger = ::Logger.new(STDOUT).tap do |log|
    log.progname = 'ssrf-proxy'
    log.level = ::Logger::WARN
    log.datetime_format = '%Y-%m-%d %H:%M:%S '
  end

  # SSRF configuration options
  @proxy = nil
  @placeholder = placeholder.to_s || 'xxURLxx'
  @method = 'GET'
  @headers ||= {}
  @post_data = post_data.to_s || ''
  @rules = rules.to_s.split(/,/) || []
  @no_urlencode = no_urlencode || false

  # client request modification
  @ip_encoding = nil
  @forward_method = forward_method || false
  @forward_headers = forward_headers || false
  @forward_body = forward_body || false
  @forward_cookies = forward_cookies || false
  @body_to_uri = body_to_uri || false
  @auth_to_uri = auth_to_uri || false
  @cookies_to_uri = cookies_to_uri || false
  @cache_buster = cache_buster || false

  # SSRF connection options
  @user = ''
  @pass = ''
  @timeout = timeout.to_i || 10
  @insecure = insecure || false

  # HTTP response modification options
  @match_regex = match.to_s || '\A(.*)\z'
  @strip = strip.to_s.downcase.split(/,/) || []
  @decode_html = decode_html || false
  @unescape = unescape || false
  @guess_status = guess_status || false
  @guess_mime = guess_mime || false
  @sniff_mime = sniff_mime || false
  @detect_headers = detect_headers || false
  @fail_no_content = fail_no_content || false
  @timeout_ok = timeout_ok || false
  @cors = cors || false

  # ensure either a URL or file path was provided
  if url.to_s.eql?('') && file.to_s.eql?('')
    raise ArgumentError,
          "Option 'url' or 'file' must be provided."
  end

  # parse HTTP request file
  unless file.to_s.eql?('')
    unless url.to_s.eql?('')
      raise ArgumentError,
            "Options 'url' and 'file' are mutually exclusive."
    end

    if file.is_a?(String)
      if File.exist?(file) && File.readable?(file)
        http = File.read(file).to_s
      else
        raise SSRFProxy::HTTP::Error::InvalidSsrfRequest.new,
              "Invalid SSRF request specified : Could not read file #{file.inspect}"
      end
    elsif file.is_a?(StringIO)
      http = file.read
    end

    req = parse_http_request(http)
    url = req['uri']
    @method = req['method']
    @headers = {}
    req['headers'].each do |k, v|
      @headers[k.downcase] = v.flatten.first
    end
    @headers.delete('host')
    @post_data = req['body']
  end

  # parse target URL
  begin
    @url = URI.parse(url.to_s)
  rescue URI::InvalidURIError
    raise SSRFProxy::HTTP::Error::InvalidSsrfRequest.new,
          'Invalid SSRF request specified : Could not parse URL.'
  end

  if @url.scheme.nil? || @url.host.nil? || @url.port.nil?
    raise SSRFProxy::HTTP::Error::InvalidSsrfRequest.new,
          'Invalid SSRF request specified : Invalid URL.'
  end

  unless @url.scheme.eql?('http') || @url.scheme.eql?('https')
    raise SSRFProxy::HTTP::Error::InvalidSsrfRequest.new,
          'Invalid SSRF request specified : URL scheme must be http(s).'
  end

  if proxy
    begin
      @proxy = URI.parse(proxy.to_s)
    rescue URI::InvalidURIError
      raise SSRFProxy::HTTP::Error::InvalidUpstreamProxy.new,
            'Invalid upstream proxy specified.'
    end
    if @proxy.host.nil? || @proxy.port.nil?
      raise SSRFProxy::HTTP::Error::InvalidUpstreamProxy.new,
            'Invalid upstream proxy specified.'
    end
    if @proxy.scheme !~ /\A(socks|https?)\z/
      raise SSRFProxy::HTTP::Error::InvalidUpstreamProxy.new,
            'Unsupported upstream proxy specified. ' \
            'Scheme must be http(s) or socks.'
    end
  end

  if ssl
    @url.scheme = 'https'
  end

  if method
    case method.to_s.downcase
    when 'get'
      @method = 'GET'
    when 'head'
      @method = 'HEAD'
    when 'delete'
      @method = 'DELETE'
    when 'post'
      @method = 'POST'
    when 'put'
      @method = 'PUT'
    when 'options'
      @method = 'OPTIONS'
    else
      raise SSRFProxy::HTTP::Error::InvalidSsrfRequestMethod.new,
            'Invalid SSRF request method specified. ' \
            "Supported methods: #{@SUPPORTED_METHODS.join(', ')}."
    end
  end

  if ip_encoding
    unless @SUPPORTED_IP_ENCODINGS.include?(ip_encoding)
      raise SSRFProxy::HTTP::Error::InvalidIpEncoding.new,
            'Invalid IP encoding method specified.'
    end
    @ip_encoding = ip_encoding.to_s
  end

  if cookie
    @headers['cookie'] = cookie.to_s
  end

  if user
    if user.to_s =~ /^(.*?):(.*)/
      @user = $1
      @pass = $2
    else
      @user = user.to_s
    end
  end

  if user_agent
    @headers['user-agent'] = user_agent
  end

  # Ensure a URL placeholder was provided
  unless @url.request_uri.to_s.include?(@placeholder) ||
         @post_data.to_s.include?(@placeholder) ||
         @headers.to_s.include?(@placeholder)
    raise SSRFProxy::HTTP::Error::NoUrlPlaceholder.new,
          'You must specify a URL placeholder with ' \
          "'#{@placeholder}' in the SSRF request"
  end
end

Instance Attribute Details

#headers ⇒ Hash (readonly)

Returns SSRF request HTTP headers.

Returns:

• (Hash) —

  SSRF request HTTP headers

# File 'lib/ssrf_proxy/http.rb', line 42

def headers
  @headers
end

#logger ⇒ Logger (readonly)

Returns logger.

Returns:

• (Logger) —

  logger

# File 'lib/ssrf_proxy/http.rb', line 34

def logger
  @logger
end

#method ⇒ String (readonly)

Returns SSRF request HTTP method.

Returns:

• (String) —

  SSRF request HTTP method

# File 'lib/ssrf_proxy/http.rb', line 40

def method
  @method
end

#post_data ⇒ String (readonly)

Returns SSRF request HTTP body.

Returns:

• (String) —

  SSRF request HTTP body

# File 'lib/ssrf_proxy/http.rb', line 44

def post_data
  @post_data
end

#proxy ⇒ URI (readonly)

Returns upstream proxy.

Returns:

• (URI) —

  upstream proxy

# File 'lib/ssrf_proxy/http.rb', line 38

def proxy
  @proxy
end

#url ⇒ URI (readonly)

Returns SSRF URL.

Returns:

• (URI) —

  SSRF URL

# File 'lib/ssrf_proxy/http.rb', line 36

def url
  @url
end

Instance Method Details

#send_request(request, use_ssl: false) ⇒ Hash

Parse a raw HTTP request as a string, then send the requested URL and HTTP headers to #send_uri

Parameters:

• request (String) —

  Raw HTTP request

• use_ssl (Boolean) (defaults to: false) —

  Connect using SSL/TLS

Returns:

• (Hash) —

  HTTP response hash (version, code, message, headers, body)

# File 'lib/ssrf_proxy/http.rb', line 467

def send_request(request, use_ssl: false)
  req = parse_http_request(request)
  req['uri'].scheme = 'https' if use_ssl
  send_uri(req['uri'],
           method: req['method'],
           headers: req['headers'],
           body: req['body'])
end

#send_uri(uri, method: 'GET', headers: {}, body: '') ⇒ Hash

Fetch a URI via SSRF

Parameters:

• uri (String) —

  URI to fetch

• method (String) (defaults to: 'GET') —

  HTTP request method

• headers (Hash) (defaults to: {}) —

  HTTP request headers

• body (String) (defaults to: '') —

  HTTP request body

Returns:

• (Hash) —

  HTTP response hash (version, code, message, headers, body, etc)

Raises:

• (SSRFProxy::HTTP::Error::InvalidClientRequest) —

  An invalid client HTTP request was supplied.

# File 'lib/ssrf_proxy/http.rb', line 489

def send_uri(uri, method: 'GET', headers: {}, body: '')
  uri = uri.to_s
  body = body.to_s
  headers = {} unless headers.is_a?(Hash)

  # validate url
  unless uri.start_with?('http://', 'https://')
    raise SSRFProxy::HTTP::Error::InvalidClientRequest,
          'Invalid request URI'
  end

  # set request method
  if @forward_method
    if @SUPPORTED_METHODS.include?(method)
      request_method = method
    else
      raise SSRFProxy::HTTP::Error::InvalidClientRequest,
            "Request method '#{method}' is not supported"
    end
  else
    request_method = @method
  end

  # parse request headers
  client_headers = {}
  headers.each do |k, v|
    if v.is_a?(Array)
      client_headers[k.downcase] = v.flatten.first
    elsif v.is_a?(String)
      client_headers[k.downcase] = v.to_s
    else
      raise SSRFProxy::HTTP::Error::InvalidClientRequest,
            "Request header #{k.inspect} value is malformed: #{v}"
    end
  end

  # reject websocket requests
  if client_headers['upgrade'].to_s.start_with?('WebSocket')
    logger.warn('WebSocket tunneling is not supported')
    raise SSRFProxy::HTTP::Error::InvalidClientRequest,
          'WebSocket tunneling is not supported'
  end

  # copy request body to URL
  if @body_to_uri && !body.eql?('')
    logger.debug("Parsing request body: #{body}")
    separator = uri.include?('?') ? '&' : '?'
    uri = "#{uri}#{separator}#{body}"
    logger.info("Added request body to URI: #{body.inspect}")
  end

  # copy basic authentication credentials to uri
  if @auth_to_uri && client_headers['authorization'].to_s.downcase.start_with?('basic ')
    logger.debug("Parsing basic authentication header: #{client_headers['authorization']}")
    begin
      creds = client_headers['authorization'].split(' ')[1]
      user = Base64.decode64(creds).chomp
      uri = uri.gsub!(%r{://}, "://#{CGI.escape(user).gsub(/\+/, '%20').gsub('%3A', ':')}@")
      logger.info("Using basic authentication credentials: #{user}")
    rescue
      logger.warn('Could not parse request authorization header: ' \
                  "#{client_headers['authorization']}")
    end
  end

  # copy cookies to uri
  cookies = []
  if @cookies_to_uri && !client_headers['cookie'].nil?
    logger.debug("Parsing request cookies: #{client_headers['cookie']}")
    client_headers['cookie'].split(/;\s*/).each do |c|
      cookies << c.to_s unless c.nil?
    end
    separator = uri.include?('?') ? '&' : '?'
    uri = "#{uri}#{separator}#{cookies.join('&')}"
    logger.info("Added cookies to URI: #{cookies.join('&')}")
  end

  # add cache buster
  if @cache_buster
    separator = uri.include?('?') ? '&' : '?'
    junk = "#{rand(36**6).to_s(36)}=#{rand(36**6).to_s(36)}"
    uri = "#{uri}#{separator}#{junk}"
  end

  # set request headers
  request_headers = @headers.dup

  # forward request cookies
  new_cookie = []
  new_cookie << @headers['cookie'] unless @headers['cookie'].to_s.eql?('')
  if @forward_cookies && !client_headers['cookie'].nil?
    client_headers['cookie'].split(/;\s*/).each do |c|
      new_cookie << c.to_s unless c.nil?
    end
  end
  unless new_cookie.empty?
    request_headers['cookie'] = new_cookie.uniq.join('; ')
    logger.info("Using cookie: #{new_cookie.join('; ')}")
  end

  # forward request headers and strip proxy headers
  if @forward_headers && !client_headers.empty?
    client_headers.each do |k, v|
      next if k.eql?('proxy-connection')
      next if k.eql?('proxy-authorization')
      if v.is_a?(Array)
        request_headers[k.downcase] = v.flatten.first
      elsif v.is_a?(String)
        request_headers[k.downcase] = v.to_s
      end
    end
  end

  # encode target host ip
  ip_encoded_uri = @ip_encoding ? encode_ip(uri, @ip_encoding) : uri

  # run request URI through rules
  target_uri = run_rules(ip_encoded_uri, @rules).to_s

  # URL encode target URI
  unless @no_urlencode
    target_uri = CGI.escape(target_uri).gsub(/\+/, '%20').to_s
  end

  # set path and query string
  if @url.query.to_s.eql?('')
    ssrf_url = @url.path.to_s
  else
    ssrf_url = "#{@url.path}?#{@url.query}"
  end

  # replace xxURLxx placeholder in request URL
  ssrf_url.gsub!(/#{@placeholder}/, target_uri)

  # replace xxURLxx placeholder in request body
  post_data = @post_data.gsub(/#{@placeholder}/, target_uri)

  # set request body
  if @forward_body && !body.eql?('')
    request_body = post_data.eql?('') ? body : "#{post_data}&#{body}"
  else
    request_body = post_data
  end

  # replace xxURLxx in request header values
  request_headers.each do |k, v|
    request_headers[k] = v.gsub(/#{@placeholder}/, target_uri)
  end

  # set content type
  if request_headers['content-type'].nil? && !request_body.eql?('')
    request_headers['content-type'] = 'application/x-www-form-urlencoded'
  end

  # set content length
  request_headers['content-length'] = request_body.length.to_s

  # send request
  response = nil
  start_time = Time.now
  begin
    response = send_http_request(ssrf_url,
                                 request_method,
                                 request_headers,
                                 request_body)
    if response['content-encoding'].to_s.downcase.eql?('gzip') && response.body
      begin
        sio = StringIO.new(response.body)
        gz = Zlib::GzipReader.new(sio)
        response.body = gz.read
      rescue
        logger.warn('Could not decompress response body')
      end
    end

    result = { 'url'          => uri,
               'http_version' => response.http_version,
               'code'         => response.code,
               'message'      => response.message,
               'headers'      => '',
               'body'         => response.body.to_s || '' }
  rescue SSRFProxy::HTTP::Error::ConnectionTimeout => e
    unless @timeout_ok
      raise SSRFProxy::HTTP::Error::ConnectionTimeout, e.message
    end
    result = { 'url'          => uri,
               'http_version' => '1.0',
               'code'         => 200,
               'message'      => 'Timeout',
               'headers'      => '',
               'body'         => '' }
    logger.info('Changed HTTP status code 504 to 200')
  end

  # set duration
  end_time = Time.now
  duration = ((end_time - start_time) * 1000).round(3)
  result['duration'] = duration

  # body content encoding
  result['body'].force_encoding('BINARY')
  unless result['body'].valid_encoding?
    begin
      result['body'] = result['body'].encode(
        'UTF-8',
        'binary',
        :invalid => :replace,
        :undef   => :replace,
        :replace => ''
      )
    rescue
    end
  end

  logger.info("Received #{result['body'].bytes.length} bytes in #{duration} ms")

  # match response content
  unless @match_regex.nil?
    matches = result['body'].scan(/#{@match_regex}/m)
    if !matches.empty?
      result['body'] = matches.flatten.first.to_s
      logger.info("Response body matches pattern '#{@match_regex}'")
    else
      result['body'] = ''
      logger.warn("Response body does not match pattern '#{@match_regex}'")
    end
  end

  # return 502 if matched response body is empty
  if @fail_no_content
    if result['body'].to_s.eql?('')
      result['code'] = 502
      result['message'] = 'Bad Gateway'
      result['status_line'] = "HTTP/#{result['http_version']} #{result['code']} #{result['message']}"
      return result
    end
  end

  # unescape response body
  if @unescape
    # unescape slashes
    result['body'] = result['body'].tr('\\', '\\')
    result['body'] = result['body'].gsub('\\/', '/')
    # unescape whitespace
    result['body'] = result['body'].gsub('\r', "\r")
    result['body'] = result['body'].gsub('\n', "\n")
    result['body'] = result['body'].gsub('\t', "\t")
    # unescape quotes
    result['body'] = result['body'].gsub('\"', '"')
    result['body'] = result['body'].gsub("\\'", "'")
  end

  # decode HTML entities
  if @decode_html
    result['body'] = HTMLEntities.new.decode(result['body'])
  end

  # set title
  result['title'] = result['body'][0..8192] =~ %r{<title>([^<]*)</title>}im ? $1.to_s : ''

  # guess HTTP response code and message
  if @guess_status
    head = result['body'][0..8192]
    status = guess_status(head)
    unless status.empty?
      result['code'] = status['code']
      result['message'] = status['message']
      logger.info("Using HTTP response status: #{result['code']} #{result['message']}")
    end
  end

  # replace timeout response with 200 OK
  if @timeout_ok
    if result['code'].eql?('504')
      logger.info('Changed HTTP status code 504 to 200')
      result['code'] = 200
    end
  end

  # detect headers in response body
  if @detect_headers
    headers = ''
    head = result['body'][0..8192] # use first 8192 byes
    detected_headers = head.scan(%r{HTTP/(1\.\d) (\d+) (.*?)\r?\n(.*?)\r?\n\r?\n}m)

    if detected_headers.empty?
      logger.info('Found no HTTP response headers in response body.')
    else
      # HTTP redirects may contain more than one set of HTTP response headers
      # Use the last
      logger.info("Found #{detected_headers.count} sets of HTTP response headers in reponse. Using last.")
      version = detected_headers.last[0]
      code = detected_headers.last[1]
      message = detected_headers.last[2]
      detected_headers.last[3].split(/\r?\n/).each do |line|
        if line =~ /^[A-Za-z0-9\-_\.]+: /
          k = line.split(': ').first
          v = line.split(': ')[1..-1].flatten.first
          headers << "#{k}: #{v}\n"
        else
          logger.warn('Could not use response headers in response body : Headers are malformed.')
          headers = ''
          break
        end
      end
    end
    unless headers.eql?('')
      result['http_version'] = version
      result['code'] = code.to_i
      result['message'] = message
      result['headers'] = headers
      result['body'] = result['body'].split(/\r?\n\r?\n/)[detected_headers.count..-1].flatten.join("\n\n")
    end
  end

  # set status line
  result['status_line'] = "HTTP/#{result['http_version']} #{result['code']} #{result['message']}"

  # strip unwanted HTTP response headers
  unless response.nil?
    response.each_header do |header_name, header_value|
      if header_name.downcase.eql?('content-encoding')
        next if header_value.downcase.eql?('gzip')
      end

      if @strip.include?(header_name.downcase)
        logger.info("Removed response header: #{header_name}")
        next
      end
      result['headers'] << "#{header_name}: #{header_value}\n"
    end
  end

  # add wildcard CORS header
  if @cors
    result['headers'] << "Access-Control-Allow-Origin: *\n"
  end

  # advise client to close HTTP connection
  if result['headers'] =~ /^connection:.*$/i
    result['headers'].gsub!(/^connection:.*$/i, 'Connection: close')
  else
    result['headers'] << "Connection: close\n"
  end

  # guess mime type and add content-type header
  content_type = nil
  if @sniff_mime
    head = result['body'][0..8192] # use first 8192 byes
    content_type = sniff_mime(head)
    if content_type.nil?
      content_type = guess_mime(File.extname(uri.to_s.split('?').first))
    end
  elsif @guess_mime
    content_type = guess_mime(File.extname(uri.to_s.split('?').first))
  end

  unless content_type.nil?
    logger.info("Using content-type: #{content_type}")
    if result['headers'] =~ /^content\-type:.*$/i
      result['headers'].gsub!(/^content\-type:.*$/i,
                              "Content-Type: #{content_type}")
    else
      result['headers'] << "Content-Type: #{content_type}\n"
    end
  end

  # prompt for password if unauthorised
  if result['code'] == 401
    if result['headers'] !~ /^WWW-Authenticate:.*$/i
      auth_uri = URI.parse(uri.to_s.split('?').first)
      realm = "#{auth_uri.host}:#{auth_uri.port}"
      result['headers'] << "WWW-Authenticate: Basic realm=\"#{realm}\"\n"
      logger.info("Added WWW-Authenticate header for realm: #{realm}")
    end
  end

  # set location header if redirected
  if result['code'] == 301 || result['code'] == 302
    if result['headers'] !~ /^location:.*$/i
      location = nil
      if result['body'] =~ /This document may be found <a href="(.+)">/i
        location = $1
      elsif result['body'] =~ /The document has moved <a href="(.+)">/i
        location = $1
      end
      unless location.nil?
        result['headers'] << "Location: #{location}\n"
        logger.info("Added Location header: #{location}")
      end
    end
  end

  # set content length
  content_length = result['body'].length
  if result['headers'] =~ /^transfer\-encoding:.*$/i
    result['headers'].gsub!(/^transfer\-encoding:.*$/i,
                            "Content-Length: #{content_length}")
  elsif result['headers'] =~ /^content\-length:.*$/i
    result['headers'].gsub!(/^content\-length:.*$/i,
                            "Content-Length: #{content_length}")
  else
    result['headers'] << "Content-Length: #{content_length}\n"
  end

  # return HTTP response
  logger.debug("Response:\n" \
               "#{result['status_line']}\n" \
               "#{result['headers']}\n" \
               "#{result['body']}")
  result
end