Class: Ferrum::Network

Inherits:
Object
  • Object
show all
Defined in:
lib/ferrum/network.rb,
lib/ferrum/network/error.rb,
lib/ferrum/network/request.rb,
lib/ferrum/network/exchange.rb,
lib/ferrum/network/response.rb,
lib/ferrum/network/auth_request.rb,
lib/ferrum/network/request_params.rb,
lib/ferrum/network/intercepted_request.rb

Overview

Tracks a page's network activity, exposing it as a list of #traffic Exchanges built from the underlying CDP Network.* events. Also provides request interception/authorization (intercept, authorize, blacklist=/whitelist=) and network condition emulation (emulate_network_conditions, offline_mode).

Defined Under Namespace

Modules: RequestParams Classes: AuthRequest, Error, Exchange, InterceptedRequest, Request, Response

Constant Summary collapse

CLEAR_TYPE = 
%i[traffic cache].freeze
AUTHORIZE_TYPE = 
%i[server proxy].freeze
REQUEST_STAGES = 
%i[Request Response].freeze
RESOURCE_TYPES = 
%i[Document Stylesheet Image Media Font Script TextTrack
XHR Fetch Prefetch EventSource WebSocket Manifest
SignedExchange Ping CSPViolationReport Preflight Other].freeze
AUTHORIZE_BLOCK_MISSING = 
"Block is missing, call `authorize(...) { |r| r.continue } " \
"or subscribe to `on(:request)` events before calling it"
AUTHORIZE_TYPE_WRONG = 
":type should be in #{AUTHORIZE_TYPE}".freeze
ALLOWED_CONNECTION_TYPE = 
%w[none cellular2g cellular3g cellular4g bluetooth ethernet wifi wimax other].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(page) ⇒ Network

Returns a new instance of Network.



42
43
44
45
46
47
48
49
 
# File 'lib/ferrum/network.rb', line 42

def initialize(page)
  @page = page
  @traffic = []
  @exchange = nil
  @blacklist = nil
  @whitelist = nil
  @mutex = Mutex.new
end

Instance Attribute Details

#traffic ⇒ Array<Exchange> (readonly)

Network traffic.

Examples:

browser.go_to("https://github.com/")
browser.network.traffic # => [#<Ferrum::Network::Exchange, ...]

Returns:

  • Returns all information about network traffic as Exchange instance which in general is a wrapper around request, response and error.



40
41
42
 
# File 'lib/ferrum/network.rb', line 40

def traffic
  @traffic
end

Instance Method Details

#authorize(user:, password:, type: :server) {|request| ... } ⇒ Object

Sets HTTP Basic-Auth credentials.

Examples:

browser.network.authorize(user: "login", password: "pass") { |req| req.continue }
browser.go_to("http://example.com/authenticated")
puts browser.network.status # => 200
puts browser.body # => Welcome, authenticated client

Parameters:

  • The username to send.

  • The password to send.

  • (defaults to: :server)

    Specifies whether the credentials are for a website or a proxy.

Yields:

  • (request) —

    The given block will be passed each authenticated request and can allow or deny the request.

Yield Parameters:

  • request (Request) —

    An HTTP request.

Raises:



305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
 
# File 'lib/ferrum/network.rb', line 305

def authorize(user:, password:, type: :server, &block)
  raise ArgumentError, AUTHORIZE_TYPE_WRONG unless AUTHORIZE_TYPE.include?(type)
  raise ArgumentError, AUTHORIZE_BLOCK_MISSING if !block_given? && !@page.subscribed?("Fetch.requestPaused")

  @authorized_ids ||= {}
  @authorized_ids[type] ||= []

  intercept

  @page.on(:request, &block)

  @page.on(:auth) do |request, index, total|
    if request.auth_challenge?(type)
      response = authorized_response(@authorized_ids[type],
                                     request.request_id,
                                     user, password)

      @authorized_ids[type] << request.request_id
      request.continue(authChallengeResponse: response)
    elsif index + 1 < total
      next # There are other callbacks that can handle this
    else
      request.abort
    end
  end
end

#authorized_response(ids, request_id, username, password) ⇒ Hash?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Builds the authChallengeResponse sent back to Chrome for an authenticated request, used by authorize.

Parameters:

  • Request ids that were already given credentials, to avoid an infinite retry loop if the credentials are rejected.

Returns:

API:

  • private



362
363
364
365
366
367
368
369
370
 
# File 'lib/ferrum/network.rb', line 362

def authorized_response(ids, request_id, username, password)
  if ids.include?(request_id)
    { response: "CancelAuth" }
  elsif username && password
    { response: "ProvideCredentials",
      username: username,
      password: password }
  end
end

#blacklist=(patterns) ⇒ Object Also known as: blocklist=

Sets a list of patterns for URLs that should be blocked from loading. Aborts any request whose URL matches one of the given patterns, and continues all others. Can't be used together with whitelist=.

Examples:

browser.network.blacklist = /jquery/
browser.go_to("https://example.com/")

Parameters:



215
216
217
218
 
# File 'lib/ferrum/network.rb', line 215

def blacklist=(patterns)
  @blacklist = Array(patterns)
  blacklist_subscribe
end

#build_exchange(id) ⇒ Exchange

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Builds a new Exchange for the given request id and appends it to traffic.

Parameters:

Returns:

API:

  • private



393
394
395
 
# File 'lib/ferrum/network.rb', line 393

def build_exchange(id)
  Network::Exchange.new(@page, id).tap { |e| @traffic << e }
end

#cache(disable:) ⇒ Object

Toggles ignoring cache for each request. If true, cache will not be used.

Examples:

browser.network.cache(disable: true)


472
473
474
 
# File 'lib/ferrum/network.rb', line 472

def cache(disable:)
  @page.command("Network.setCacheDisabled", cacheDisabled: disable)
end

#clear(type) ⇒ true

Clear browser's cache or collected traffic.

Examples:

traffic = browser.network.traffic # => []
browser.go_to("https://github.com/")
traffic.size # => 51
browser.network.clear(:traffic)
traffic.size # => 0

Parameters:

  • The type of traffic to clear.

Returns:

Raises:



190
191
192
193
194
195
196
197
198
199
200
 
# File 'lib/ferrum/network.rb', line 190

def clear(type)
  raise ArgumentError, ":type should be in #{CLEAR_TYPE}" unless CLEAR_TYPE.include?(type)

  if type == :traffic
    @traffic.clear
  else
    @page.command("Network.clearBrowserCache")
  end

  true
end

#emulate_network_conditions(offline: false, latency: 0, download_throughput: -1,, upload_throughput: -1,, connection_type: nil) ⇒ Object

Activates emulation of network conditions.

Examples:

browser.network.emulate_network_conditions(connection_type: "cellular2g")
browser.go_to("https://github.com/")

Parameters:

  • (defaults to: false)

    Emulate internet disconnection,

  • (defaults to: 0)

    Minimum latency from request sent to response headers received (ms).

  • (defaults to: -1,)

    Maximal aggregated download throughput (bytes/sec).

  • (defaults to: -1,)

    Maximal aggregated upload throughput (bytes/sec).

  • (defaults to: nil)

    Connection type if known:

    • "none"
    • "cellular2g"
    • "cellular3g"
    • "cellular4g"
    • "bluetooth"
    • "ethernet"
    • "wifi"
    • "wimax"
    • "other"


439
440
441
442
443
444
445
446
447
448
449
450
451
452
 
# File 'lib/ferrum/network.rb', line 439

def emulate_network_conditions(offline: false, latency: 0,
                               download_throughput: -1, upload_throughput: -1,
                               connection_type: nil)
  params = {
    offline: offline, latency: latency,
    downloadThroughput: download_throughput,
    uploadThroughput: upload_throughput
  }

  params[:connectionType] = connection_type if connection_type && ALLOWED_CONNECTION_TYPE.include?(connection_type)

  @page.command("Network.emulateNetworkConditions", **params)
  true
end

#find_or_build_exchange(id) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Network.requestWillBeSent and Fetch.requestPaused are handled on separate threads (see Client::Subscriber), so the "find the existing exchange for this id or build a new one" check has to be atomic, otherwise both threads can race past the select before either has appended, and end up building two exchanges for the same request.

API:

  • private



404
405
406
 
# File 'lib/ferrum/network.rb', line 404

def find_or_build_exchange(id)
  @mutex.synchronize { select(id).last || build_exchange(id) }
end

#finished_connections ⇒ Integer

Number of network connections that have finished, i.e. were blocked, got a loaded response, errored, or are otherwise no longer pending.

Returns:



121
122
123
 
# File 'lib/ferrum/network.rb', line 121

def finished_connections
  @traffic.count(&:finished?)
end

#idle?(connections = 0) ⇒ Boolean

Whether the network is idle, i.e. no more than connections connections are still pending.

Parameters:

  • (defaults to: 0)

    How many connections are allowed for network to be idling.

Returns:



101
102
103
 
# File 'lib/ferrum/network.rb', line 101

def idle?(connections = 0)
  pending_connections <= connections
end

#intercept(pattern: "*", resource_type: nil, request_stage: nil, handle_auth_requests: true) ⇒ Object

Set request interception for given options. This method is only sets request interception, you should use on callback to catch requests and abort or continue them.

Examples:

browser = Ferrum::Browser.new
browser.network.intercept
browser.on(:request) do |request|
  if request.match?(/bla-bla/)
    request.abort
  elsif request.match?(/lorem/)
    request.respond(body: "Lorem ipsum")
  else
    request.continue
  end
end
browser.go_to("https://google.com")

Parameters:



264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/ferrum/network.rb', line 264

def intercept(pattern: "*", resource_type: nil, request_stage: nil, handle_auth_requests: true)
  pattern = { urlPattern: pattern }

  if resource_type && RESOURCE_TYPES.none?(resource_type.to_sym)
    raise ArgumentError, "Unknown resource type '#{resource_type}' must be #{RESOURCE_TYPES.join(' | ')}"
  end

  if request_stage && REQUEST_STAGES.none?(request_stage.to_sym)
    raise ArgumentError, "Unknown request stage '#{request_stage}' must be #{REQUEST_STAGES.join(' | ')}"
  end

  pattern[:resourceType] = resource_type if resource_type
  pattern[:requestStage] = request_stage if request_stage
  @page.command("Fetch.enable", patterns: [pattern], handleAuthRequests: handle_auth_requests)
end

#offline_mode ⇒ Object

Activates offline mode for a page.

Examples:

browser.network.offline_mode
browser.go_to("https://github.com/")
  # => Request to https://github.com/ failed (net::ERR_INTERNET_DISCONNECTED) (Ferrum::StatusError)


462
463
464
 
# File 'lib/ferrum/network.rb', line 462

def offline_mode
  emulate_network_conditions(offline: true, latency: 0, download_throughput: 0, upload_throughput: 0)
end

#pending_connections ⇒ Integer

Number of network connections that are still pending, i.e. haven't finished yet.

Returns:



131
132
133
 
# File 'lib/ferrum/network.rb', line 131

def pending_connections
  total_connections - finished_connections
end

#request ⇒ Request?

Page request of the main frame.

Examples:

browser.go_to("https://github.com/")
browser.network.request # => #<Ferrum::Network::Request...

Returns:



144
145
146
 
# File 'lib/ferrum/network.rb', line 144

def request
  @exchange&.request
end

#response ⇒ Response?

Page response of the main frame.

Examples:

browser.go_to("https://github.com/")
browser.network.response # => #<Ferrum::Network::Response...

Returns:



157
158
159
 
# File 'lib/ferrum/network.rb', line 157

def response
  @exchange&.response
end

#select(request_id) ⇒ Array<Exchange>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Finds the exchanges in traffic with the given request id.

Parameters:

Returns:

API:

  • private



380
381
382
 
# File 'lib/ferrum/network.rb', line 380

def select(request_id)
  @traffic.select { |e| e.id == request_id }
end

#status ⇒ Integer?

Contains the status code of the main page response (e.g., 200 for a success). This is just a shortcut for response.status.

Examples:

browser.go_to("https://github.com/")
browser.network.status # => 200

Returns:



171
172
173
 
# File 'lib/ferrum/network.rb', line 171

def status
  response&.status
end

#subscribe ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Subscribes to the CDP events needed to keep track of traffic. Called once when the page is initialized.

API:

  • private



337
338
339
340
341
342
343
 
# File 'lib/ferrum/network.rb', line 337

def subscribe
  subscribe_request_will_be_sent
  subscribe_response_received
  subscribe_loading_finished
  subscribe_loading_failed
  subscribe_log_entry_added
end

#total_connections ⇒ Integer

Total number of network connections seen since the traffic was last cleared.

Returns:



111
112
113
 
# File 'lib/ferrum/network.rb', line 111

def total_connections
  @traffic.size
end

#wait_for_idle(connections: 0, duration: 0.05, timeout: @page.timeout) ⇒ Boolean

Waits for network idle.

Examples:

browser.go_to("https://example.com/")
browser.at_xpath("//a[text() = 'No UI changes button']").click
browser.network.wait_for_idle # => false

Parameters:

  • (defaults to: 0)

    how many connections are allowed for network to be idling,

  • (defaults to: 0.05)

    Sleep for given amount of time and check again.

  • (defaults to: @page.timeout)

    During what time we try to check idle.

Returns:



70
71
72
73
74
75
76
77
78
79
80
 
# File 'lib/ferrum/network.rb', line 70

def wait_for_idle(connections: 0, duration: 0.05, timeout: @page.timeout)
  start = Utils::ElapsedTime.monotonic_time

  until idle?(connections)
    return false if Utils::ElapsedTime.timeout?(start, timeout)

    sleep(duration)
  end

  true
end

#wait_for_idle! ⇒ Object

Waits for network idle or raises TimeoutError error. Accepts same arguments as wait_for_idle.

Raises:



87
88
89
90
 
# File 'lib/ferrum/network.rb', line 87

def wait_for_idle!(...)
  result = wait_for_idle(...)
  raise TimeoutError unless result
end

#whitelist=(patterns) ⇒ Object Also known as: allowlist=

Sets a list of patterns for URLs that are the only ones allowed to load. Continues any request whose URL matches one of the given patterns, and aborts all others. Can't be used together with blacklist=.

Examples:

browser.network.whitelist = /example/
browser.go_to("https://example.com/")

Parameters:



234
235
236
237
 
# File 'lib/ferrum/network.rb', line 234

def whitelist=(patterns)
  @whitelist = Array(patterns)
  whitelist_subscribe
end