Class: Arachni::RPC::Server::Instance

Inherits:
Object
  • Object
show all
Includes:
UI::Output, Utilities
Defined in:
lib/arachni/rpc/server/instance.rb

Overview

Note:

Ignore:

  • Inherited methods and attributes – only public methods of this class are

    accessible over RPC.

  • ‘block` parameters, they are an RPC implementation detail for methods which

    perform asynchronous operations.
Note:

Avoid calling methods which return Arachni-specific objects (like AuditStore, Issue, etc.) when you don’t have these objects available on the client-side (like when working from a non-Ruby platform or not having the Arachni framework installed).

Note:

Methods which expect ‘Symbol` type parameters will also accept `String` types as well.

For example, the following:

instance.service.scan url: 'http://testfire.net'

Is the same as:

instance.service.scan 'url' => 'http://testfire.net'

Represents an Arachni instance (or multiple instances when running a high-performance scan) and serves as a central point of access to the scanner’s components:

# Convenience methods

The ‘service` RPC handler (which is this class) provides convenience methods which cover the most commonly used functionality so that you won’t have to concern yourself with any other RPC handler.

This should be the only RPC API you’ll ever need.

Provided methods for:

(A nice simple example can be found in the RPC command-line client interface.)

Examples:

A minimalistic example – assumes Arachni is installed and available.

require 'arachni'
require 'arachni/rpc/client'

instance = Arachni::RPC::Client::Instance.new( Arachni::Options.instance,
                                               'localhost:1111', 's3cr3t' )

instance.service.scan url: 'http://testfire.net',
                      audit_links: true,
                      audit_forms: true,
                      # load all XSS modules
                      modules: 'xss*'

print 'Running.'
while instance.service.busy?
    print '.'
    sleep 1
end

# Grab the report as a native AuditStore object
report = instance.service.auditstore

# Kill the instance and its process, no zombies please...
instance.service.shutdown

puts
puts
puts 'Logged issues:'
report.issues.each do |issue|
    puts "  * #{issue.name} for input '#{issue.var}' at '#{issue.url}'."
end

Author:

Instance Method Summary collapse

Methods included from Utilities

#available_port, #cookie_encode, #cookies_from_document, #cookies_from_file, #cookies_from_response, #exception_jail, #exclude_path?, #extract_domain, #follow_protocol?, #form_decode, #form_encode, #form_parse_request_body, #forms_from_document, #forms_from_response, #generate_token, #get_path, #html_decode, #html_encode, #include_path?, #links_from_document, #links_from_response, #normalize_url, #page_from_response, #page_from_url, #parse_query, #parse_set_cookie, #parse_url_vars, #path_in_domain?, #path_too_deep?, #port_available?, #rand_port, #redundant_path?, #remove_constants, #seed, #skip_page?, #skip_path?, #skip_resource?, #to_absolute, #uri_decode, #uri_encode, #uri_parse, #uri_parser, #url_sanitize

Methods included from UI::Output

#debug?, #debug_off, #debug_on, #disable_only_positives, #error_logfile, #flush_buffer, #log_error, #mute, #muted?, old_reset_output_options, #only_positives, #only_positives?, #print_bad, #print_debug, #print_debug_backtrace, #print_debug_pp, #print_error, #print_error_backtrace, #print_info, #print_line, #print_ok, #print_status, #print_verbose, #reroute_to_file, #reroute_to_file?, reset_output_options, #set_buffer_cap, #set_error_logfile, #uncap_buffer, #unmute, #verbose, #verbose?

Constructor Details

#initialize(opts, token) ⇒ Instance

Initializes the RPC interface and the framework.

Parameters:



145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/arachni/rpc/server/instance.rb', line 145

def initialize( opts, token )
    @opts   = opts
    @token  = token

    @framework      = Server::Framework.new( Options.instance )
    @active_options = Server::ActiveOptions.new( @framework )

    @server = Base.new( @opts, token )
    @server.logger.level = @opts.datastore[:log_level] if @opts.datastore[:log_level]

    @opts.datastore[:token] = token

    debug if @opts.debug

    if @opts.reroute_to_logfile
        reroute_to_file "#{@opts.dir['logs']}/Instance - #{Process.pid}-#{@opts.rpc_port}.log"
    else
        reroute_to_file false
    end

    set_error_logfile "#{@opts.dir['logs']}/Instance - #{Process.pid}-#{@opts.rpc_port}.error.log"

    set_handlers( @server )

    # trap interrupts and exit cleanly when required
    %w(QUIT INT).each do |signal|
        trap( signal ){ shutdown if !@opts.datastore[:do_not_trap] } if Signal.list.has_key?( signal )
    end

    @consumed_pids = []

    ::EM.run do
        run
    end
end

Instance Method Details

#abort_and_report(report_type = :hash, &block) ⇒ Hash, AuditStore

Note:

Don’t forget to #shutdown the instance once you get the report.

Cleans up and returns the report.

Parameters:

  • report_type (Symbol) (defaults to: :hash) —

    Report type to return, ‘:hash` for #report or `:audistore` for #auditstore.

Returns:

See Also:



257
258
259
260
261
 
# File 'lib/arachni/rpc/server/instance.rb', line 257

def abort_and_report( report_type = :hash, &block )
    @framework.clean_up do
        block.call report_type.to_sym == :auditstore ? auditstore : report
    end
end

#abort_and_report_as(name, &block) ⇒ String

Note:

Don’t forget to #shutdown the instance once you get the report.

Cleans up and delegates to #report_as.

Parameters:

  • name (String) —

    Name of the report component to run, as presented by #list_reports‘s `:shortname` key.

Returns:

See Also:



275
276
277
278
279
 
# File 'lib/arachni/rpc/server/instance.rb', line 275

def abort_and_report_as( name, &block )
    @framework.clean_up do
        block.call report_as( name )
    end
end

#alive? ⇒ true

Returns:

  • (true) 


182
183
184
 
# File 'lib/arachni/rpc/server/instance.rb', line 182

def alive?
    @server.alive?
end

#auditstore ⇒ AuditStore

Returns Scan results.

Returns:

See Also:



283
284
285
 
# File 'lib/arachni/rpc/server/instance.rb', line 283

def auditstore
    @framework.auditstore
end

#busy?(&block) ⇒ Bool

Returns ‘true` if the scan is initializing or running, `false` otherwise. If a scan is started by #scan then this method should be used instead of Framework#busy?.

Returns:

  • (Bool) —

    ‘true` if the scan is initializing or running, `false` otherwise. If a scan is started by #scan then this method should be used instead of Framework#busy?.



190
191
192
193
194
195
196
197
 
# File 'lib/arachni/rpc/server/instance.rb', line 190

def busy?( &block )
    if @scan_initializing
        block.call( true ) if block_given?
        return true
    end

    @framework.busy?( &block )
end

#consumed_pids ⇒ Object 



728
729
730
 
# File 'lib/arachni/rpc/server/instance.rb', line 728

def consumed_pids
    @consumed_pids | [Process.pid]
end

#error_test(str, &block) ⇒ Object 



724
725
726
 
# File 'lib/arachni/rpc/server/instance.rb', line 724

def error_test( str, &block )
    @framework.error_test( str, &block )
end

#errors(starting_line = 0, &block) ⇒ Object 



201
202
203
 
# File 'lib/arachni/rpc/server/instance.rb', line 201

def errors( starting_line = 0, &block )
    @framework.errors( starting_line, &block )
end

#list_modules ⇒ Array<Hash>

Returns Information about all available modules.

Returns:

  • (Array<Hash>) —

    Information about all available modules.



211
212
213
 
# File 'lib/arachni/rpc/server/instance.rb', line 211

def list_modules
    @framework.list_modules
end

#list_platforms ⇒ Array<Hash>

Returns Information about all available platforms.

Returns:

  • (Array<Hash>) —

    Information about all available platforms.



206
207
208
 
# File 'lib/arachni/rpc/server/instance.rb', line 206

def list_platforms
    @framework.list_platforms
end

#list_plugins ⇒ Array<Hash>

Returns Information about all available plugins.

Returns:

  • (Array<Hash>) —

    Information about all available plugins.



216
217
218
 
# File 'lib/arachni/rpc/server/instance.rb', line 216

def list_plugins
    @framework.list_plugins
end

#list_reports ⇒ Array<Hash>

Returns Information about all available reports.

Returns:

  • (Array<Hash>) —

    Information about all available reports.



221
222
223
 
# File 'lib/arachni/rpc/server/instance.rb', line 221

def list_reports
    @framework.list_reports
end

#output(&block) ⇒ Object

Deprecated. 


719
720
721
 
# File 'lib/arachni/rpc/server/instance.rb', line 719

def output( &block )
    @framework.output( &block )
end

#pause(&block) ⇒ Object

Pauses the running scan on a best effort basis.

See Also:



229
230
231
 
# File 'lib/arachni/rpc/server/instance.rb', line 229

def pause( &block )
    @framework.pause( &block )
end

#progress(options = {}, &block) ⇒ Hash

Simplified version of Framework::MultiInstance#progress.

# Recommended usage

Please request from the method only the things you are going to actually
use, otherwise you'll just be wasting bandwidth.
In addition, ask to **not** be served data you already have, like issues
or error messages.

To be kept completely up to date on the progress of a scan (i.e. receive
new issues and error messages asap) in an efficient manner, you will need
to keep track of the issues and error messages you already have and
explicitly tell the method to not send the same data back to you on
subsequent calls.

## Retrieving errors (‘:errors` option) without duplicate data

This is done by telling the method how many error messages you already
have and you will be served the errors from the error-log that are past
that line.
So, if you were to use a loop to get fresh progress data it would look
like so:

  error_cnt = 0
  i = 0
  while sleep 1
      # Test method, triggers an error log...
      instance.service.error_test "BOOM! #{i+=1}"

      # Only request errors we don't already have
      errors = instance.service.progress( with: { errors: error_cnt } )['errors']
      error_cnt += errors.size

      # You will only see new errors
      puts errors.join("\n")
  end

## Retrieving issues without duplicate data

In order to be served only new issues you will need to let the method
know which issues you already have. This is done by providing a list
of {Issue#digest digests} for the issues you already know about.

  issue_digests = []
  while sleep 1
      issues = instance.service.progress(
                   # Ask for native Arachni::Issue object instead of hashes
                   with: :native_issues,
                   # Only request issues we don't already have
                   without: { issues: issue_digests  }
               )['issues']

      issue_digests |= issues.map( &:digest )

      # You will only see new issues
      issues.each do |issue|
          puts "  * #{issue.name} for input '#{issue.var}' at '#{issue.url}'."
      end
  end

_If your client is on a platform that has no access to native Arachni
objects, you'll have to calculate the {Issue#digest digests} yourself._

Parameters:

  • options (Hash) (defaults to: {}) —

    Options about what progress data to retrieve and return.

Options Hash (options):

  • :with (Array<Symbol, Hash>) —

    Specify data to include:

    • :native_issues – Discovered issues as Issue objects.

    • :issues – Discovered issues as hashes.

    • :instances – Statistics and info for slave instances.

    • :errors – Errors and the line offset to use for #errors. Pass as a hash, like: ‘{ errors: 10 }`

  • :without (Array<Symbol, Hash>) —

    Specify data to exclude:

    • :stats – Don’t include runtime statistics.

    • :issues – Don’t include issues with the given digests. Pass as a hash, like: ‘{ issues: […] }`

Returns:

  • (Hash) —

    • ‘stats` – General runtime statistics (merged when part of Grid)

      (enabled by default)

    • ‘status` – #status

    • ‘busy` – #busy?

    • ‘issues` – Framework#issues_as_hash or Framework#issues

      (disabled by default)

    • ‘instances` – Raw `stats` for each running instance (only when part

      of Grid) (disabled by default)

    • ‘errors` – #errors (disabled by default)



401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
 
# File 'lib/arachni/rpc/server/instance.rb', line 401

def progress( options = {}, &block )
    with    = parse_progress_opts( options, :with )
    without = parse_progress_opts( options, :without )

    @framework.progress( as_hash:   !with.include?( :native_issues ),
                         issues:    with.include?( :native_issues ) ||
                                        with.include?( :issues ),
                         stats:     !without.include?( :stats ),
                         slaves:    with.include?( :instances ),
                         messages:  false,
                         errors:    with[:errors]
    ) do |data|
        data['instances'] ||= [] if with.include?( :instances )
        data['busy'] = busy?

        if data['issues']
            data['issues'] = data['issues'].dup

            if without[:issues].is_a? Array
                data['issues'].reject! do |i|
                    without[:issues].include?( i.is_a?( Hash ) ? i['digest'] : i.digest )
                end
            end
        end

        block.call( data )
    end
end

#report ⇒ Hash

Returns Audit results as a hash.

Returns:

See Also:



289
290
291
 
# File 'lib/arachni/rpc/server/instance.rb', line 289

def report
    @framework.report
end

#report_as(name) ⇒ String

Returns Scan report.

Parameters:

  • name (String) —

    Name of the report component to run, as presented by #list_reports‘s `:shortname` key.

Returns:

See Also:



299
300
301
 
# File 'lib/arachni/rpc/server/instance.rb', line 299

def report_as( name )
    @framework.report_as( name )
end

#resume(&block) ⇒ Object

Resumes a paused scan.

See Also:



237
238
239
 
# File 'lib/arachni/rpc/server/instance.rb', line 237

def resume( &block )
    @framework.resume( &block )
end

#scan(opts = {}, &block) ⇒ Object

Note:

If you use this method to start the scan use #busy? instead of Framework#busy? to check if the scan is still running.

Note:

Options marked with an asterisk are required.

Note:

Options which expect patterns will interpret their arguments as regular expressions regardless of their type.

Note:

When using more than one Instance, the ‘http_req_limit` and `link_count_limit` options will be divided by the number of Instance to be used.

Configures and runs a scan.

Parameters:

  • opts (Hash) (defaults to: {}) —

    Scan options to be passed to Options#set (along with some extra ones to keep configuration in one place).

    _The options presented here are the most commonly used ones, in actuality, you can use any Options attribute._

Options Hash (opts):

  • *:url (String) —

    Target URL to audit.

  • :audit_links (Boolean) — default: false —

    Enable auditing of link inputs.

  • :audit_forms (Boolean) — default: false —

    Enable auditing of form inputs.

  • :audit_cookies (Boolean) — default: false —

    Enable auditing of cookie inputs.

  • :audit_headers (Boolean) — default: false —

    Enable auditing of header inputs.

  • :modules (String, Array<String>) — default: [] —

    Modules to load, by name.

    # To load all modules use the wildcard on its own
'*'

# To load all XSS and SQLi modules:
[ 'xss*', 'sqli*' ]
  • :plugins (Hash<Hash>) — default: {} —

    Plugins to load, by name, along with their options.

    {
    'proxy'      => {}, # empty options
    'autologin'  => {
        'url'    => 'http://demo.testfire.net/bank/login.aspx',
        'params' => 'uid=jsmith&passw=Demo1234',
        'check'  => 'MY ACCOUNT'
    },
}
  • :platforms (String, Symbol, Array<String, Symbol>) — default: [] —

    Initialize the fingerprinter with the given platforms. The fingerprinter cannot identify database servers so specifying the remote DB backend will greatly enhance performance and reduce bandwidth consumption.

  • :no_fingerprinting (Integer) — default: false —

    Disable platform fingerprinting and include all payloads in the audit. Use this option in addition to the ‘:platforms` one to restrict the audit payloads to explicitly specified platforms.

  • :link_count_limit (Integer) — default: nil —

    Limit the amount of pages to be crawled and audited.

  • :depth_limit (Integer) — default: nil —

    Directory depth limit.

    How deep Arachni should go into the website structure.

  • :exclude (Array<String, Regexp>) — default: [] —

    URLs that match any of the given patterns will be ignored.

    [ 'logout', /skip.*.me too/i ]
  • :exclude_pages (Array<String, Regexp>) — default: [] —

    Exclude pages from the crawl and audit processes based on their content (i.e. HTTP response bodies).

    [ /.*forbidden.*/, "I'm a weird 404 and I should be ignored" ]
  • :exclude_vectors (Array<String>) — default: [] —

    Exclude input vectors from the audit, by name.

    [ 'sessionid', 'please_dont_audit_me' ]
  • :include (Array<String, Regexp>) — default: [] —

    Only URLs that match any of the given patterns will be followed and audited.

    [ 'only-follow-me', 'follow-me-as-well' ]
  • :redundant (Hash<<String, Regexp>,Integer>) — default: {} —

    Redundancy patterns to limit how many times certain paths should be followed.

    { "follow_me_3_times" => 3, /follow_me_5_times/ => 5 }

    Useful when scanning pages that create an large number of pages like galleries and calendars.

  • :restrict_paths (Array<String>) — default: [] —

    Restrict the audit to the provided paths.

    The crawl phase gets skipped and these paths are used instead.

  • :extend_paths (Array<String>) — default: [] —

    Extend the scope of the crawl and audit with the provided paths.

  • :cookies (Array<String, Hash, Arachni::Element::Cookie>) — default: {} —

    Cookies to use for the HTTP requests.

    [
    'secret=blah',
    {
        'userid' => '1',
    },
    {
        name:      'sessionid',
        value:     'fdfdfDDfsdfszdf',
        http_only: true
    }
]

    _Cookie values will be encoded automatically._

  • :http_req_limit (Integer) — default: 20 —

    HTTP request concurrency limit.

  • :user_agent (String) — default: 'Arachni/v<version>' —

    User agent to use.

  • :authed_by (String) — default: nil —

    The e-mail address of the person who authorized the scan.

    john.doe@bigscanners.com
  • :slaves (Array<Hash>) —

    Info of Instances to enslave.

    [
    { url: 'address:port', token: 's3cr3t' },
    { url: 'anotheraddress:port', token: '3v3nm0r3s3cr3t' }
]
  • :grid (Bool) — default: false —

    Uses the Dispatcher Grid to obtain slave instances for a multi-Instance scan.

    If set to ‘true`, it serves as a shorthand for:

    grid_mode: :balance
  • :grid_mode (String, Symbol) — default: nil —

    Grid mode to use, available modes are:

    • ‘nil` – No grid.

    • ‘:balance` – Slave Instances will be provided by the least burdened

      grid members to keep the overall Grid workload even across all Dispatchers.

    • ‘:aggregate` – Same as `:balance` but with high-level line-aggregation.

      Will only request Instances from Grid members with different Pipe-IDs.
  • :spawns (Integer) — default: 0 —

    The amount of slaves to spawn. The behavior of this option changes depending on the ‘grid_mode` setting:

    • ‘nil` – All slave Instances will be spawned by this Instance directly,

      and thus reside in the same machine. This has the added benefit of
using UNIX-domain sockets for inter-process communication and avoiding
the overhead of TCP/IP.

    • ‘:balance` – Slaves will be provided by the least burdened Grid Dispatchers.

    • ‘:aggregate` – Slaves will be provided by Grid Dispatchers with unique

      Pipe-IDs and the value of this option will be treated as a possible
maximum rather than a hard setting. Actual spawn count will be determined
by Dispatcher availability and the size of the workload.



594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
 
# File 'lib/arachni/rpc/server/instance.rb', line 594

def scan( opts = {}, &block )
    # If the instance isn't clean bail out now.
    if busy? || @called
        block.call false
        return false
    end

    # Normalize this sucker to have symbols as keys -- but not recursively.
    opts = opts.symbolize_keys( false )

    slaves      = opts[:slaves] || []
    spawn_count = opts[:spawns].to_i

    if opts[:platforms]
        begin
            Platform::Manager.new( [opts[:platforms]].flatten.compact )
        rescue => e
            fail ArgumentError, e.to_s
        end
    end

    if opts[:grid_mode]
        @framework.opts.grid_mode = opts[:grid_mode]
    end

    if (opts[:grid] || opts[:grid_mode]) && spawn_count <= 0
        fail ArgumentError,
             'Option \'spawns\' must be greater than 1 for Grid scans.'
    end

    if ((opts[:grid] || opts[:grid_mode]) || spawn_count > 0) && [opts[:restrict_paths]].flatten.compact.any?
        fail ArgumentError,
             'Option \'restrict_paths\' is not supported when in multi-Instance mode.'
    end

    # There may be follow-up/retry calls by the client in cases of network
    # errors (after the request has reached us) so we need to keep minimal
    # track of state in order to bail out on subsequent calls.
    @called = @scan_initializing = true

    # Plugins option needs to be a hash...
    if opts[:plugins] && opts[:plugins].is_a?( Array )
        opts[:plugins] = opts[:plugins].inject( {} ) { |h, n| h[n] = {}; h }
    end

    @active_options.set( opts )

    if @framework.opts.url.to_s.empty?
        fail ArgumentError, 'Option \'url\' is mandatory.'
    end

    # Undocumented option, used internally to distribute workload and knowledge
    # for multi-Instance scans.
    if opts[:multi]
        @framework.update_page_queue( opts[:multi][:pages] || [] )
        @framework.restrict_to_elements( opts[:multi][:elements] || [] )

        if Options.fingerprint?
            Platform::Manager.update_light( opts[:multi][:platforms] || {} )
        end
    end

    opts[:modules] ||= opts[:mods]
    @framework.modules.load opts[:modules] if opts[:modules]
    @framework.plugins.load opts[:plugins] if opts[:plugins]

    # Starts the scan after all necessary options have been set.
    after = proc { block.call @framework.run; @scan_initializing = false }

    if @framework.opts.grid?
        # If a Grid scan has been selected then just set us as the master
        # and set the spawn count as max slaves.
        #
        # The Framework will sort out the rest...
        @framework.set_as_master
        @framework.opts.max_slaves = spawn_count

        # Rock n' roll!
        after.call
    else
        # Handles each spawn, enslaving it for a multi-Instance scan.
        each  = proc do |slave, iter|
            @framework.enslave( slave ){ iter.next }
        end

        spawn( spawn_count ) do |spawns|
            # Add our spawns to the slaves list which was passed as an option.
            slaves |= spawns

            # Process the Instances.
            ::EM::Iterator.new( slaves, slaves.empty? ? 1 : slaves.size ).
                each( each, after )
        end
    end

    true
end

#shutdown(&block) ⇒ Object

Makes the server go bye-bye…Lights out!



693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
 
# File 'lib/arachni/rpc/server/instance.rb', line 693

def shutdown( &block )
    print_status 'Shutting down...'

    ::EM.defer do
        t = []
        @framework.instance_eval do
            next if !has_slaves?
            @instances.each do |instance|
                # Don't know why but this works better than EM's stuff
                t << Thread.new { connect_to_instance( instance ).service.shutdown }
            end
        end
        t.join

        @server.shutdown

        block.call true if block_given?
    end

    true
end

#status ⇒ String

Returns Status of the instance, possible values are (in order):

  • ‘ready` – Initialised and waiting for instructions.

  • ‘preparing` – Getting ready to start (i.e. initing plugins etc.).

  • ‘crawling` – The instance is crawling the target webapp.

  • ‘auditing` – The instance is currently auditing the webapp.

  • ‘paused` – The instance has been paused (if applicable).

  • ‘cleanup` – The scan has completed and the instance is cleaning up

    after itself (i.e. waiting for plugins to finish etc.).

  • ‘done` – The scan has completed, you can grab the report and shutdown.

Returns:

  • (String) —

    Status of the instance, possible values are (in order):

    • ‘ready` – Initialised and waiting for instructions.

    • ‘preparing` – Getting ready to start (i.e. initing plugins etc.).

    • ‘crawling` – The instance is crawling the target webapp.

    • ‘auditing` – The instance is currently auditing the webapp.

    • ‘paused` – The instance has been paused (if applicable).

    • ‘cleanup` – The scan has completed and the instance is cleaning up

      after itself (i.e. waiting for plugins to finish etc.).

    • ‘done` – The scan has completed, you can grab the report and shutdown.

See Also:



305
306
307
 
# File 'lib/arachni/rpc/server/instance.rb', line 305

def status
    @framework.status
end