Class: Arachni::Framework

Inherits:
Object show all
Includes:
Mixins::Observable, UI::Output, Utilities
Defined in:
lib/arachni/framework.rb

Overview

The Framework class ties together all the components.

It’s the brains of the operation, it bosses the rest of the classes around. It runs the audit, loads modules and reports and runs them according to user options.

Author:

Direct Known Subclasses

RPC::Server::Framework

Defined Under Namespace

Classes: Error

Constant Summary collapse

REVISION =

The version of this class.

'0.2.8'
AUDIT_PAGE_MAX_TRIES =

How many times to request a page upon failure.

5

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Mixins::Observable

#clear_observers, #method_missing

Methods included from Utilities

#available_port, #cookie_encode, #cookies_from_document, #cookies_from_file, #cookies_from_response, #exception_jail, #exclude_path?, #extract_domain, #follow_protocol?, #form_decode, #form_encode, #form_parse_request_body, #forms_from_document, #forms_from_response, #generate_token, #get_path, #html_decode, #html_encode, #include_path?, #links_from_document, #links_from_response, #normalize_url, #page_from_response, #page_from_url, #parse_query, #parse_set_cookie, #parse_url_vars, #path_in_domain?, #path_too_deep?, #port_available?, #rand_port, #redundant_path?, #remove_constants, #seed, #skip_page?, #skip_path?, #skip_resource?, #to_absolute, #uri_decode, #uri_encode, #uri_parse, #uri_parser, #url_sanitize

Methods included from UI::Output

#debug?, #debug_off, #debug_on, #disable_only_positives, #error_logfile, #flush_buffer, #log_error, #mute, #muted?, old_reset_output_options, #only_positives, #only_positives?, #print_bad, #print_debug, #print_debug_backtrace, #print_debug_pp, #print_error, #print_error_backtrace, #print_info, #print_line, #print_ok, #print_status, #print_verbose, #reroute_to_file, #reroute_to_file?, reset_output_options, #set_buffer_cap, #set_error_logfile, #uncap_buffer, #unmute, #verbose, #verbose?

Constructor Details

#initialize(opts = Arachni::Options.instance, &block) ⇒ Framework

Returns a new instance of Framework.

Parameters:



133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/arachni/framework.rb', line 133

def initialize( opts = Arachni::Options.instance, &block )

    Encoding.default_external = 'BINARY'
    Encoding.default_internal = 'BINARY'

    @opts = opts

    @modules = Module::Manager.new( self )
    @reports = Report::Manager.new( @opts )
    @plugins = Plugin::Manager.new( self )

    @session = Session.new( @opts )
    reset_spider
    @http    = HTTP.instance

    reset_trainer

    # will store full-fledged pages generated by the Trainer since these
    # may not be be accessible simply by their URL
    @page_queue = Support::Database::Queue.new
    @page_queue_total_size = 0

    # will hold paths found by the spider in order to be converted to pages
    # and ultimately audited by the modules
    @url_queue = Queue.new
    @url_queue_total_size = 0

    # deep clone the redundancy rules to preserve their counter
    # for the reports
    @orig_redundant = @opts.redundant.deep_clone

    @running = false
    @status  = :ready
    @paused  = []

    @auditmap = []
    @sitemap  = []

    @current_url = ''

    # Holds page URLs which returned no response.
    @failures = []
    @retries  = {}

    if block_given?
        block.call self
        reset
    end
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method in the class Arachni::Mixins::Observable

Instance Attribute Details

#failures ⇒ Array<String> (readonly)

Returns Page URLs which elicited no response from the server and were not audited. Not determined by HTTP status codes, we’re talking network failures here.

Returns:

  • (Array<String>) —

    Page URLs which elicited no response from the server and were not audited. Not determined by HTTP status codes, we’re talking network failures here.



126
127
128
 
# File 'lib/arachni/framework.rb', line 126

def failures
  @failures
end

#http ⇒ Arachni::HTTP (readonly)

Returns:



109
110
111
 
# File 'lib/arachni/framework.rb', line 109

def http
  @http
end

#modules ⇒ Arachni::Module::Manager (readonly)

Returns:



97
98
99
 
# File 'lib/arachni/framework.rb', line 97

def modules
  @modules
end

#opts ⇒ Options (readonly)

Returns Instance options.

Returns:



91
92
93
 
# File 'lib/arachni/framework.rb', line 91

def opts
  @opts
end

#page_queue_total_size ⇒ Integer (readonly)

Returns Total number of pages added to their audit queue.

Returns:

  • (Integer) —

    Total number of pages added to their audit queue.



118
119
120
 
# File 'lib/arachni/framework.rb', line 118

def page_queue_total_size
  @page_queue_total_size
end

#plugins ⇒ Arachni::Plugin::Manager (readonly)

Returns:



100
101
102
 
# File 'lib/arachni/framework.rb', line 100

def plugins
  @plugins
end

#reports ⇒ Arachni::Report::Manager (readonly)

Returns:



94
95
96
 
# File 'lib/arachni/framework.rb', line 94

def reports
  @reports
end

#session ⇒ Session (readonly)

Returns Web application session manager.

Returns:

  • (Session) —

    Web application session manager.



103
104
105
 
# File 'lib/arachni/framework.rb', line 103

def session
  @session
end

#sitemap ⇒ Array (readonly)

Returns URLs of all discovered pages.

Returns:

  • (Array) —

    URLs of all discovered pages.



112
113
114
 
# File 'lib/arachni/framework.rb', line 112

def sitemap
  @sitemap
end

#spider ⇒ Spider (readonly)

Returns Web application spider.

Returns:

  • (Spider) —

    Web application spider.



106
107
108
 
# File 'lib/arachni/framework.rb', line 106

def spider
  @spider
end

#trainer ⇒ Trainer (readonly)

Returns:



115
116
117
 
# File 'lib/arachni/framework.rb', line 115

def trainer
  @trainer
end

#url_queue_total_size ⇒ Integer (readonly)

Returns Total number of urls added to their audit queue.

Returns:

  • (Integer) —

    Total number of urls added to their audit queue.



121
122
123
 
# File 'lib/arachni/framework.rb', line 121

def url_queue_total_size
  @url_queue_total_size
end

Class Method Details

.reset ⇒ Object

Resets everything and allows the framework to be re-used.

You should first update Options.



668
669
670
671
672
673
674
675
676
677
678
 
# File 'lib/arachni/framework.rb', line 668

def self.reset
    UI::Output.reset_output_options
    Platform::Manager.reset
    Module::Auditor.reset
    ElementFilter.reset
    Element::Capabilities::Auditable.reset
    Module::Manager.reset
    Plugin::Manager.reset
    Report::Manager.reset
    HTTP.reset
end

Instance Method Details

#audit_page(page) ⇒ Object

Runs loaded modules against a given ‘page`

It will audit just the given page and not use the Trainer – i.e. ignore any new elements that might appear as a result.

Parameters:



214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
 
# File 'lib/arachni/framework.rb', line 214

def audit_page( page )
    return if !page

    if skip_page? page
        print_info "Ignoring page due to exclusion criteria: #{page.url}"
        return false
    end

    @auditmap << page.url
    @sitemap |= @auditmap
    @sitemap.uniq!

    print_line
    print_status "Auditing: [HTTP: #{page.code}] #{page.url}"

    if page.platforms.any?
        print_info "Identified as: #{page.platforms.to_a.join( ', ' )}"
    end

    call_on_audit_page( page )

    @current_url = page.url.to_s

    @modules.schedule.each do |mod|
        wait_if_paused
        run_module_against_page( mod, page )
    end

    harvest_http_responses

    if !Module::Auditor.timeout_candidates.empty?
        print_line
        print_status "Verifying timeout-analysis candidates for: #{page.url}"
        print_info '---------------------------------------'
        Module::Auditor.timeout_audit_run
    end

    true
end

#audit_store ⇒ AuditStore Also known as: auditstore

Returns Scan results.

Returns:

See Also:



400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
 
# File 'lib/arachni/framework.rb', line 400

def audit_store
    opts = @opts.to_hash.deep_clone

    # restore the original redundancy rules and their counters
    opts['redundant'] = @orig_redundant
    opts['mods'] = @modules.keys

    AuditStore.new(
        version:  version,
        revision: revision,
        options:  opts,
        sitemap:  (auditstore_sitemap || []).sort,
        issues:   @modules.results,
        plugins:  @plugins.results
    )
end

#clean_up ⇒ Object

Cleans up the framework; should be called after running the audit or after canceling a running scan.

It stops the clock and waits for the plugins to finish up.



607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
 
# File 'lib/arachni/framework.rb', line 607

def clean_up
    @status = :cleanup

    @page_queue.clear

    @opts.finish_datetime  = Time.now
    @opts.start_datetime ||= Time.now

    @opts.delta_time = @opts.finish_datetime - @opts.start_datetime

    # make sure this is disabled or it'll break report output
    disable_only_positives

    @running = false

    # wait for the plugins to finish
    @plugins.block

    true
end

#link_count_limit_reached? ⇒ Bool

Returns ‘true` if the Options#link_count_limit has been reached, `false` otherwise.

Returns:



262
263
264
 
# File 'lib/arachni/framework.rb', line 262

def link_count_limit_reached?
    @opts.link_count_limit_reached? @sitemap.size
end

#list_modules ⇒ Array<Hash> Also known as: lsmod

Returns Information about all available modules.

Returns:

  • (Array<Hash>) —

    Information about all available modules.



463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
 
# File 'lib/arachni/framework.rb', line 463

def list_modules
    loaded = @modules.loaded

    begin
        @modules.clear
        @modules.available.map do |name|
            path = @modules.name_to_path( name )
            next if !lsmod_match?( path )

            @modules[name].info.merge(
                mod_name:  name,
                shortname: name,
                author:    [@modules[name].info[:author]].
                               flatten.map { |a| a.strip },
                path:      path.strip
            )
        end.compact
    ensure
        @modules.clear
        @modules.load loaded
    end
end

#list_platforms ⇒ Array<Hash> Also known as: lsplat

Returns Information about all available platforms.

Returns:

  • (Array<Hash>) —

    Information about all available platforms.



538
539
540
541
542
543
544
545
546
 
# File 'lib/arachni/framework.rb', line 538

def list_platforms
    platforms = Platform::Manager.new
    platforms.valid.inject({}) do |h, platform|
        type = Platform::Manager::TYPES[platforms.find_type( platform )]
        h[type] ||= {}
        h[type][platform] = platforms.fullname( platform )
        h
    end
end

#list_plugins ⇒ Array<Hash> Also known as: lsplug

Returns Information about all available plugins.

Returns:

  • (Array<Hash>) —

    Information about all available plugins.



513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
 
# File 'lib/arachni/framework.rb', line 513

def list_plugins
    loaded = @plugins.loaded

    begin
        @plugins.clear
        @plugins.available.map do |plugin|
            path = @plugins.name_to_path( plugin )
            next if !lsplug_match?( path )

            @plugins[plugin].info.merge(
                plug_name: plugin,
                shortname: plugin,
                path:      path,
                author:    [@plugins[plugin].info[:author]].
                               flatten.map { |a| a.strip }
            )
        end.compact
    ensure
        @plugins.clear
        @plugins.load loaded
    end
end

#list_reports ⇒ Array<Hash> Also known as: lsrep

Returns Information about all available reports.

Returns:

  • (Array<Hash>) —

    Information about all available reports.



488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
 
# File 'lib/arachni/framework.rb', line 488

def list_reports
    loaded = @reports.loaded

    begin
        @reports.clear
        @reports.available.map do |report|
            path = @reports.name_to_path( report )
            next if !lsrep_match?( path )

            @reports[report].info.merge(
                rep_name:  report,
                shortname: report,
                path:      path,
                author:    [@reports[report].info[:author]].
                               flatten.map { |a| a.strip }
            )
        end.compact
    ensure
        @reports.clear
        @reports.load loaded
    end
end

#on_audit_page(&block) ⇒ Object Also known as: on_run_mods 



254
255
256
 
# File 'lib/arachni/framework.rb', line 254

def on_audit_page( &block )
    add_on_audit_page( &block )
end

#pause ⇒ TrueClass

Returns Pauses the framework on a best effort basis, might take a while to take effect.

Returns:

  • (TrueClass) —

    Pauses the framework on a best effort basis, might take a while to take effect.



578
579
580
581
582
 
# File 'lib/arachni/framework.rb', line 578

def pause
    spider.pause
    @paused << caller
    true
end

#paused? ⇒ Bool

Returns ‘true` if the framework is paused or in the process of.

Returns:

  • (Bool) —

    ‘true` if the framework is paused or in the process of.



572
573
574
 
# File 'lib/arachni/framework.rb', line 572

def paused?
    !@paused.empty?
end

#push_to_page_queue(page) ⇒ Bool

Pushes a page to the page audit queue and updates #page_queue_total_size

Parameters:

Returns:

  • (Bool) —

    ‘true` if push was successful, `false` if the `page` matched any exclusion criteria.



364
365
366
367
368
369
370
371
372
 
# File 'lib/arachni/framework.rb', line 364

def push_to_page_queue( page )
    return false if skip_page? page

    @page_queue << page
    @page_queue_total_size += 1

    @sitemap |= [page.url]
    true
end

#push_to_url_queue(url) ⇒ Bool

Pushes a URL to the URL audit queue and updates #url_queue_total_size

Parameters:

Returns:

  • (Bool) —

    ‘true` if push was successful, `false` if the `url` matched any exclusion criteria.



383
384
385
386
387
388
389
390
391
392
393
 
# File 'lib/arachni/framework.rb', line 383

def push_to_url_queue( url )
    return false if skip_path? url

    abs = to_absolute( url )

    @url_queue.push( abs ? abs : url )
    @url_queue_total_size += 1

    @sitemap |= [url]
    false
end

#report_as(name, external_report = auditstore) ⇒ String

Runs a report component and returns the contents of the generated report.

Only accepts reports which support an ‘outfile` option.

Parameters:

  • name (String) —

    Name of the report component to run, as presented by #list_reports‘s `:shortname` key.

  • external_report (AuditStore) (defaults to: auditstore) —

    Report to use – defaults to the local one.

Returns:

Raises:



437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
 
# File 'lib/arachni/framework.rb', line 437

def report_as( name, external_report = auditstore )
    if !@reports.available.include?( name.to_s )
        fail Component::Error::NotFound, "Report '#{name}' could not be found."
    end

    loaded = @reports.loaded
    begin
        @reports.clear

        if !@reports[name].has_outfile?
            fail Component::Options::Error::Invalid,
                 "Report '#{name}' cannot format the audit results as a String."
        end

        outfile = "/#{Dir.tmpdir}/arachn_report_as.#{name}"
        @reports.run_one( name, external_report, 'outfile' => outfile )

        IO.read( outfile )
    ensure
        File.delete( outfile ) if outfile
        @reports.clear
        @reports.load loaded
    end
end

#reset ⇒ Object

Resets everything and allows the framework to be re-used.

You should first update Options.

Prefer this if you already have an instance.



643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
 
# File 'lib/arachni/framework.rb', line 643

def reset
    @page_queue_total_size = 0
    @url_queue_total_size  = 0
    @failures.clear
    @retries.clear
    @sitemap.clear
    @page_queue.clear

    # this needs to be first so that the HTTP lib will be reset before
    # the rest
    self.class.reset

    clear_observers
    reset_trainer
    reset_spider
    @modules.clear
    @reports.clear
    @plugins.clear
end

#reset_spider ⇒ Object 



628
629
630
 
# File 'lib/arachni/framework.rb', line 628

def reset_spider
    @spider = Spider.new( @opts )
end

#reset_trainer ⇒ Object 



632
633
634
 
# File 'lib/arachni/framework.rb', line 632

def reset_trainer
    @trainer = Trainer.new( self )
end

#resume ⇒ TrueClass

Returns Resumes the scan/audit.

Returns:

  • (TrueClass) —

    Resumes the scan/audit.



585
586
587
588
589
 
# File 'lib/arachni/framework.rb', line 585

def resume
    @paused.delete( caller )
    spider.resume
    true
end

#revision ⇒ String

Returns the revision of the Arachni::Framework (this) class.

Returns:



597
598
599
 
# File 'lib/arachni/framework.rb', line 597

def revision
    REVISION
end

#run(&block) ⇒ Object

Starts the scan.

Parameters:

  • block (Block) —

    A block to call after the audit has finished but before running the reports.



189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
 
# File 'lib/arachni/framework.rb', line 189

def run( &block )
    prepare

    # catch exceptions so that if something breaks down or the user opted to
    # exit the reports will still run with whatever results Arachni managed to gather
    exception_jail( false ){ audit }

    clean_up
    exception_jail( false ){ block.call } if block_given?
    @status = :done

    # run reports
    @reports.run( audit_store ) if !@reports.empty?

    true
end

#running? ⇒ Bool

Returns ‘true` if the framework is running, `false` otherwise.

Returns:

  • (Bool) —

    ‘true` if the framework is running, `false` otherwise.



567
568
569
 
# File 'lib/arachni/framework.rb', line 567

def running?
    @running
end

#stats(refresh_time = false, override_refresh = false) ⇒ Hash

Returns the following framework stats:

  • ‘:requests` – HTTP request count

  • ‘:responses` – HTTP response count

  • ‘:time_out_count` – Amount of timed-out requests

  • ‘:time` – Amount of running time

  • ‘:avg` – Average requests per second

  • ‘:sitemap_size` – Number of discovered pages

  • ‘:auditmap_size` – Number of audited pages

  • ‘:progress` – Progress percentage

  • ‘:curr_res_time` – Average response time for the current burst of requests

  • ‘:curr_res_cnt` – Amount of responses for the current burst

  • ‘:curr_avg` – Average requests per second for the current burst

  • ‘:average_res_time` – Average response time

  • ‘:max_concurrency` – Current maximum concurrency of HTTP requests

  • ‘:current_page` – URL of the currently audited page

  • ‘:eta` – Estimated time of arrival i.e. estimated remaining time

Parameters:

  • refresh_time (Bool) (defaults to: false) —

    updates the running time of the audit (usefully when you want stats while paused without messing with the clocks)

  • override_refresh (Bool) (defaults to: false)

Returns:



292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
 
# File 'lib/arachni/framework.rb', line 292

def stats( refresh_time = false, override_refresh = false )
    req_cnt = http.request_count
    res_cnt = http.response_count

    @opts.start_datetime = Time.now if !@opts.start_datetime

    sitemap_sz  = @sitemap.size
    auditmap_sz = @auditmap.size

    if( !refresh_time || auditmap_sz == sitemap_sz ) && !override_refresh
        @opts.delta_time ||= Time.now - @opts.start_datetime
    else
        @opts.delta_time = Time.now - @opts.start_datetime
    end

    avg = 0
    avg = (res_cnt / @opts.delta_time).to_i if res_cnt > 0

    # We need to remove URLs that lead to redirects from the sitemap
    # when calculating the progress %.
    #
    # This is because even though these URLs are valid webapp paths
    # they are not actual pages and thus can't be audited;
    # so the sitemap and auditmap will never match and the progress will
    # never get to 100% which may confuse users.
    #
    sitemap_sz -= spider.redirects.size
    sitemap_sz = 0 if sitemap_sz < 0

    # Progress of audit is calculated as:
    #     amount of audited pages / amount of all discovered pages
    progress = (Float( auditmap_sz ) / sitemap_sz) * 100

    progress = Float( sprintf( '%.2f', progress ) ) rescue 0.0

    # Sometimes progress may slightly exceed 100% which can cause a few
    # strange stuff to happen.
    progress = 100.0 if progress > 100.0

    # Make sure to keep weirdness at bay.
    progress = 0.0 if progress < 0.0

    pb = Mixins::ProgressBar.eta( progress, @opts.start_datetime )

    {
        requests:         req_cnt,
        responses:        res_cnt,
        time_out_count:   http.time_out_count,
        time:             audit_store.delta_time,
        avg:              avg,
        sitemap_size:     auditstore_sitemap.size,
        auditmap_size:    auditmap_sz,
        progress:         progress,
        curr_res_time:    http.curr_res_time,
        curr_res_cnt:     http.curr_res_cnt,
        curr_avg:         http.curr_res_per_second,
        average_res_time: http.average_res_time,
        max_concurrency:  http.max_concurrency,
        current_page:     @current_url,
        eta:              pb
    }
end

#status ⇒ String

Returns Status of the instance, possible values are (in order):

  • ‘ready` – Initialised and waiting for instructions.

  • ‘preparing` – Getting ready to start (i.e. initing plugins etc.).

  • ‘crawling` – The instance is crawling the target webapp.

  • ‘auditing` – The instance is currently auditing the webapp.

  • ‘paused` – The instance has been paused (if applicable).

  • ‘cleanup` – The scan has completed and the instance is cleaning up

    after itself (i.e. waiting for plugins to finish etc.).

  • ‘done` – The scan has completed, you can grab the report and shutdown.

Returns:

  • (String) —

    Status of the instance, possible values are (in order):

    • ‘ready` – Initialised and waiting for instructions.

    • ‘preparing` – Getting ready to start (i.e. initing plugins etc.).

    • ‘crawling` – The instance is crawling the target webapp.

    • ‘auditing` – The instance is currently auditing the webapp.

    • ‘paused` – The instance has been paused (if applicable).

    • ‘cleanup` – The scan has completed and the instance is cleaning up

      after itself (i.e. waiting for plugins to finish etc.).

    • ‘done` – The scan has completed, you can grab the report and shutdown.



561
562
563
564
 
# File 'lib/arachni/framework.rb', line 561

def status
    return 'paused' if paused?
    @status.to_s
end

#version ⇒ String

Returns the version of the framework.

Returns:

  • (String) —

    Returns the version of the framework.



592
593
594
 
# File 'lib/arachni/framework.rb', line 592

def version
    Arachni::VERSION
end