Class: Autobuild::Importer

Inherits:
Object
  • Object
show all
Defined in:
lib/autobuild/importer.rb

Direct Known Subclasses

ArchiveImporter, CVSImporter, DarcsImporter, Git, Hg, SVN

Defined Under Namespace

Classes: Hook, Status

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options) ⇒ Importer

Creates a new Importer object. The options known to Importer are:

:patches

a list of patch to apply after import

More options are specific to each importer type.



149
150
151
152
153
154
155
156
 
# File 'lib/autobuild/importer.rb', line 149

def initialize(options)
    @options = options.dup
    @options[:retry_count] = Integer(@options[:retry_count] || 0)
    @repository_id = options[:repository_id] || "#{self.class.name}:#{object_id}"
    @interactive = options[:interactive]
    @source_id = options[:source_id] || @repository_id
    @post_hooks = Array.new
end

Class Attribute Details

.fallback_handlers ⇒ Object (readonly)

The set of handlers registered by Importer.fallback



27
28
29
 
# File 'lib/autobuild/importer.rb', line 27

def fallback_handlers
  @fallback_handlers
end

Instance Attribute Details

#interactive=(value) ⇒ Object (writeonly)

Changes whether this importer is interactive or not



187
188
189
 
# File 'lib/autobuild/importer.rb', line 187

def interactive=(value)
  @interactive = value
end

#options ⇒ Hash (readonly)

Returns the original option hash as given to #initialize.

Returns:

  • (Hash) —

    the original option hash as given to #initialize



143
144
145
 
# File 'lib/autobuild/importer.rb', line 143

def options
  @options
end

#post_hooks ⇒ Object (readonly)

A list of hooks that are called after a successful checkout or update

They are added either at the instance level with #add_post_hook or globally for all importers of a given type with add_post_hook



277
278
279
 
# File 'lib/autobuild/importer.rb', line 277

def post_hooks
  @post_hooks
end

#repository_id ⇒ String (readonly)

Returns a string that identifies the remote repository uniquely

This can be used to check whether two importers are pointing to the same repository, regardless of e.g. the access protocol used. For instance, two git importers that point to the same repository but different branches would have the same repository_id but different source_id

Returns:

  • (String)

See Also:



167
168
169
 
# File 'lib/autobuild/importer.rb', line 167

def repository_id
  @repository_id
end

#source_id ⇒ String (readonly)

Returns a string that identifies the remote source uniquely

This can be used to check whether two importers are pointing to the same code base inside the same repository. For instance, two git importers that point to the same repository but different branches would have the same repository_id but different source_id

Returns:

  • (String)

See Also:



178
179
180
 
# File 'lib/autobuild/importer.rb', line 178

def source_id
  @source_id
end

Class Method Details

.add_post_hook(always: false) {|importer, package| ... } ⇒ Object

Define a post-import hook for all instances of this class

Yield Parameters:

  • importer (Importer) —

    the importer that finished

  • package (Package) —

    the package we’re acting on

See Also:



286
287
288
289
290
 
# File 'lib/autobuild/importer.rb', line 286

def self.add_post_hook(always: false, &hook)
    @post_hooks ||= Array.new
    @post_hooks << Hook.new(always, hook)
    nil
end

.cache_dirs(type) ⇒ nil, Array<String>

The cache directories for the given importer type.

This is used by some importers to save disk space and/or avoid downloading the same things over and over again

The default global cache directory is initialized from the AUTOBUILD_CACHE_DIR environment variable. Per-importer cache directories can be overriden by setting AUTOBUILD_TYPE_CACHE_DIR (e.g. AUTOBUILD_GIT_CACHE_DIR)

The following importers use caches:

  • the archive importer saves downloaded files in the cache. They are saved under an archives/ subdirectory of the default cache if set, or to the value of AUTOBUILD_ARCHIVES_CACHE_DIR

  • the git importer uses the cache directories as alternates for the git checkouts

Parameters:

  • type (String) —

    the importer type. If set, it Given a root cache directory X, and importer specific cache is setup as a subdirectory of X with e.g. X/git or X/archives. The subdirectory name is defined by this argument

Returns:

  • (nil, Array<String>)

See Also:



97
98
99
100
101
102
103
 
# File 'lib/autobuild/importer.rb', line 97

def self.cache_dirs(type)
    if @cache_dirs[type] || (env = ENV["AUTOBUILD_#{type.upcase}_CACHE_DIR"])
        @cache_dirs[type] ||= env.split(":")
    elsif (dirs = default_cache_dirs)
        dirs.map { |d| File.join(d, type) }
    end
end

.default_cache_dirs ⇒ Array<String>?

Returns the default cache directory if there is one

Returns:

  • (Array<String>, nil)

See Also:



109
110
111
112
113
114
115
 
# File 'lib/autobuild/importer.rb', line 109

def self.default_cache_dirs
    if @default_cache_dirs
        @default_cache_dirs
    elsif (from_env = ENV['AUTOBUILD_CACHE_DIR'])
        @default_cache_dirs = [from_env]
    end
end

.default_cache_dirs=(dirs) ⇒ Object

Sets the default cache directory

Parameters:

  • the (Array<String>, String) —

    directories

See Also:



130
131
132
 
# File 'lib/autobuild/importer.rb', line 130

def self.default_cache_dirs=(dirs)
    @default_cache_dirs = Array(dirs)
end

.each_post_hook(error: false) ⇒ Object

Enumerate the post-import hooks defined for all instances of this class



293
294
295
296
297
298
299
 
# File 'lib/autobuild/importer.rb', line 293

def self.each_post_hook(error: false)
    return enum_for(__method__) unless block_given?

    (@post_hooks ||= Array.new).each do |hook|
        yield(hook.callback) if hook.always || !error
    end
end

.fallback(&block) ⇒ Object

call-seq:

Autobuild::Importer.fallback { |package, importer| ... }

If called, registers the given block as a fallback mechanism for failing imports.

Fallbacks are tried in reverse order with the failing importer object as argument. The first valid importer object that has been returned will be used instead.

It is the responsibility of the fallback handler to make sure that it does not do infinite recursions and stuff like that.



21
22
23
 
# File 'lib/autobuild/importer.rb', line 21

def self.fallback(&block)
    @fallback_handlers.unshift(block)
end

.set_cache_dirs(type, *dirs) ⇒ Object

Sets the cache directory for a given importer type

Parameters:

  • type (String) —

    the importer type

  • dir (String) —

    the cache directory

See Also:



122
123
124
 
# File 'lib/autobuild/importer.rb', line 122

def self.set_cache_dirs(type, *dirs)
    @cache_dirs[type] = dirs
end

.unset_cache_dirs ⇒ Object

Unset all cache directories



135
136
137
138
 
# File 'lib/autobuild/importer.rb', line 135

def self.unset_cache_dirs
    @cache_dirs = Hash.new
    @default_cache_dirs = nil
end

Instance Method Details

#add_post_hook(always: false) {|importer, package| ... } ⇒ Object

Add a block that should be called when the import has successfully finished

Yield Parameters:

  • importer (Importer) —

    the importer that finished

  • package (Package) —

    the package we’re acting on

See Also:



316
317
318
 
# File 'lib/autobuild/importer.rb', line 316

def add_post_hook(always: false, &hook)
    post_hooks << Hook.new(always, hook)
end

#apply(package, path, patch_level = 0) ⇒ Object 



531
532
533
 
# File 'lib/autobuild/importer.rb', line 531

def apply(package, path, patch_level = 0)
    call_patch(package, false, path, patch_level)
end

#call_patch(package, reverse, file, patch_level) ⇒ Object 



525
526
527
528
529
 
# File 'lib/autobuild/importer.rb', line 525

def call_patch(package, reverse, file, patch_level)
    package.run(:patch, Autobuild.tool('patch'),
                "-p#{patch_level}", (reverse ? '-R' : nil), '--forward',
                input: file, working_directory: package.importdir)
end

#currently_applied_patches(package) ⇒ Object 



553
554
555
556
557
558
559
560
561
562
563
564
565
566
 
# File 'lib/autobuild/importer.rb', line 553

def currently_applied_patches(package)
    patches_file = patchlist(package)
    return parse_patch_list(package, patches_file) if File.exist?(patches_file)

    patches_file = File.join(package.importdir, "patches-autobuild-stamp")
    if File.exist?(patches_file)
        cur_patches = parse_patch_list(package, patches_file)
        save_patch_state(package, cur_patches)
        FileUtils.rm_f patches_file
        return currently_applied_patches(package)
    end

    []
end

#each_post_hook(error: false, &block) ⇒ Object

Enumerate the post-import hooks for this importer



321
322
323
324
325
326
327
328
329
 
# File 'lib/autobuild/importer.rb', line 321

def each_post_hook(error: false, &block)
    return enum_for(__method__, error: false) unless block_given?

    self.class.each_post_hook(error: error, &block)

    post_hooks.each do |hook|
        yield(hook.callback) if hook.always || !error
    end
end

#execute_post_hooks(package, error: false) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Call the post-import hooks added with #add_post_hook



304
305
306
307
308
 
# File 'lib/autobuild/importer.rb', line 304

def execute_post_hooks(package, error: false)
    each_post_hook(error: error) do |block|
        block.call(self, package)
    end
end

#fallback(error, package, *args, &block) ⇒ Object

Tries to find a fallback importer because of the given error.



501
502
503
504
505
506
507
508
509
510
511
512
513
 
# File 'lib/autobuild/importer.rb', line 501

def fallback(error, package, *args, &block)
    Importer.fallback_handlers.each do |handler|
        fallback_importer = handler.call(package, self)
        if fallback_importer.kind_of?(Importer)
            begin
                return fallback_importer.send(*args, &block)
            rescue Exception
                raise error
            end
        end
    end
    raise error
end

#fingerprint(package) ⇒ Object

Returns a unique hash representing the state of the imported package as a whole unit, including its dependencies and patches



200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/autobuild/importer.rb', line 200

def fingerprint(package)
    vcs_fingerprint_string = vcs_fingerprint(package)
    return unless vcs_fingerprint_string

    patches_fingerprint_string = patches_fingerprint(package)
    if patches_fingerprint_string
        Digest::SHA1.hexdigest(vcs_fingerprint_string +
                               patches_fingerprint_string)
    elsif patches.empty?
        vcs_fingerprint_string
    end
end

#import(package, *old_boolean, ignore_errors: false, checkout_only: false, allow_interactive: true, **options) ⇒ Object

Imports the given package

The importer will checkout or update code in package.importdir. No update will be done if update? returns false.

Parameters:

  • options (Hash) —

    a customizable set of options

Options Hash (**options):

  • :checkout_only (Boolean) — default: false —

    if true, the importer will not update an already checked-out package.

  • :only_local (Boolean) — default: false —

    if true, will only perform actions that do not require network access. Importers that do not support this mode will simply do nothing

  • :reset (Boolean) — default: false —

    if true, the importer’s configuration is interpreted as a hard state in which it should put the working copy. Otherwise, it tries to update the local repository with the remote information. For instance, a git importer for which a commit ID is given will, in this mode, reset the repository to the requested ID (if that does not involve losing commits). Otherwise, it will only ensure that the requested commit ID is present in the current HEAD.



463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
 
# File 'lib/autobuild/importer.rb', line 463

def import( # rubocop:disable Metrics/ParameterLists
    package, *old_boolean,
    ignore_errors: false, checkout_only: false, allow_interactive: true, **options
)
    # Backward compatibility
    unless old_boolean.empty?
        old_boolean = old_boolean.first
        Autoproj.warn "calling #import with a boolean as second argument "\
            "is deprecated, switch to the named argument interface instead"
        Autoproj.warn "   e.g. call import(package, only_local: #{old_boolean})"
        Autoproj.warn "   #{caller(1..1).first}"
        options[:only_local] = old_boolean
    end

    importdir = package.importdir
    if File.directory?(importdir)
        package.isolate_errors(mark_as_failed: false,
                               ignore_errors: ignore_errors) do
            if !checkout_only && package.update?
                perform_update(package, checkout_only: false, **options)
            elsif Autobuild.verbose
                package.message "%s: not updating"
            end
        end

    elsif File.exist?(importdir)
        raise ConfigException.new(package, 'import'),
              "#{importdir} exists but is not a directory"
    else
        package.isolate_errors(mark_as_failed: true,
                               ignore_errors: ignore_errors) do
            perform_checkout(package, allow_interactive: allow_interactive)
            true
        end
    end
end

#interactive? ⇒ Boolean

Whether this importer will need interaction with the user, for instance to give credentials

Returns:

  • (Boolean) 


182
183
184
 
# File 'lib/autobuild/importer.rb', line 182

def interactive?
    @interactive
end

#parse_patch_list(package, patches_file) ⇒ Object 



539
540
541
542
543
544
545
546
547
548
549
550
551
 
# File 'lib/autobuild/importer.rb', line 539

def parse_patch_list(package, patches_file)
    File.readlines(patches_file).map do |line|
        line = line.rstrip
        if line =~ /^(.*)\s+(\d+)$/
            path = File.expand_path($1, package.srcdir)
            level = Integer($2)
        else
            path = File.expand_path(line, package.srcdir)
            level = 0
        end
        [path, level, File.read(path)]
    end
end

#patch(package, patches = self.patches) ⇒ Object 



568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
 
# File 'lib/autobuild/importer.rb', line 568

def patch(package, patches = self.patches)
    # Get the list of already applied patches
    cur_patches = currently_applied_patches(package)

    cur_patches_state = cur_patches.map { |_, level, content| [level, content] }
    patches_state     = patches.map { |_, level, content| [level, content] }
    return false if cur_patches_state == patches_state

    # Do not be smart, remove all already applied patches
    # and then apply the new ones
    begin
        apply_count = (patches - cur_patches).size
        unapply_count = (cur_patches - patches).size
        if apply_count > 0 && unapply_count > 0
            package.message "patching %s: applying #{apply_count} and "\
                "unapplying #{unapply_count} patch(es)"
        elsif apply_count > 0
            package.message "patching %s: applying #{apply_count} patch(es)"
        elsif unapply_count > 0
            package.message "patching %s: unapplying #{unapply_count} patch(es)"
        end

        while (p = cur_patches.last)
            p, level, = *p
            unapply(package, p, level)
            cur_patches.pop
        end

        patches.to_a.each do |new_patch, new_patch_level, content|
            apply(package, new_patch, new_patch_level)
            cur_patches << [new_patch, new_patch_level, content]
        end
    ensure
        save_patch_state(package, cur_patches)
    end

    true
end

#patchdir(package) ⇒ Object 



515
516
517
 
# File 'lib/autobuild/importer.rb', line 515

def patchdir(package)
    File.join(package.importdir, ".autobuild-patches")
end

#patches ⇒ Object 



238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
 
# File 'lib/autobuild/importer.rb', line 238

def patches
    patches =
        if @options[:patches].respond_to?(:to_ary)
            @options[:patches]
        elsif !@options[:patches]
            []
        else
            [[@options[:patches], 0]]
        end

    single_patch = (patches.size == 2 &&
        patches[0].respond_to?(:to_str) &&
        patches[1].respond_to?(:to_int))

    patches = [patches] if single_patch
    patches.map do |obj|
        if obj.respond_to?(:to_str)
            path  = obj
            level = 0
        elsif obj.respond_to?(:to_ary)
            path, level = obj
        else
            raise Arguments, "wrong patch specification #{obj.inspect}"
        end
        [path, level, File.read(path)]
    end
end

#patches_fingerprint(package) ⇒ Object

fingerprint for patches associated to this package



222
223
224
225
226
227
228
 
# File 'lib/autobuild/importer.rb', line 222

def patches_fingerprint(package)
    cur_patches = currently_applied_patches(package)
    cur_patches.map(&:shift) # leave only level and source information
    if !patches.empty? && cur_patches
      Digest::SHA1.hexdigest(cur_patches.sort.flatten.join(""))
    end
end

#patchlist(package) ⇒ Object

We assume that package.importdir already exists (checkout is supposed to have been called)



521
522
523
 
# File 'lib/autobuild/importer.rb', line 521

def patchlist(package)
    File.join(patchdir(package), "list")
end

#perform_checkout(package, **options) ⇒ Object 



405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
 
# File 'lib/autobuild/importer.rb', line 405

def perform_checkout(package, **options)
    last_error = nil
    package.progress_start "checking out %s", :done_message => 'checked out %s' do
        retry_count = 0
        begin
            checkout(package, **options)
            execute_post_hooks(package)
        rescue Interrupt
            if last_error then raise last_error
            else
                raise
            end
        rescue ::Exception => e
            last_error = e
            retry_count = update_retry_count(e, retry_count)
            raise unless retry_count

            package.message "checkout of %s failed, "\
                "deleting the source directory #{package.importdir} "\
                "and retrying (#{retry_count}/#{self.retry_count})"
            FileUtils.rm_rf package.importdir
            retry
        end
    end

    patch(package)
    package.updated = true
rescue Interrupt
    raise
rescue ::Exception # rubocop:disable Lint/ShadowedException
    package.message "checkout of %s failed, "\
        "deleting the source directory #{package.importdir}"
    FileUtils.rm_rf package.importdir
    raise
rescue Autobuild::Exception => e
    FileUtils.rm_rf package.importdir
    fallback(e, package, :import, package)
end

#perform_update(package, only_local = false) ⇒ Object 



331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
 
# File 'lib/autobuild/importer.rb', line 331

def perform_update(package, only_local = false)
    cur_patches    = currently_applied_patches(package)
    needed_patches = patches
    patch_changed = cur_patches.map(&:last) != needed_patches.map(&:last)
    patch(package, []) if patch_changed

    last_error = nil
    retry_count = 0
    package.progress_start "updating %s"
    begin
        begin
            did_update = update(package, only_local)
            execute_post_hooks(package, error: false)
        rescue ::Exception
            execute_post_hooks(package, error: true)
            raise
        end

        message = if did_update == false
                      Autobuild.color('already up-to-date', :green)
                  else
                      Autobuild.color('updated', :yellow)
                  end

        did_update
    rescue Interrupt
        message = Autobuild.color('interrupted', :red)
        if last_error
            raise last_error
        else
            raise
        end
    rescue ::Exception => e
        message = Autobuild.color('update failed', :red)
        last_error = e
        # If the package is patched, it might be that the update
        # failed because we needed to unpatch first. Try it out
        #
        # This assumes that importing data with conflict will
        # make the import fail, but not make the patch
        # un-appliable. Importers that do not follow this rule
        # will have to unpatch by themselves.
        cur_patches = currently_applied_patches(package)
        unless cur_patches.empty?
            package.progress_done
            package.message "update failed and some patches are applied, "\
                "removing all patches and retrying"
            begin
                patch(package, [])
                return perform_update(package, only_local)
            rescue Interrupt
                raise
            rescue ::Exception
                raise e
            end
        end

        retry_count = update_retry_count(e, retry_count)
        raise unless retry_count

        package.message "update failed in #{package.importdir}, "\
            "retrying (#{retry_count}/#{self.retry_count})"
        retry
    ensure
        package.progress_done "#{message} %s"
    end

    patch(package)
    package.updated = true
    did_update
rescue Autobuild::Exception => e
    fallback(e, package, :import, package)
end

#retry_count ⇒ Object

The number of times update / checkout should be retried before giving up. The default is 0 (do not retry)

Set either with #retry_count= or by setting the :retry_count option when constructing this importer



194
195
196
 
# File 'lib/autobuild/importer.rb', line 194

def retry_count
    @options[:retry_count] || 0
end

#retry_count=(count) ⇒ Object

Sets the number of times update / checkout should be retried before giving up. 0 (the default) disables retrying.

See also #retry_count



234
235
236
 
# File 'lib/autobuild/importer.rb', line 234

def retry_count=(count)
    @options[:retry_count] = Integer(count)
end

#save_patch_state(package, cur_patches) ⇒ Object 



607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
 
# File 'lib/autobuild/importer.rb', line 607

def save_patch_state(package, cur_patches)
    patch_dir = patchdir(package)
    FileUtils.mkdir_p patch_dir
    cur_patches = cur_patches.each_with_index.
        map do |(_path, level, content), idx|
            path = File.join(patch_dir, idx.to_s)
            File.open(path, 'w') do |patch_io|
                patch_io.write content
            end
            [path, level]
        end
    File.open(patchlist(package), 'w') do |f|
        patch_state = cur_patches.map do |path, level|
            path = Pathname.new(path).
                relative_path_from(Pathname.new(package.srcdir)).to_s
            "#{path} #{level}"
        end
        f.write(patch_state.join("\n"))
    end
end

#supports_relocation? ⇒ Boolean

Returns:

  • (Boolean) 


628
629
630
 
# File 'lib/autobuild/importer.rb', line 628

def supports_relocation?
    false
end

#unapply(package, path, patch_level = 0) ⇒ Object 



535
536
537
 
# File 'lib/autobuild/importer.rb', line 535

def unapply(package, path, patch_level = 0)
    call_patch(package, true, path, patch_level)
end

#update_retry_count(original_error, retry_count) ⇒ Object 



266
267
268
269
270
271
 
# File 'lib/autobuild/importer.rb', line 266

def update_retry_count(original_error, retry_count)
    return if !original_error.respond_to?(:retry?) || !original_error.retry?

    retry_count += 1
    retry_count if retry_count <= self.retry_count
end

#vcs_fingerprint(package) ⇒ Object

basic fingerprint of the package and its dependencies



214
215
216
217
218
219
 
# File 'lib/autobuild/importer.rb', line 214

def vcs_fingerprint(package)
    # each importer type should implement its own
    Autoproj.warn "Fingerprint in #{package.name} has not been implemented "\
                  "for this type of packages, results should be discarded"
    nil
end