Module: Ductr

Defined in:
lib/ductr.rb,
lib/ductr/job.rb,
lib/ductr/store.rb,
lib/ductr/adapter.rb,
lib/ductr/etl_job.rb,
lib/ductr/trigger.rb,
lib/ductr/version.rb,
lib/ductr/cli/main.rb,
lib/ductr/kiba_job.rb,
lib/ductr/pipeline.rb,
lib/ductr/registry.rb,
lib/ductr/scheduler.rb,
lib/ductr/etl/parser.rb,
lib/ductr/etl/runner.rb,
lib/ductr/job_status.rb,
lib/ductr/log/logger.rb,
lib/ductr/cli/default.rb,
lib/ductr/configuration.rb,
lib/ductr/pipeline_step.rb,
lib/ductr/rufus_trigger.rb,
lib/ductr/job_etl_runner.rb,
lib/ductr/etl/kiba_runner.rb,
lib/ductr/pipeline_runner.rb,
lib/ductr/store/job_store.rb,
lib/ductr/etl/fiber_runner.rb,
lib/ductr/etl/fiber_control.rb,
lib/ductr/etl/controls/source.rb,
lib/ductr/sequel_base/adapter.rb,
lib/ductr/etl/controls/control.rb,
lib/ductr/store/job_serializer.rb,
lib/ductr/store/pipeline_store.rb,
lib/ductr/etl/controls/transform.rb,
lib/ductr/log/outputs/file_output.rb,
lib/ductr/etl/controls/destination.rb,
lib/ductr/sequel_base/basic_lookup.rb,
lib/ductr/sequel_base/basic_source.rb,
lib/ductr/sequel_base/match_lookup.rb,
lib/ductr/cli/new_project_generator.rb,
lib/ductr/store/pipeline_serializer.rb,
lib/ductr/log/outputs/standard_output.rb,
lib/ductr/sequel_base/buffered_lookup.rb,
lib/ductr/sequel_base/polling_handler.rb,
lib/ductr/sequel_base/polling_trigger.rb,
lib/ductr/sequel_base/paginated_source.rb,
lib/ductr/etl/controls/paginated_source.rb,
lib/ductr/sequel_base/basic_destination.rb,
lib/ductr/log/formatters/color_formatter.rb,
lib/ductr/etl/controls/buffered_transform.rb,
lib/ductr/sequel_base/buffered_destination.rb,
lib/ductr/etl/controls/buffered_destination.rb,
lib/ductr/sequel_base/buffered_upsert_destination.rb

Overview

The main Ductr module.

Defined Under Namespace

Modules: CLI, ETL, JobETLRunner, JobStatus, Log, SequelBase, Store Classes: Adapter, AdapterNotFoundError, Configuration, ControlNotFoundError, ETLJob, InconsistentPaginationError, Job, KibaJob, NotFoundInRegistryError, Pipeline, PipelineRunner, PipelineStep, Registry, RufusTrigger, Scheduler, Trigger

Constant Summary collapse

VERSION =

Returns The ductr’s version number.

Returns:

  • (String) —

    The ductr’s version number

"0.2.3"

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.config ⇒ Configuration (readonly)

Contains all the Ductr configuration.

Returns:



21
22
23
 
# File 'lib/ductr.rb', line 21

def config
  @config
end

Class Method Details

.adapter_registry ⇒ Registry

The adapter classes registry, all declared adapters are in the registry.

Returns:



28
29
30
 
# File 'lib/ductr.rb', line 28

def adapter_registry
  @adapter_registry ||= Registry.new(:adapter)
end

.configure {|config| ... } ⇒ void

This method returns an undefined value.

The configure block allows to configure Ductr internals. You must calls this method one and only one time to use the framework.

Yields:

  • (config) —

    Configure the framework

Yield Parameters:

Raises:

  • (ScriptError) —

    Raises when called more than one time



78
79
80
81
82
83
84
 
# File 'lib/ductr.rb', line 78

def configure
  raise ScriptError, "Ductr::configure must be called only once" if @config

  @config = Configuration.new(env)
  yield(@config)
  @config.apply_active_job_config
end

.development? ⇒ Boolean

Determines if Ductr is in development mode.

Returns:

  • (Boolean) —

    True if DUCTR_ENV is set to “development” or nil



56
57
58
 
# File 'lib/ductr.rb', line 56

def development?
  env == "development"
end

.env ⇒ String

The Ductr current environment, “development” by default. You can change it by setting the ‘DUCTR_ENV` environment variable.

Returns:

  • (String) —

    The Ductr environment



47
48
49
 
# File 'lib/ductr.rb', line 47

def env
  @env ||= ENV.fetch("DUCTR_ENV", "development").downcase
end

.logger ⇒ Log::Logger

The Ductr main logger instance.

Returns:



91
92
93
 
# File 'lib/ductr.rb', line 91

def logger
  @logger ||= config.logging.new
end

.production? ⇒ Boolean

Determines if Ductr is in production mode.

Returns:

  • (Boolean) —

    True if DUCTR_ENV is set to “production”



65
66
67
 
# File 'lib/ductr.rb', line 65

def production?
  env == "production"
end

.store ⇒ ActiveSupport::Cache::Store

The Ductr store, used to share information across different instances.

Returns:

  • (ActiveSupport::Cache::Store) —

    The store instance



100
101
102
103
104
105
106
107
108
109
110
111
 
# File 'lib/ductr.rb', line 100

def store
  adapter = config.store_adapter
  params = config.store_parameters
  options = config.store_options

  @store ||= \
    if adapter.is_a? Class
      adapter.new(*params, **options)
    else
      ActiveSupport::Cache.lookup_store(adapter, *params, **options)
    end
end

.trigger_registry ⇒ Registry

The trigger classes registry, all declared triggers are in the registry.

Returns:



37
38
39
 
# File 'lib/ductr.rb', line 37

def trigger_registry
  @trigger_registry ||= Registry.new(:trigger)
end