Class: Google::Apis::DataflowV1b3::Source

Inherits:
Object
  • Object
show all
Includes:
Core::Hashable, Core::JsonObjectSupport
Defined in:
generated/google/apis/dataflow_v1b3/classes.rb,
generated/google/apis/dataflow_v1b3/representations.rb,
generated/google/apis/dataflow_v1b3/representations.rb

Overview

A source that records can be read and decoded from.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Core::JsonObjectSupport

#to_json

Methods included from Core::Hashable

process_value, #to_h

Constructor Details

#initialize(**args) ⇒ Source

Returns a new instance of Source.


4433
4434
4435
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4433

def initialize(**args)
   update!(**args)
end

Instance Attribute Details

#base_specsArray<Hash<String,Object>>

While splitting, sources may specify the produced bundles as differences against another source, in order to save backend-side memory and allow bigger jobs. For details, see SourceSplitRequest. To support this use case, the full set of parameters of the source is logically obtained by taking the latest explicitly specified value of each parameter in the order: base_specs (later items win), spec (overrides anything in base_specs). Corresponds to the JSON property baseSpecs

Returns:

  • (Array<Hash<String,Object>>)

4399
4400
4401
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4399

def base_specs
  @base_specs
end

#codecHash<String,Object>

The codec to use to decode data read from the source. Corresponds to the JSON property codec

Returns:

  • (Hash<String,Object>)

4404
4405
4406
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4404

def codec
  @codec
end

#does_not_need_splittingBoolean Also known as: does_not_need_splitting?

Setting this value to true hints to the framework that the source doesn't need splitting, and using SourceSplitRequest on it would yield SOURCE_SPLIT_OUTCOME_USE_CURRENT. E.g. a file splitter may set this to true when splitting a single file into a set of byte ranges of appropriate size, and set this to false when splitting a filepattern into individual files. However, for efficiency, a file splitter may decide to produce file subranges directly from the filepattern to avoid a splitting round-trip. See SourceSplitRequest for an overview of the splitting process. This field is meaningful only in the Source objects populated by the user (e.g. when filling in a DerivedSource). Source objects supplied by the framework to the user don' t have this field populated. Corresponds to the JSON property doesNotNeedSplitting

Returns:

  • (Boolean)

4419
4420
4421
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4419

def does_not_need_splitting
  @does_not_need_splitting
end

#metadataGoogle::Apis::DataflowV1b3::SourceMetadata

Metadata about a Source useful for automatically optimizing and tuning the pipeline, etc. Corresponds to the JSON property metadata


4426
4427
4428
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4426

def 
  @metadata
end

#specHash<String,Object>

The source to read from, plus its parameters. Corresponds to the JSON property spec

Returns:

  • (Hash<String,Object>)

4431
4432
4433
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4431

def spec
  @spec
end

Instance Method Details

#update!(**args) ⇒ Object

Update properties of this object


4438
4439
4440
4441
4442
4443
4444
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 4438

def update!(**args)
  @base_specs = args[:base_specs] if args.key?(:base_specs)
  @codec = args[:codec] if args.key?(:codec)
  @does_not_need_splitting = args[:does_not_need_splitting] if args.key?(:does_not_need_splitting)
  @metadata = args[:metadata] if args.key?(:metadata)
  @spec = args[:spec] if args.key?(:spec)
end