Class: Aws::Glue::Types::IntegrationPartition

Inherits:
Struct
  • Object
show all
Includes:
Structure
Defined in:
lib/aws-sdk-glue/types.rb

Overview

A structure that describes how data is partitioned on the target.

See Also:

Constant Summary collapse

SENSITIVE = 
[]

Instance Attribute Summary collapse

Instance Attribute Details

#conversion_spec ⇒ String

Specifies the timestamp format of the source data. Valid values are:

  • epoch_sec - Unix epoch timestamp in seconds

  • epoch_milli - Unix epoch timestamp in milliseconds

  • iso - ISO 8601 formatted timestamp

<note markdown=“1”> Only specify ConversionSpec when using timestamp-based partition functions (year, month, day, or hour). Glue Zero-ETL uses this parameter to correctly transform source data into timestamp format before partitioning.

Do not use high-cardinality columns with the `identity` partition

function. High-cardinality columns include:

* Primary keys

  • Timestamp fields (such as LastModifiedTimestamp, CreatedDate)

  • System-generated timestamps

Using high-cardinality columns with identity partitioning creates

many small partitions, which can significantly degrade ingestion performance.

</note>

Returns:

  • (String) 


17130
17131
17132
17133
17134
17135
17136
 
# File 'lib/aws-sdk-glue/types.rb', line 17130

class IntegrationPartition < Struct.new(
  :field_name,
  :function_spec,
  :conversion_spec)
  SENSITIVE = []
  include Aws::Structure
end

#field_name ⇒ String

The field name used to partition data on the target. Avoid using columns that have unique values for each row (for example, LastModifiedTimestamp, SystemModTimeStamp) as the partition column. These columns are not suitable for partitioning because they create a large number of small partitions, which can lead to performance issues.

Returns:

  • (String) 


17130
17131
17132
17133
17134
17135
17136
 
# File 'lib/aws-sdk-glue/types.rb', line 17130

class IntegrationPartition < Struct.new(
  :field_name,
  :function_spec,
  :conversion_spec)
  SENSITIVE = []
  include Aws::Structure
end

#function_spec ⇒ String

Specifies the function used to partition data on the target. The accepted values for this parameter are:

  • identity - Uses source values directly without transformation

  • year - Extracts the year from timestamp values (e.g., 2023)

  • month - Extracts the month from timestamp values (e.g., 2023-01)

  • day - Extracts the day from timestamp values (e.g., 2023-01-15)

  • hour - Extracts the hour from timestamp values (e.g., 2023-01-15-14)

Returns:

  • (String) 


17130
17131
17132
17133
17134
17135
17136
 
# File 'lib/aws-sdk-glue/types.rb', line 17130

class IntegrationPartition < Struct.new(
  :field_name,
  :function_spec,
  :conversion_spec)
  SENSITIVE = []
  include Aws::Structure
end