Module: OnlineMigrations::BackgroundMigrations::MigrationHelpers

Included in:
SchemaStatements
Defined in:
lib/online_migrations/background_migrations/migration_helpers.rb

Instance Method Summary collapse

Instance Method Details

#backfill_column_for_type_change_in_background(table_name, column_name, model_name: nil, type_cast_function: nil, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to use more flexible backfill_column_for_type_change.

Backfills data from the old column to the new column using background migrations.

Examples:

backfill_column_for_type_change_in_background(:files, :size)

With type casting

backfill_column_for_type_change_in_background(:users, :settings, type_cast_function: "jsonb")

Additional background migration options

backfill_column_for_type_change_in_background(:files, :size, batch_size: 10_000)

Parameters:

  • table_name (String, Symbol)
  • column_name (String, Symbol)
  • model_name (String) (defaults to: nil) —

    If Active Record multiple databases feature is used, the class name of the model to get connection from.

  • type_cast_function (String, Symbol) (defaults to: nil) —

    Some type changes require casting data to a new type. For example when changing from text to jsonb. In this case, use the type_cast_function option. You need to make sure there is no bad data and the cast will always succeed

  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:



86
87
88
89
90
91
92
93
94
95
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 86

def backfill_column_for_type_change_in_background(table_name, column_name, model_name: nil,
                                                  type_cast_function: nil, **options)
  backfill_columns_for_type_change_in_background(
    table_name,
    column_name,
    model_name: model_name,
    type_cast_functions: { column_name => type_cast_function },
    **options
  )
end

#backfill_column_in_background(table_name, column_name, value, model_name: nil, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to use more flexible update_column_in_batches.

Note:

Consider backfill_columns_in_background when backfilling multiple columns to avoid rewriting the table multiple times.

Backfills column data using background migrations.

Examples:

backfill_column_in_background(:users, :admin, false)

Additional background migration options

backfill_column_in_background(:users, :admin, false, batch_size: 10_000)

Parameters:

  • table_name (String, Symbol)
  • column_name (String, Symbol)
  • value
  • model_name (String) (defaults to: nil) —

    If Active Record multiple databases feature is used, the class name of the model to get connection from.

  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:



30
31
32
33
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 30

def backfill_column_in_background(table_name, column_name, value, model_name: nil, **options)
  backfill_columns_in_background(table_name, { column_name => value },
                                 model_name: model_name, **options)
end

#backfill_columns_for_type_change_in_background(table_name, *column_names, model_name: nil, type_cast_functions: {}, **options) ⇒ Object

Same as backfill_column_for_type_change_in_background but for multiple columns.

Parameters:

  • type_cast_functions (Hash) (defaults to: {}) —

    if not empty, keys - column names, values - corresponding type cast functions

See Also:



104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 104

def backfill_columns_for_type_change_in_background(table_name, *column_names, model_name: nil,
                                                   type_cast_functions: {}, **options)
  if model_name.nil? && Utils.multiple_databases?
    raise ArgumentError, "You must pass a :model_name when using multiple databases."
  end

  tmp_columns = column_names.map { |column_name| "#{column_name}_for_type_change" }
  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "CopyColumn",
    table_name,
    column_names,
    tmp_columns,
    model_name,
    type_cast_functions,
    **options
  )
end

#backfill_columns_in_background(table_name, updates, model_name: nil, **options) ⇒ Object

Same as backfill_column_in_background but for multiple columns.

Examples:

backfill_columns_in_background(:users, { admin: false, status: "active" })

Parameters:

  • updates (Hash) —

    keys - column names, values - corresponding values

See Also:



44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 44

def backfill_columns_in_background(table_name, updates, model_name: nil, **options)
  if model_name.nil? && Utils.multiple_databases?
    raise ArgumentError, "You must pass a :model_name when using multiple databases."
  end

  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "BackfillColumn",
    table_name,
    updates,
    model_name,
    **options
  )
end

#copy_column_in_background(table_name, copy_from, copy_to, model_name: nil, type_cast_function: nil, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to use more flexible update_column_in_batches.

Copies data from the old column to the new column using background migrations.

Examples:

copy_column_in_background(:users, :id, :id_for_type_change)

Parameters:

  • table_name (String, Symbol)
  • copy_from (String, Symbol) —

    source column name

  • copy_to (String, Symbol) —

    destination column name

  • model_name (String) (defaults to: nil) —

    If Active Record multiple databases feature is used, the class name of the model to get connection from.

  • type_cast_function (String, Symbol) (defaults to: nil) —

    Some type changes require casting data to a new type. For example when changing from text to jsonb. In this case, use the type_cast_function option. You need to make sure there is no bad data and the cast will always succeed

  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:



145
146
147
148
149
150
151
152
153
154
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 145

def copy_column_in_background(table_name, copy_from, copy_to, model_name: nil, type_cast_function: nil, **options)
  copy_columns_in_background(
    table_name,
    [copy_from],
    [copy_to],
    model_name: model_name,
    type_cast_functions: { copy_from => type_cast_function },
    **options
  )
end

#copy_columns_in_background(table_name, copy_from, copy_to, model_name: nil, type_cast_functions: {}, **options) ⇒ Object

Same as copy_column_in_background but for multiple columns.

Parameters:

  • type_cast_functions (Hash) (defaults to: {}) —

    if not empty, keys - column names, values - corresponding type cast functions

See Also:



163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 163

def copy_columns_in_background(table_name, copy_from, copy_to, model_name: nil, type_cast_functions: {}, **options)
  if model_name.nil? && Utils.multiple_databases?
    raise ArgumentError, "You must pass a :model_name when using multiple databases."
  end

  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "CopyColumn",
    table_name,
    copy_from,
    copy_to,
    model_name,
    type_cast_functions,
    **options
  )
end

#delete_associated_records_in_background(model_name, record_id, association, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to directly delete associated records.

Deletes associated records for a specific parent record using background migrations. This is useful when you are planning to remove a parent object (user, account etc) and needs to remove lots of its associated objects.

Examples:

delete_associated_records_in_background("Link", 1, :clicks)

Parameters:

  • model_name (String)
  • record_id (Integer, String) —

    parent record primary key's value

  • association (String, Symbol) —

    association name for which records will be removed

  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:



266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 266

def delete_associated_records_in_background(model_name, record_id, association, **options)
  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "DeleteAssociatedRecords",
    model_name,
    record_id,
    association,
    **options
  )
end

#delete_orphaned_records_in_background(model_name, *associations, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to directly find and delete orpahed records.

Deletes records with one or more missing relations using background migrations. This is useful when some referential integrity in the database is broken and you want to delete orphaned records.

Examples:

delete_orphaned_records_in_background("Post", :author)

Parameters:

  • model_name (String)
  • associations (Array)
  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:



237
238
239
240
241
242
243
244
245
246
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 237

def delete_orphaned_records_in_background(model_name, *associations, **options)
  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "DeleteOrphanedRecords",
    model_name,
    associations,
    **options
  )
end

#enqueue_background_migration(migration_name, *arguments, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

For convenience, the enqueued background migration is run inline in development and test environments

Creates a background migration for the given job class name.

A background migration runs one job at a time, computing the bounds of the next batch based on the current migration settings and the previous batch bounds. Each job's execution status is tracked in the database as the migration runs.

Examples:

enqueue_background_migration("BackfillProjectIssuesCount",
    batch_size: 10_000, batch_max_attempts: 10)

# Given the background migration exists:

class BackfillProjectIssuesCount < OnlineMigrations::BackgroundMigration
  def relation
    Project.all
  end

  def process_batch(projects)
    projects.update_all(
      "issues_count = (SELECT COUNT(*) FROM issues WHERE issues.project_id = projects.id)"
    )
  end

  # To be able to track progress, you need to define this method
  def count
    Project.maximum(:id)
  end
end

Parameters:

  • migration_name (String, Class) —

    Background migration job class name

  • arguments (Array) —

    Extra arguments to pass to the job instance when the migration runs

  • options (Hash) —

    a customizable set of options

Options Hash (**options):

  • :batch_column_name (Symbol, String) — default: primary key —

    Column name the migration will batch over

  • :min_value (Integer) —

    Value in the column the batching will begin at, defaults to SELECT MIN(batch_column_name)

  • :max_value (Integer) —

    Value in the column the batching will end at, defaults to SELECT MAX(batch_column_name)

  • :batch_size (Integer) — default: 20_000 —

    Number of rows to process in a single background migration run

  • :sub_batch_size (Integer) — default: 1000 —

    Smaller batches size that the batches will be divided into

  • :batch_pause (Integer) — default: 0 —

    Pause interval between each background migration job's execution (in seconds)

  • :sub_batch_pause_ms (Integer) — default: 100 —

    Number of milliseconds to sleep between each sub_batch execution

  • :batch_max_attempts (Integer) — default: 5 —

    Maximum number of batch run attempts

Returns:



372
373
374
375
376
377
378
379
380
381
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 372

def enqueue_background_migration(migration_name, *arguments, **options)
  migration = create_background_migration(migration_name, *arguments, **options)

  if Utils.run_background_migrations_inline?
    runner = MigrationRunner.new(migration)
    runner.run_all_migration_jobs
  end

  migration
end

#perform_action_on_relation_in_background(model_name, conditions, action, updates: nil, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to directly perform the action on associated records.

Performs specific action on a relation or individual records. This is useful when you want to delete/destroy/update/etc records based on some conditions.

Examples:

Delete records

perform_action_on_relation_in_background("User", { banned: true }, :delete_all)

Destroy records

perform_action_on_relation_in_background("User", { banned: true }, :destroy_all)

Update records

perform_action_on_relation_in_background("User", { banned: nil }, :update_all, updates: { banned: false })

Perform custom method on individual records

class User < ApplicationRecord
  def generate_invite_token
    self.invite_token = # some complex logic
  end
end

perform_action_on_relation_in_background("User", { invite_token: nil }, :generate_invite_token)

Parameters:

  • model_name (String)
  • conditions (Array, Hash, String) —

    conditions to filter the relation

  • action (String, Symbol) —

    action to perform on the relation or individual records. Relation-wide available actions: :delete_all, :destroy_all, and :update_all.

  • updates (Hash) (defaults to: nil) —

    updates to perform when action is set to :update_all

  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:



312
313
314
315
316
317
318
319
320
321
322
323
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 312

def perform_action_on_relation_in_background(model_name, conditions, action, updates: nil, **options)
  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "PerformActionOnRelation",
    model_name,
    conditions,
    action,
    { updates: updates },
    **options
  )
end

#remove_background_migration(migration_name, *arguments) ⇒ Object

Removes the background migration for the given class name and arguments, if exists.

Examples:

remove_background_migration("BackfillProjectIssuesCount")

Parameters:

  • migration_name (String, Class) —

    Background migration job class name

  • arguments (Array) —

    Extra arguments the migration was originally created with



391
392
393
394
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 391

def remove_background_migration(migration_name, *arguments)
  migration_name = migration_name.name if migration_name.is_a?(Class)
  Migration.for_configuration(migration_name, arguments).delete_all
end

#reset_counters_in_background(model_name, *counters, touch: nil, **options) ⇒ OnlineMigrations::BackgroundMigrations::Migration

Note:

This method is better suited for large tables (10/100s of millions of records). For smaller tables it is probably better and easier to use reset_counters from the Active Record.

Resets one or more counter caches to their correct value using background migrations. This is useful when adding new counter caches, or if the counter has been corrupted or modified directly by SQL.

Examples:

reset_counters_in_background("User", :projects, :friends, touch: true)

Touch specific column

reset_counters_in_background("User", :projects, touch: :touched_at)

Touch with specific time value

reset_counters_in_background("User", :projects, touch: [time: 2.days.ago])

Parameters:

  • model_name (String)
  • counters (Array)
  • touch (Boolean, Symbol, Array) (defaults to: nil) —

    touch timestamp columns when updating.

    • when true - will touch updated_at and/or updated_on
    • when Symbol or Array - will touch specific column(s)
  • options (Hash) —

    used to control the behavior of background migration. See #enqueue_background_migration

Returns:

See Also:



208
209
210
211
212
213
214
215
216
217
218
 
# File 'lib/online_migrations/background_migrations/migration_helpers.rb', line 208

def reset_counters_in_background(model_name, *counters, touch: nil, **options)
  model_name = model_name.name if model_name.is_a?(Class)

  enqueue_background_migration(
    "ResetCounters",
    model_name,
    counters,
    { touch: touch },
    **options
  )
end