Cloud Bigtable
Cloud Bigtable is a petabyte-scale, fully managed NoSQL database service for large analytical and operational workloads. Ideal for ad tech, fintech, and IoT, Cloud Bigtable offers consistent sub-10ms latency. Replication provides higher availability, higher durability, and resilience in the face of zonal failures. Cloud Bigtable is designed with a storage engine for machine learning applications and provides easy integration with open source big data tools.
For more information about Cloud Bigtable, read the Cloud Bigtable Documentation.
The goal of google-cloud is to provide an API that is comfortable to Rubyists. Your authentication credentials are detected automatically in Google Cloud Platform (GCP), including Google Compute Engine (GCE), Google Kubernetes Engine (GKE), Google App Engine (GAE), Google Cloud Functions (GCF) and Cloud Run. In other environments you can configure authentication easily, either directly in your code or via environment variables. Read more about the options for connecting in the Authentication Guide.
Creating instances and clusters
When you first use Cloud Bigtable, you must create an instance, which is an allocation of resources that are used by Cloud Bigtable. When you create an instance, you must specify at least one cluster. Clusters describe where your data is stored and how many nodes are used for your data.
Use Project#create_instance to create an instance. The following example creates a production instance with one cluster and three nodes:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
job = bigtable.create_instance(
"my-instance",
display_name: "Instance for user data",
labels: { "env" => "dev"}
) do |clusters|
clusters.add("test-cluster", "us-east1-b", nodes: 3, storage_type: :SSD)
end
job.done? #=> false
# To block until the operation completes.
job.wait_until_done!
job.done? #=> true
if job.error?
status = job.error
else
instance = job.instance
end
You can also create a low-cost development instance for development and testing,
with performance limited to the equivalent of a one-node cluster. There are no
monitoring or throughput guarantees; replication is not available; and the SLA
does not apply. When creating a development instance, you do not specify nodes
for your clusters:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
job = bigtable.create_instance(
"my-instance",
display_name: "Instance for user data",
type: :DEVELOPMENT,
labels: { "env" => "dev"}
) do |clusters|
clusters.add("test-cluster", "us-east1-b") # nodes not allowed
end
job.done? #=> false
# Reload job until completion.
job.wait_until_done!
job.done? #=> true
if job.error?
status = job.error
else
instance = job.instance
end
You can upgrade a development instance to a production instance at any time.
Creating tables
Cloud Bigtable stores data in massively scalable tables, each of which is a sorted key/value map. The table is composed of rows, each of which typically describes a single entity, and columns, which contain individual values for each row. Each row is indexed by a single row key, and columns that are related to one another are typically grouped together into a column family. Each column is identified by a combination of the column family and a column qualifier, which is a unique name within the column family.
Each row/column intersection can contain multiple cells, or versions, at different timestamps, providing a record of how the stored data has been altered over time. Cloud Bigtable tables are sparse; if a cell does not contain any data, it does not take up any space.
Use Project#create_table or Instance#create_table to create a table:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.create_table("my-instance", "my-table")
puts table.name
When you create a table, you may specify the column families to use in the table, as well as a list of row keys that will be used to initially split the table into several tablets (tablets are similar to HBase regions):
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
initial_splits = ["user-00001", "user-100000", "others"]
table = bigtable.create_table("my-instance", "my-table", initial_splits: initial_splits) do |cfm|
cfm.add('cf1', gc_rule: Google::Cloud::Bigtable::GcRule.max_versions(5))
cfm.add('cf2', gc_rule: Google::Cloud::Bigtable::GcRule.max_age(600))
gc_rule = Google::Cloud::Bigtable::GcRule.union(
Google::Cloud::Bigtable::GcRule.max_age(1800),
Google::Cloud::Bigtable::GcRule.max_versions(3)
)
cfm.add('cf3', gc_rule: gc_rule)
end
puts table
You may also add, update, and delete column families later by passing a block to Table#column_families:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table", perform_lookup: true)
table.column_families do |cfm|
cfm.add "cf4", gc_rule: Google::Cloud::Bigtable::GcRule.max_age(600)
cfm.add "cf5", gc_rule: Google::Cloud::Bigtable::GcRule.max_versions(5)
rule_1 = Google::Cloud::Bigtable::GcRule.max_versions(3)
rule_2 = Google::Cloud::Bigtable::GcRule.max_age(600)
rule_union = Google::Cloud::Bigtable::GcRule.union(rule_1, rule_2)
cfm.update "cf2", gc_rule: rule_union
cfm.delete "cf3"
end
puts table.column_families["cf3"] #=> nil
Writing data
The Table class allows you to perform the following types of writes:
- Simple writes
- Increments and appends
- Conditional writes
- Batch writes
See Cloud Bigtable writes for detailed information about writing data.
Simple writes
Use Table#mutate_row to make one or more mutations to a single row:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
entry = table.new_mutation_entry("user-1")
entry.set_cell(
"cf-1",
"field-1",
"XYZ",
timestamp: (Time.now.to_f * 1000000).round(-3) # microseconds
).delete_cells("cf2", "field02")
table.mutate_row(entry)
Increments and appends
If you want to append data to an existing value or increment an existing numeric value, use Table#read_modify_write_row:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
rule_1 = table.new_read_modify_write_rule("cf", "field01")
rule_1.append("append-xyz")
rule_2 = table.new_read_modify_write_rule("cf", "field01")
rule_2.increment(1)
row = table.read_modify_write_row("user01", [rule_1, rule_2])
puts row.cells
Do not use read_modify_write_row if you are using an app profile that has
multi-cluster routing. (See
AppProfile#routing_policy.)
Conditional writes
To check a row for a condition and then, depending on the result, write data to that row, use Table#check_and_mutate_row:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
predicate_filter = Google::Cloud::Bigtable::RowFilter.key("user-10")
on_match_mutations = Google::Cloud::Bigtable::MutationEntry.new
on_match_mutations.set_cell(
"cf-1",
"field-1",
"XYZ",
timestamp: (Time.now.to_f * 1000000).round(-3) # microseconds
).delete_cells("cf2", "field02")
otherwise_mutations = Google::Cloud::Bigtable::MutationEntry.new
otherwise_mutations.delete_from_family("cf3")
predicate_matched = table.check_and_mutate_row(
"user01",
predicate_filter,
on_match: on_match_mutations,
otherwise: otherwise_mutations
)
if predicate_matched
puts "All predicates matched"
end
Do not use check_and_mutate_row if you are using an app profile that has
multi-cluster routing. (See
AppProfile#routing_policy.)
Batch writes
You can write more than one row in a single RPC using Table#mutate_rows:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
entries = []
entries << table.new_mutation_entry("row-1").set_cell("cf1", "field1", "XYZ")
entries << table.new_mutation_entry("row-2").set_cell("cf1", "field1", "ABC")
responses = table.mutate_rows(entries)
responses.each do |response|
puts response.status.description
end
Each entry in the request is atomic, but the request as a whole is not. As shown above, Cloud Bigtable returns a list of responses corresponding to the entries.
Reading data
The Table class also enables you to read data.
Use Table#read_row to read a single row by key:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
row = table.read_row("user-1")
If desired, you can apply a filter:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
filter = Google::Cloud::Bigtable::RowFilter.cells_per_row(3)
row = table.read_row("user-1", filter: filter)
For multiple rows, the Table#read_rows method streams back the contents of all requested rows in key order:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
table.read_rows(keys: ["user-1", "user-2"]).each do |row|
puts row
end
Instead of specifying individual keys (or a range), you can often just use a filter:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
filter = table.filter.key("user-*")
# OR
# filter = Google::Cloud::Bigtable::RowFilter.key("user-*")
table.read_rows(filter: filter).each do |row|
puts row
end
Deleting rows, tables, and instances
Use Table#drop_row_range to delete some or all of the rows in a table:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
table = bigtable.table("my-instance", "my-table")
# Delete rows using row key prefix.
table.drop_row_range(row_key_prefix: "user-100")
# Delete all data With timeout
table.drop_row_range(delete_all_data: true, timeout: 120) # 120 seconds.
Delete tables and instances using Table#delete and Instance#delete, respectively:
require "google/cloud/bigtable"
bigtable = Google::Cloud::Bigtable.new
instance = bigtable.instance("my-instance")
table = instance.table("my-table")
table.delete
instance.delete
Additional information
Google Bigtable can be configured to use an emulator or to enable gRPC's logging. To learn more, see the Emulator guide and Logging guide.