Class: DiscordStore::BlobStore

Inherits:
Object
  • Object
show all
Defined in:
lib/discord_store/blob_store.rb

Overview

Stores arbitrary binary data as chunked message attachments.

The hard problem here is not chunking, it is addressing. In late 2023 Discord began signing CDN links with an HMAC over an expiry timestamp, and every project in this genre that had stored URLs in a database woke up to a dead index roughly a day later. The community's answer was to stand up caching proxies on Cloudflare Workers to re-resolve links on demand, which works and which also concedes the entire premise: the storage is only free if you ignore the server you now have to run.

This library stores channel_id, message_id and attachment_id, never a URL, and re-resolves at read time. Rails users get the rest for free: an ActiveStorage service in proxy mode routes reads through ActiveStorage::Blobs::ProxyController, which is the Cloudflare Worker everybody rebuilt, already in the framework.

Defined Under Namespace

Classes: Manifest, ManifestIndex

Constant Summary collapse

MANIFEST_STREAM = 
"__manifest"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(rest:, config:, index: nil) ⇒ BlobStore

Returns a new instance of BlobStore.

Parameters:



58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/discord_store/blob_store.rb', line 58

def initialize(rest:, config:, index: nil)
  @rest = rest
  @config = config
  @cipher = Cipher.build(config)
  @limits = GuildLimits.new(rest: rest, config: config)
  blob_channels = config.blob_channel_ids.empty? ? config.log_channel_ids : config.blob_channel_ids
  @shard = ChannelShard.new(blob_channels)
  @manifest_log = Log.new(
    rest: rest, config: config,
    channel_ids: [config.manifest_channel_id || @shard.channel_ids.first]
  )
  @index = index || ManifestIndex.new(log: @manifest_log)
end

Instance Attribute Details

#config ⇒ Object (readonly)

Returns the value of attribute config.



52
53
54
 
# File 'lib/discord_store/blob_store.rb', line 52

def config
  @config
end

#index ⇒ Object (readonly)

Returns the value of attribute index.



52
53
54
 
# File 'lib/discord_store/blob_store.rb', line 52

def index
  @index
end

#rest ⇒ Object (readonly)

Returns the value of attribute rest.



52
53
54
 
# File 'lib/discord_store/blob_store.rb', line 52

def rest
  @rest
end

Instance Method Details

#chunk_size ⇒ Integer

Returns bytes per chunk, discovered from the guild.

Returns:

  • (Integer) —

    bytes per chunk, discovered from the guild



73
 
# File 'lib/discord_store/blob_store.rb', line 73

def chunk_size = @limits.chunk_size

#delete(key) ⇒ void

This method returns an undefined value.

Parameters:

  • key (String) 


219
220
221
222
223
224
225
226
 
# File 'lib/discord_store/blob_store.rb', line 219

def delete(key)
  manifest = @index.get(key)
  return if manifest.nil?

  retire(manifest)
  @index.delete(key)
  nil
end

#delete_prefix(prefix) ⇒ Integer

Deletes every blob whose key starts with prefix.

Parameters:

  • prefix (String)

Returns:

  • (Integer) —

    how many were deleted



232
233
234
235
236
 
# File 'lib/discord_store/blob_store.rb', line 232

def delete_prefix(prefix)
  keys = @index.keys.select { |key| key.start_with?(prefix) }
  keys.each { |key| delete(key) }
  keys.size
end

#download(key) {|bytes| ... } ⇒ void

This method returns an undefined value.

Streams the blob without holding all of it.

Parameters:

  • key (String)

Yield Parameters:

  • bytes (String) 


170
171
172
 
# File 'lib/discord_store/blob_store.rb', line 170

def download(key, &)
  each_chunk_body(fetch_manifest!(key), &)
end

#exist?(key) ⇒ Boolean

Parameters:

  • key (String)

Returns:

  • (Boolean) 


211
 
# File 'lib/discord_store/blob_store.rb', line 211

def exist?(key) = !@index.get(key).nil?

#get(key) ⇒ String

Returns the blob's bytes.

Parameters:

  • key (String)

Returns:

  • (String) —

    the blob's bytes

Raises:



157
158
159
160
161
162
163
 
# File 'lib/discord_store/blob_store.rb', line 157

def get(key)
  manifest = fetch_manifest!(key)
  buffer = +""
  buffer.force_encoding(Encoding::BINARY)
  each_chunk_body(manifest) { |bytes| buffer << bytes }
  buffer
end

#get_range(key, range) ⇒ String

Reads a byte range.

Only the chunks the range touches are fetched, so a range read of a large blob costs a couple of requests rather than all of them. Whether the CDN honours an HTTP Range header on top of that is a bonus, not a dependency.

Parameters:

  • key (String)
  • range (Range)

Returns:

  • (String) 


183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
 
# File 'lib/discord_store/blob_store.rb', line 183

def get_range(key, range)
  manifest = fetch_manifest!(key)
  first, last = clamp_range(range, manifest.byte_size)
  return +"" if first > last

  out = +""
  out.force_encoding(Encoding::BINARY)
  offset = 0

  manifest.chunks.each_with_index do |chunk, ordinal|
    chunk_first = offset
    chunk_last = offset + chunk[:size] - 1
    offset += chunk[:size]

    next if chunk_last < first
    break if chunk_first > last

    bytes = fetch_chunk(manifest.key, ordinal, chunk)
    from = [first - chunk_first, 0].max
    to = [last - chunk_first, bytes.bytesize - 1].min
    out << bytes.byteslice(from, to - from + 1).to_s
  end

  out
end

#orphans(search: nil) ⇒ Array<Hash>

Chunks that no live manifest points at.

A blob is a manifest plus its chunks, written in that order and deleted in the other. Anything that interrupts the middle -- a crash, a rate limit that outlived its retries, a delete that got halfway -- leaves chunks with nothing referring to them. They are invisible to every other method here, because every other method starts from a manifest, and they count against the guild's storage forever.

Finding them means asking a question no channel scan answers well: which attachments exist that nothing references. Discord's search can answer it directly, filtering on an extension that is already public and says nothing about the contents.

Costs one search page per 25 chunks and needs the MESSAGE_CONTENT intent.

Parameters:

  • search (Search, nil) (defaults to: nil) —

    defaults to one built from this config

Returns:

  • (Array<Hash>) —

    each channel_id:, filename:, size:

Raises:



132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
 
# File 'lib/discord_store/blob_store.rb', line 132

def orphans(search: nil)
  finder = search || Search.new(rest: @rest, config: @config)
  referenced = referenced_message_ids

  found = []
  finder.each(has: "file",
              attachment_extension: Search::EXTENSION,
              channel_ids: @config.blob_channel_ids) do |message|
    next if referenced.include?(message["id"].to_s)

    Array(message["attachments"]).each do |attachment|
      # Spilled log records share the extension and are not blob chunks.
      next if attachment["filename"].to_s == Codec::SPILL_FILENAME

      found << { message_id: message["id"].to_s, channel_id: message["channel_id"].to_s,
                 filename: attachment["filename"].to_s, size: attachment["size"].to_i }
    end
  end

  found
end

#put(key, data, content_type: "application/octet-stream", checksum: nil) ⇒ Manifest

Stores data under key, replacing anything already there.

Parameters:

  • key (String)
  • data (String, IO)
  • content_type (String) (defaults to: "application/octet-stream")
  • checksum (String, nil) (defaults to: nil) —

    expected MD5, base64; verified if given

Returns:



82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
 
# File 'lib/discord_store/blob_store.rb', line 82

def put(key, data, content_type: "application/octet-stream", checksum: nil)
  io = data.respond_to?(:read) ? data : StringIO.new(data.to_s.dup.force_encoding(Encoding::BINARY))
  channel_id = @shard.for(key)

  chunks = []
  digest = Digest::MD5.new
  total = 0

  each_chunk(io) do |bytes, ordinal|
    digest << bytes
    total += bytes.bytesize
    chunks << upload_chunk(channel_id, key, ordinal, bytes)
  end

  computed = [digest.digest].pack("m0")

  if checksum && checksum != computed
    chunks.each { |chunk| safe_delete(chunk[:channel_id], chunk[:message_id]) }
    raise Error, "checksum mismatch for #{key}: expected #{checksum}, computed #{computed}"
  end

  manifest = Manifest.new(key: key, byte_size: total, checksum: computed,
                          content_type: content_type, chunks: chunks, channel_id: channel_id)

  previous = @index.get(key)
  @index.put(key, manifest)
  retire(previous) if previous

  manifest
end

#size(key) ⇒ Integer?

Returns byte size, without fetching the data.

Parameters:

  • key (String)

Returns:

  • (Integer, nil) —

    byte size, without fetching the data



215
 
# File 'lib/discord_store/blob_store.rb', line 215

def size(key) = @index.get(key)&.byte_size

#url(key) ⇒ String

A freshly signed CDN URL for the blob.

Only possible for a single-chunk blob: a blob split across attachments has no single URL, and there is no way to make one without a server that concatenates the pieces. That is the whole argument for proxy mode.

Parameters:

  • key (String)

Returns:

  • (String)

Raises:

  • (Error) —

    if the blob has more than one chunk



247
248
249
250
251
252
253
254
255
256
257
 
# File 'lib/discord_store/blob_store.rb', line 247

def url(key)
  manifest = fetch_manifest!(key)

  if manifest.chunk_count != 1
    raise Error,
          "blob #{key} spans #{manifest.chunk_count} attachments and has no single URL. " \
          "Configure the ActiveStorage service with a proxy route, or store smaller objects."
  end

  resolve_urls(manifest.chunks).fetch(manifest.chunks.first[:attachment_id])
end