Class: DiscordStore::Search

Inherits:
Object
  • Object
show all
Defined in:
lib/discord_store/search.rb

Overview

Discord's message index, exposed deliberately and used for almost nothing.

Discord does have a query API. GET /guilds/{id}/messages/search takes a full-text content filter, channel and author filters, snowflake bounds, attachment filename and extension filters, and sorts by relevance or time. It is more of a query interface than a chat platform owes anybody.

It is still not where this library reads from, for four reasons that are Discord's documentation rather than an opinion:

1. It is allowed to under-return. "Search may return slightly fewer
 results than the limit specified", and clients "should not rely on the
 length of the messages array to paginate". An index that silently drops
 rows gives you wrong answers, not slow ones, and a query layer that is
 sometimes wrong is worse than no query layer.
2. It is eventually consistent. Fresh messages answer 202 until indexed,
 so there is no read-your-writes.
3. It pages at 25 and cannot offset past 9975, capping any single query
 at ten thousand rows across four hundred round trips.
4. Everything this library stores is AES-256-GCM ciphertext. Discord's
 index tokenises words; there are none. Making the payloads searchable
 would mean storing them in the clear on somebody else's servers, which
 is a much larger concession than the one on the front of the README.

So SQLite remains the query layer, and this exists for the things search is genuinely better at than a channel scan: finding our own messages across a guild without knowing which channel they are in. The metadata it matches on -- the DS1 framing prefix and the .ds1 attachment extension -- is plaintext already, and deliberately says nothing about the contents.

See Also:

Defined Under Namespace

Classes: Result

Constant Summary collapse

MARKER =

The plaintext framing prefix every message this library writes begins with. Not a secret, and not informative: it identifies the format, not the contents.

"DS1"
EXTENSION =

Attachment extension used by both blob chunks and spilled log records.

"ds1"
PAGE_SIZE =

One page. Discord's own maximum.

Transport::REST::SEARCH_PAGE_LIMIT

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(rest:, config:, guild_id: nil) ⇒ Search

Returns a new instance of Search.

Parameters:

Raises:



77
78
79
80
81
82
83
 
# File 'lib/discord_store/search.rb', line 77

def initialize(rest:, config:, guild_id: nil)
  @rest = rest
  @config = config
  @guild_id = (guild_id || config.guild_id)&.to_s

  raise ConfigurationError, "guild_id is required to search" if @guild_id.nil? || @guild_id.empty?
end

Instance Attribute Details

#config ⇒ Object (readonly)

Returns the value of attribute config.



72
73
74
 
# File 'lib/discord_store/search.rb', line 72

def config
  @config
end

Instance Method Details

#attachments(**options) ⇒ Result

Every attachment this library uploaded: blob chunks and spilled records.

Returns:



138
139
140
 
# File 'lib/discord_store/search.rb', line 138

def attachments(**options)
  messages(has: "file", attachment_extension: EXTENSION, **options)
end

#each(**options) {|message| ... } ⇒ Enumerator, void

Pages through results.

Pages by offset rather than by the size of the last page, because Discord says not to paginate on that size. Stops at the offset ceiling with an explicit error rather than quietly returning a truncated set.

Yield Parameters:

  • message (Hash)

Returns:

  • (Enumerator, void) 


150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/discord_store/search.rb', line 150

def each(**options, &block)
  return enum_for(:each, **options) unless block

  offset = options.delete(:offset).to_i
  seen = 0

  loop do
    page = messages(**options, offset: offset, limit: PAGE_SIZE)
    page.each(&block)

    seen += page.size
    offset += PAGE_SIZE

    break if page.empty? || seen >= page.total_results
    break if exhausted?(offset, page)
  end
end

#messages(content: nil, channel_ids: nil, after: nil, before: nil, has: nil, attachment_extension: nil, attachment_filename: nil, pinned: nil, sort_by: nil, sort_order: nil, limit: PAGE_SIZE, offset: 0, author_id: nil) ⇒ Result

Searches for messages this bot wrote.

Parameters:

  • content (String, nil) (defaults to: nil) —

    full-text filter; only ever matches the plaintext framing, never a payload

  • channel_ids (Array<String>, nil) (defaults to: nil) —

    restrict to these channels

  • after (Time, String, Integer, nil) (defaults to: nil) —

    exclusive lower bound

  • before (Time, String, Integer, nil) (defaults to: nil) —

    exclusive upper bound

  • has (Array<String>, String, nil) (defaults to: nil) —

    e.g. "file"

  • attachment_extension (String, nil) (defaults to: nil)
  • attachment_filename (String, nil) (defaults to: nil)
  • pinned (Boolean, nil) (defaults to: nil)
  • sort_by (String, nil) (defaults to: nil) —

    "timestamp" or "relevance"

  • sort_order (String, nil) (defaults to: nil) —

    "asc" or "desc"

  • limit (Integer) (defaults to: PAGE_SIZE)
  • offset (Integer) (defaults to: 0)
  • author_id (Array<String>, nil) (defaults to: nil) —

    refused while own_messages_only is on

Returns:



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
# File 'lib/discord_store/search.rb', line 102

def messages(content: nil, channel_ids: nil, after: nil, before: nil, has: nil,
             attachment_extension: nil, attachment_filename: nil, pinned: nil,
             sort_by: nil, sort_order: nil, limit: PAGE_SIZE, offset: 0, author_id: nil)
  params = {
    content: content,
    channel_id: Array(channel_ids).map(&:to_s),
    min_id: bound(after),
    max_id: bound(before),
    has: Array(has),
    attachment_extension: Array(attachment_extension),
    attachment_filename: Array(attachment_filename),
    pinned: pinned,
    sort_by: sort_by,
    sort_order: sort_order,
    limit: limit.to_i.clamp(1, PAGE_SIZE),
    offset: assert_offset!(offset),
    author_id: scope_authors(author_id)
  }

  body = @rest.search_messages(@guild_id, **params)

  Result.new(
    messages: body["messages"],
    total_results: body["total_results"],
    indexing: body["indexing"]
  )
end

#ours(**options) ⇒ Result

Every message this library wrote, anywhere in the guild.

Returns:



133
 
# File 'lib/discord_store/search.rb', line 133

def ours(**options) = messages(content: MARKER, **options)