Searchkick

:rocket: Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

stemming - tomatoes matches tomato
special characters - jalapeno matches jalapeño
extra whitespace - dishwasher matches dish washer
misspellings - zuchini matches zucchini
custom synonyms - pop matches soda

Plus:

query like SQL - no need to learn a new query language
reindex without downtime
easily personalize results for each user
autocomplete
“Did you mean” suggestions
supports many languages
works with ActiveRecord, Mongoid, and NoBrainer

Check out Searchjoy for analytics and Autosuggest for query suggestions

:tangerine: Battle-tested at Instacart

Getting Started
Querying
Indexing
Intelligent Search
Instant Search / Autocomplete
Aggregations
Testing
Deployment
Performance
Elasticsearch DSL
Reference
Contributing

Getting Started

Install Elasticsearch. For Homebrew, use:

brew install elasticsearch
brew services start elasticsearch

Add this line to your application’s Gemfile:

gem 'searchkick'

The latest version works with Elasticsearch 6 and 7. For Elasticsearch 5, use version 3.1.3 and this readme.

Add searchkick to models you want to search.

class Product < ApplicationRecord
  searchkick
end

Add data to the search index.

Product.reindex

And to query, use:

products = Product.search("apples")
products.each do |product|
  puts product.name
end

Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.

Querying

Query like SQL

Product.search "apples", where: {in_stock: true}, limit: 10, offset: 50

Search specific fields

fields: [:name, :brand]

Where

where: {
  expires_at: {gt: Time.now},   # lt, gte, lte also available
  orders_count: 1..10,          # equivalent to {gte: 1, lte: 10}
  aisle_id: [25, 30],           # in
  store_id: {not: 2},           # not
  aisle_id: {not: [25, 30]},    # not in
  user_ids: {all: [1, 3]},      # all elements in array
  category: {like: "%frozen%"}, # like
  category: /frozen .+/,        # regexp
  category: {prefix: "frozen"}, # prefix
  store_id: {exists: true},     # exists
  _or: [{in_stock: true}, {backordered: true}],
  _and: [{in_stock: true}, {backordered: true}],
  _not: {store_id: 1}           # negate a condition
}

Order

order: {_score: :desc} # most relevant first - default

All of these sort options are supported

Limit / offset

limit: 20, offset: 40

Select

select: [:name]

These source filtering options are supported

Results

Searches return a Searchkick::Results object. This responds like an array to most methods.

results = Product.search("milk")
results.size
results.any?
results.each { |result| ... }

By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:

Product.search("apples", load: false)

Get total results

results.total_count

Get the time the search took (in milliseconds)

results.took

Get the full response from Elasticsearch

results.response

Note: By default, Elasticsearch limits paging to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.

Boosting

Boost important fields

fields: ["title^10", "description"]

Boost by the value of a field (field must be numeric)

boost_by: [:orders_count] # give popular documents a little boost
boost_by: {orders_count: {factor: 10}} # default factor is 1

Boost matching documents

boost_where: {user_id: 1}
boost_where: {user_id: {value: 1, factor: 100}} # default factor is 1000
boost_where: {user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}]}

Boost by recency

boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}

You can also boost by:

Conversions
Distance

Get Everything

Use a * for the query.

Product.search "*"

Pagination

Plays nicely with kaminari and will_paginate.

# controller
@products = Product.search "milk", page: params[:page], per_page: 20

View with kaminari

<%= paginate @products %>

View with will_paginate

<%= will_paginate @products %>

Partial Matches

By default, results must match all words in the query.

Product.search "fresh honey" # fresh AND honey

To change this, use:

Product.search "fresh honey", operator: "or" # fresh OR honey

By default, results must match the entire word - back will not match backpack. You can change this behavior with:

class Product < ApplicationRecord
  searchkick word_start: [:name]
end

And to search (after you reindex):

Product.search "back", fields: [:name], match: :word_start

Available options are:

Option	Matches	Example
`:word`	entire word	`apple` matches `apple`
`:word_start`	start of word	`app` matches `apple`
`:word_middle`	any part of word	`ppl` matches `apple`
`:word_end`	end of word	`ple` matches `apple`
`:text_start`	start of text	`gre` matches `green apple`, `app` does not match
`:text_middle`	any part of text	`een app` matches `green apple`
`:text_end`	end of text	`ple` matches `green apple`, `een` does not match

The default is :word. The most matches will happen with :word_middle.

Exact Matches

To match a field exactly (case-sensitive), use:

User.search query, fields: [{email: :exact}, :name]

Phrase Matches

To only match the exact order, use:

User.search "fresh honey", match: :phrase

Stemming and Language

Searchkick stems words by default for better matching. apple and apples both stem to appl, so searches for either term will have the same matches.

Searchkick defaults to English for stemming. To change this, use:

class Product < ApplicationRecord
  searchkick language: "german"
end

See the list of stemmers. A few languages require plugins:

chinese - analysis-ik plugin
chinese2 - analysis-smartcn plugin
japanese - analysis-kuromoji plugin
korean - analysis-openkoreantext plugin
korean2 - analysis-nori plugin
polish - analysis-stempel plugin
ukrainian - analysis-ukrainian plugin
vietnamese - analysis-vietnamese plugin

Disable stemming with:

class Image < ApplicationRecord
  searchkick stem: false
end

Exclude certain words from stemming with:

class Image < ApplicationRecord
  searchkick stem_exclusion: ["apples"]
end

Or change how words are stemmed:

class Image < ApplicationRecord
  searchkick stemmer_override: ["apples => other"]
end

Synonyms

class Product < ApplicationRecord
  searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]]
end

Call Product.reindex after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.

For directional synonyms, use:

search_synonyms: ["lightbulb => halogenlamp"]

Dynamic Synonyms

The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.

Elasticsearch 7.3+

For Elasticsearch 7.3+, we recommend placing synonyms in a file on the Elasticsearch server (in the config directory). This allows you to reload synonyms without reindexing.

pop, soda
burger, hamburger

Then use:

search_synonyms: "synonyms.txt"

Add elasticsearch-xpack to your Gemfile:

gem 'elasticsearch-xpack', '>= 7.8.0'

And use:

Product.search_index.reload_synonyms

Elasticsearch < 7.3

You can use a library like ActsAsTaggableOn and do:

class Product < ApplicationRecord
  acts_as_taggable
  scope :search_import, -> { includes(:tags) }

  def search_data
    {
      name_tagged: "#{name} #{tags.map(&:name).join(" ")}"
    }
  end
end

Search with:

Product.search query, fields: [:name_tagged]

Misspellings

By default, Searchkick handles misspelled queries by returning results with an edit distance of one.

You can change this with:

Product.search "zucini", misspellings: {edit_distance: 2} # zucchini

To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.

Product.search "zuchini", misspellings: {below: 5}

If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.

Turn off misspellings with:

Product.search "zuchini", misspellings: false # no zucchini

Specify which fields can include misspellings with:

Product.search "zucini", fields: [:name, :color], misspellings: {fields: [:name]}

When doing this, you must also specify fields to search

Bad Matches

If a user searches butter, they may also get results for peanut butter. To prevent this, use:

Product.search "butter", exclude: ["peanut butter"]

You can map queries and terms to exclude with:

exclude_queries = {
  "butter" => ["peanut butter"],
  "cream" => ["ice cream", "whipped cream"]
}

Product.search query, exclude: exclude_queries[query]

You can demote results by boosting by a factor less than one:

Product.search("butter", boost_where: {category: {value: "pantry", factor: 0.5}})

Emoji

Search :ice_cream::cake: and get ice cream cake!

Add this line to your application’s Gemfile:

gem 'gemoji-parser'

And use:

Product.search "🍨🍰", emoji: true

Indexing

Control what data is indexed with the search_data method. Call Product.reindex after changing this method.

class Product < ApplicationRecord
  belongs_to :department

  def search_data
    {
      name: name,
      department_name: department.name,
      on_sale: sale_price.present?
    }
  end
end

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

class Product < ApplicationRecord
  scope :search_import, -> { includes(:department) }
end

By default, all records are indexed. To control which records are indexed, use the should_index? method together with the search_import scope.

class Product < ApplicationRecord
  scope :search_import, -> { where(active: true) }

  def should_index?
    active # only index active records
  end
end

If a reindex is interrupted, you can resume it with:

Product.reindex(resume: true)

For large data sets, try parallel reindexing.

To Reindex, or Not to Reindex

Reindex

when you install or upgrade searchkick
change the search_data method
change the searchkick method

No need to reindex

app starts

Strategies

There are four strategies for keeping the index synced with your database.

Inline (default)

Anytime a record is inserted, updated, or deleted

Asynchronous

Use background jobs for better performance

  class Product < ApplicationRecord
    searchkick callbacks: :async
  end

Jobs are added to a queue named searchkick.

Queuing

Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See how to set up.

Manual

Turn off automatic syncing

  class Product < ApplicationRecord
    searchkick callbacks: false
  end

You can also do bulk updates.

Searchkick.callbacks(:bulk) do
  User.find_each(&:update_fields)
end

Or temporarily skip updates.

Searchkick.callbacks(false) do
  User.find_each(&:update_fields)
end

Associations

Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:

class Image < ApplicationRecord
  belongs_to :product

  after_commit :reindex_product

  def reindex_product
    product.reindex
  end
end

Intelligent Search

The best starting point to improve your search by far is to track searches and conversions. Searchjoy makes it easy.

Product.search "apple", track: {user_id: current_user.id}

See the docs for how to install and use.

Focus on:

top searches with low conversions
top searches with no results

Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.

Add conversion data with:

class Product < ApplicationRecord
  has_many :searches, class_name: "Searchjoy::Search", as: :convertable

  searchkick conversions: [:conversions] # name of field

  def search_data
    {
      name: name,
      conversions: searches.group(:query).uniq.count(:user_id)
      # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
    }
  end
end

Reindex and set up a cron job to add new conversions daily.

rake searchkick:reindex CLASS=Product

This can make a huge difference on the quality of your search.

For a more performant way to reindex conversion data, check out performant conversions.

Personalized Results

Order results differently for each user. For example, show a user’s previously purchased products before other results.

class Product < ApplicationRecord
  def search_data
    {
      name: name,
      orderer_ids: orders.pluck(:user_id) # boost this product for these users
    }
  end
end

Reindex and search with:

Product.search "milk", boost_where: {orderer_ids: current_user.id}

Instant Search / Autocomplete

Autocomplete predicts what a user will type, making the search experience faster and easier.

Autocomplete

Note: To autocomplete on search terms rather than results, check out Autosuggest.

Note 2: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).

First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.

class Movie < ApplicationRecord
  searchkick word_start: [:title, :director]
end

Reindex and search with:

Movie.search "jurassic pa", fields: [:title], match: :word_start

Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.

Here’s how to make it work with Rails

First, add a route and controller action.

class MoviesController < ApplicationController
  def autocomplete
    render json: Movie.search(params[:query], {
      fields: ["title^5", "director"],
      match: :word_start,
      limit: 10,
      load: false,
      misspellings: {below: 5}
    }).map(&:title)
  end
end

Note: Use load: false and misspellings: {below: n} (or misspellings: false) for best performance.

Then add the search box and JavaScript code to a view.

<input type="text" id="query" name="query" />

<script src="jquery.js"></script>
<script src="typeahead.bundle.js"></script>
<script>
  var movies = new Bloodhound({
    datumTokenizer: Bloodhound.tokenizers.whitespace,
    queryTokenizer: Bloodhound.tokenizers.whitespace,
    remote: {
      url: '/movies/autocomplete?query=%QUERY',
      wildcard: '%QUERY'
    }
  });
  $('#query').typeahead(null, {
    source: movies
  });
</script>

Suggestions

Suggest

class Product < ApplicationRecord
  searchkick suggest: [:name] # fields to generate suggestions
end

Reindex and search with:

products = Product.search "peantu butta", suggest: true
products.suggestions # ["peanut butter"]

Aggregations

Aggregations provide aggregated search data.

Aggregations

products = Product.search "chuck taylor", aggs: [:product_type, :gender, :brand]
products.aggs

By default, where conditions apply to aggregations.

Product.search "wingtips", where: {color: "brandy"}, aggs: [:size]
# aggregations for brandy wingtips are returned

Change this with:

Product.search "wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false
# aggregations for all wingtips are returned

Set where conditions for each aggregation separately with:

Product.search "wingtips", aggs: {size: {where: {color: "brandy"}}}

Limit

Product.search "apples", aggs: {store_id: {limit: 10}}

Order

Product.search "wingtips", aggs: {color: {order: {"_key" => "asc"}}} # alphabetically

All of these options are supported

Ranges

price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search "*", aggs: {price: {ranges: price_ranges}}

Minimum document count

Product.search "apples", aggs: {store_id: {min_doc_count: 2}}

Script support

Product.search "*", aggs: {color: {script: {source: "'Color: ' + _value"}}}

Date histogram

Product.search "pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}}

For other aggregation types, including sub-aggregations, use body_options:

Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}}

Highlight

Specify which fields to index with highlighting.

class Product < ApplicationRecord
  searchkick highlight: [:name]
end

Highlight the search query in the results.

bands = Band.search "cinema", highlight: true

View the highlighted fields with:

bands.with_highlights.each do |band, highlights|
  highlights[:name] # "Two Door <em>Cinema</em> Club"
end

To change the tag, use:

Band.search "cinema", highlight: {tag: "<strong>"}

To highlight and search different fields, use:

Band.search "cinema", fields: [:name], highlight: {fields: [:description]}

By default, the entire field is highlighted. To get small snippets instead, use:

bands = Band.search "cinema", highlight: {fragment_size: 20}
bands.with_highlights(multiple: true).each do |band, highlights|
  highlights[:name].join(" and ")
end

Additional options can be specified for each field:

Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}}

You can find available highlight options in the Elasticsearch reference.

Similar Items

Find similar items.

product = Product.first
product.similar(fields: [:name], where: {size: "12 oz"})

Geospatial Searches

class Restaurant < ApplicationRecord
  searchkick locations: [:location]

  def search_data
    attributes.merge(location: {lat: latitude, lon: longitude})
  end
end

Reindex and search with:

Restaurant.search "pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}} # or 160km

Bounded by a box

Restaurant.search "sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}}

Note: top_right and bottom_left also work

Bounded by a polygon

Restaurant.search "dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}}

Boost By Distance

Boost results by distance - closer results are boosted more

Restaurant.search "noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}}

Also supports additional options

Restaurant.search "wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}}

Geo Shapes

You can also index and search geo shapes.

class Restaurant < ApplicationRecord
  searchkick geo_shape: [:bounds]

  def search_data
    attributes.merge(
      bounds: {
        type: "envelope",
        coordinates: [{lat: 4, lon: 1}, {lat: 2, lon: 3}]
      }
    )
  end
end

See the Elasticsearch documentation for details.

Find shapes intersecting with the query shape

Restaurant.search "soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}}

Falling entirely within the query shape

Restaurant.search "salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: [{lat: 38, lon: -123}], radius: "1km"}}}

Not touching the query shape

Restaurant.search "burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}}

Inheritance

Searchkick supports single table inheritance.

class Dog < Animal
end

In your parent model, set:

class Animal < ApplicationRecord
  searchkick inheritance: true
end

The parent and child model can both reindex.

Animal.reindex
Dog.reindex # equivalent, all animals reindexed

And to search, use:

Animal.search "*"                   # all animals
Dog.search "*"                      # just dogs
Animal.search "*", type: [Dog, Cat] # just cats and dogs

Notes:

The suggest option retrieves suggestions from the parent at the moment.

Dog.search "airbudd", suggest: true # suggestions for all animals

This relies on a type field that is automatically added to the indexed document. Be wary of defining your own type field in search_data, as it will take precedence.

Debugging Queries

To help with debugging queries, you can use:

Product.search("soap", debug: true)

This prints useful info to stdout.

See how Elasticsearch scores your queries with:

Product.search("soap", explain: true).response

See how Elasticsearch tokenizes your queries with:

Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
# ["dish", "dishwash", "washer", "washersoap", "soap"]

Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search")
# ["dishwashersoap"] - no match

Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
# ["dishwash", "soap"] - match!!

Partial matches

Product.search_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]

Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
# ["dieg"] - match!!

See the complete list of analyzers.

Testing

As you iterate on your search, it’s a good idea to add tests.

For performance, only enable Searchkick callbacks for the tests that need it.

Parallel Tests

Rails 6 enables parallel tests by default. Add to your test/test_helper.rb:

class ActiveSupport::TestCase
  parallelize_setup do |worker|
    Searchkick.index_suffix = worker

    # reindex models
    Product.reindex

    # and disable callbacks
    Searchkick.disable_callbacks
  end
end

And use:

class ProductTest < ActiveSupport::TestCase
  def setup
    Searchkick.enable_callbacks
  end

  def teardown
    Searchkick.disable_callbacks
  end

  def test_search
    Product.create!(name: "Apple")
    Product.search_index.refresh
    assert_equal ["Apple"], Product.search("apple").map(&:name)
  end
end

Minitest

Add to your test/test_helper.rb:

# reindex models
Product.reindex

# and disable callbacks
Searchkick.disable_callbacks

And use:

class ProductTest < Minitest::Test
  def setup
    Searchkick.enable_callbacks
  end

  def teardown
    Searchkick.disable_callbacks
  end

  def test_search
    Product.create!(name: "Apple")
    Product.search_index.refresh
    assert_equal ["Apple"], Product.search("apple").map(&:name)
  end
end

RSpec

Add to your spec/spec_helper.rb:

RSpec.configure do |config|
  config.before(:suite) do
    # reindex models
    Product.reindex

    # and disable callbacks
    Searchkick.disable_callbacks
  end

  config.around(:each, search: true) do |example|
    Searchkick.callbacks(nil) do
      example.run
    end
  end
end

And use:

describe Product, search: true do
  it "searches" do
    Product.create!(name: "Apple")
    Product.search_index.refresh
    assert_equal ["Apple"], Product.search("apple").map(&:name)
  end
end

Factory Bot

Use a trait and an after create hook for each indexed model:

FactoryBot.define do
  factory :product do
    # ...

    # Note: This should be the last trait in the list so `reindex` is called
    # after all the other callbacks complete.
    trait :reindex do
      after(:create) do |product, _evaluator|
        product.reindex(refresh: true)
      end
    end
  end
end

# use it
FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")

GitHub Actions

Check out setup-elasticsearch for an easy way to install Elasticsearch.

    - uses: ankane/setup-elasticsearch@v1

Deployment

Searchkick uses ENV["ELASTICSEARCH_URL"] for the Elasticsearch server. This defaults to http://localhost:9200.

Elastic Cloud
Heroku
Amazon Elasticsearch Service
Self-Hosted and Other

Elastic Cloud

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"

Then deploy and reindex:

rake searchkick:reindex:all

Heroku

Choose an add-on: Bonsai, SearchBox, or Elastic Cloud.

For Bonsai:

heroku addons:create bonsai
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`

For SearchBox:

heroku addons:create searchbox:starter
heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`

For Elastic Cloud (previously Found):

heroku addons:create foundelasticsearch
heroku addons:open foundelasticsearch

Visit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:

heroku config:get FOUNDELASTICSEARCH_URL

And add elastic:password@ right after https:// and add port 9243 at the end:

heroku config:set ELASTICSEARCH_URL=https://elastic:[email protected]:9243

Then deploy and reindex:

heroku run rake searchkick:reindex:all

Amazon Elasticsearch Service

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"

To use signed requests, include in your Gemfile:

gem 'faraday_middleware-aws-sigv4'

and add to your initializer:

Searchkick.aws_credentials = {
  access_key_id: ENV["AWS_ACCESS_KEY_ID"],
  secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
  region: "us-east-1"
}

Then deploy and reindex:

rake searchkick:reindex:all

Self-Hosted and Other

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"

Then deploy and reindex:

rake searchkick:reindex:all

Data Protection

We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send personal data of your users to Elasticsearch.

Bonsai, Elastic Cloud, and Amazon Elasticsearch all support encryption at rest and HTTPS.

Automatic Failover

Create an initializer config/initializers/elasticsearch.rb with multiple hosts:

ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"

See elasticsearch-transport for a complete list of options.

Lograge

Add the following to config/environments/production.rb:

config.lograge.custom_options = lambda do |event|
  options = {}
  options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0
  options
end

See Production Rails for other good practices.

Performance

JSON Generation

Significantly increase performance with faster JSON generation. Add Oj to your Gemfile.

gem 'oj'

This speeds up all JSON generation and parsing in your application (automatically!)

Persistent HTTP Connections

Significantly increase performance with persistent HTTP connections. Add Typhoeus to your Gemfile and it’ll automatically be used.

gem 'typhoeus'

To reduce log noise, create an initializer with:

Ethon.logger = Logger.new(nil)

If you run into issues on Windows, check out this post.

Searchable Fields

By default, all string fields are searchable (can be used in fields option). Speed up indexing and reduce index size by only making some fields searchable.

class Product < ApplicationRecord
  searchkick searchable: [:name]
end

Filterable Fields

By default, all string fields are filterable (can be used in where option). Speed up indexing and reduce index size by only making some fields filterable.

class Product < ApplicationRecord
  searchkick filterable: [:brand]
end

Note: Non-string fields are always filterable and should not be passed to this option.

Parallel Reindexing

For large data sets, you can use background jobs to parallelize reindexing.

Product.reindex(async: true)
# {index_name: "products_production_20170111210018065"}

Once the jobs complete, promote the new index with:

Product.search_index.promote(index_name)

You can optionally track the status with Redis:

Searchkick.redis = Redis.new

And use:

Searchkick.reindex_status(index_name)

You can also have Searchkick wait for reindexing to complete

Product.reindex(async: {wait: true})

You can use ActiveJob::TrafficControl to control concurrency. Install the gem:

gem 'activejob-traffic_control', '>= 0.1.3'

And create an initializer with:

ActiveJob::TrafficControl.client = Searchkick.redis

class Searchkick::BulkReindexJob
  concurrency 3
end

This will allow only 3 jobs to run at once.

Refresh Interval

You can specify a longer refresh interval while reindexing to increase performance.

Product.reindex(async: true, refresh_interval: "30s")

Note: This only makes a noticable difference with parallel reindexing.

When promoting, have it restored to the value in your mapping (defaults to 1s).

Product.search_index.promote(index_name, update_refresh_interval: true)

Queuing

Push ids of records needing reindexed to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using connection_pool.

Searchkick.redis = ConnectionPool.new { Redis.new }

And ask your models to queue updates.

class Product < ApplicationRecord
  searchkick callbacks: :queue
end

Then, set up a background job to run.

Searchkick::ProcessQueueJob.perform_later(class_name: "Product")

You can check the queue length with:

Product.search_index.reindex_queue.length

For more tips, check out Keeping Elasticsearch in Sync.

Routing

Searchkick supports Elasticsearch’s routing feature, which can significantly speed up searches.

class Business < ApplicationRecord
  searchkick routing: true

  def search_routing
    city_id
  end
end

Reindex and search with:

Business.search "ice cream", routing: params[:city_id]

Partial Reindexing

Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.

class Product < ApplicationRecord
  def search_data
    {
      name: name
    }.merge(search_prices)
  end

  def search_prices
    {
      price: price,
      sale_price: sale_price
    }
  end
end

And use:

Product.reindex(:search_prices)

Performant Conversions

Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.

class Product < ApplicationRecord
  def search_data
    {
      name: name
    }.merge(search_conversions)
  end

  def search_conversions
    {
      conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
    }
  end
end

Create a job to update the cache and reindex records with new conversions.

class ReindexConversionsJob < ApplicationJob
  def perform(class_name)
    # get records that have a recent conversion
    recently_converted_ids =
      Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
      .order(:convertable_id).uniq.pluck(:convertable_id)

    # split into groups
    recently_converted_ids.in_groups_of(1000, false) do |ids|
      # fetch conversions
      conversions =
        Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
        .group(:convertable_id, :query).uniq.count(:user_id)

      # group conversions by record
      conversions_by_record = {}
      conversions.each do |(id, query), count|
        (conversions_by_record[id] ||= {})[query] = count
      end

      # write to cache
      conversions_by_record.each do |id, conversions|
        Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
      end

      # partial reindex
      class_name.constantize.where(id: ids).reindex(:search_conversions)
    end
  end
end

Run the job with:

ReindexConversionsJob.perform_later("Product")

Advanced

Searchkick makes it easy to use the Elasticsearch DSL on its own.

Advanced Mapping

Create a custom mapping:

class Product < ApplicationRecord
  searchkick mappings: {
    properties: {
      name: {type: "keyword"}
    }
  }
end

Note: If you use a custom mapping, you'll need to use custom searching as well.

To keep the mappings and settings generated by Searchkick, use:

class Product < ApplicationRecord
  searchkick merge_mappings: true, mappings: {...}
end

Advanced Search

And use the body option to search:

products = Product.search body: {query: {match: {name: "milk"}}}

View the response with:

products.response

To modify the query generated by Searchkick, use:

products = Product.search "milk", body_options: {min_score: 1}

products =
  Product.search "apples" do |body|
    body[:min_score] = 1
  end

Elasticsearch Gem

Searchkick is built on top of the elasticsearch gem. To access the client directly, use:

Searchkick.client

Multi Search

To batch search requests for performance, use:

products = Product.search("snacks", execute: false)
coupons = Coupon.search("snacks", execute: false)
Searchkick.multi_search([products, coupons])

Then use products and coupons as typical results.

Note: Errors are not raised as with single requests. Use the error method on each query to check for errors.

Multiple Models

Search across multiple models with:

Searchkick.search "milk", models: [Product, Category]

Boost specific models with:

indices_boost: {Category => 2, Product => 1}

Multi-Tenancy

Check out this great post on the Apartment gem. Follow a similar pattern if you use another gem.

Scroll API

Searchkick also supports the scroll API. Scrolling is not intended for real time user requests, but rather for processing large amounts of data.

Product.search("*", scroll: "1m").scroll do |batch|
  # process batch ...
end

You can also scroll batches manually.

products = Product.search "*", scroll: "1m"
while products.any?
  # process batch ...

  products = products.scroll
end

products.clear_scroll

Deep Paging

By default, Elasticsearch limits paging to the first 10,000 results. Here’s why. We don’t recommend changing this, but if you really need all results, you can use:

class Product < ApplicationRecord
  searchkick deep_paging: true
end

If you just need an accurate total count with Elasticsearch 7, you can instead use:

Product.search("pears", body_options: {track_total_hits: true})

Nested Data

To query nested data, use dot notation.

User.search "san", fields: ["address.city"], where: {"address.zip_code" => 12345}

Reference

Reindex one record

product = Product.find(1)
product.reindex

Reindex multiple records

Product.where(store_id: 1).reindex

Reindex associations

store.products.reindex

Remove old indices

Product.search_index.clean_indices

Use custom settings

class Product < ApplicationRecord
  searchkick settings: {number_of_shards: 3}
end

Use a different index name

class Product < ApplicationRecord
  searchkick index_name: "products_v2"
end

Use a dynamic index name

class Product < ApplicationRecord
  searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" }
end

Prefix the index name

class Product < ApplicationRecord
  searchkick index_prefix: "datakick"
end

For all models

Searchkick.index_prefix = "datakick"

Use a different term for boosting by conversions

Product.search("banana", conversions_term: "organic banana")

Multiple conversion fields

class Product < ApplicationRecord
  has_many :searches, class_name: "Searchjoy::Search"

  # searchkick also supports multiple "conversions" fields
  searchkick conversions: ["unique_user_conversions", "total_conversions"]

  def search_data
    {
      name: name,
      unique_user_conversions: searches.group(:query).uniq.count(:user_id),
      # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
      total_conversions: searches.group(:query).count
      # {"ice cream" => 412, "chocolate" => 117, "cream" => 6}
    }
  end
end

and during query time:

Product.search("banana") # boost by both fields (default)
Product.search("banana", conversions: "total_conversions") # only boost by total_conversions
Product.search("banana", conversions: false) # no conversion boosting

Change timeout

Searchkick.timeout = 15 # defaults to 10

Set a lower timeout for searches

Searchkick.search_timeout = 3

Change the search method name

Searchkick.search_method_name = :lookup

Change search queue name

Searchkick.queue_name = :search_reindex

Eager load associations

Product.search "milk", includes: [:brand, :stores]

Eager load different associations by model

Searchkick.search("*",  models: [Product, Store], model_includes: {Product => [:store], Store => [:product]})

Run additional scopes on results

Product.search "milk", scope_results: ->(r) { r.with_attached_images }

Specify default fields to search

class Product < ApplicationRecord
  searchkick default_fields: [:name]
end

Turn off special characters

class Product < ApplicationRecord
  # A will not match Ä
  searchkick special_characters: false
end

Turn on stemming for conversions

class Product < ApplicationRecord
  searchkick stem_conversions: true
end

Make search case-sensitive

class Product < ApplicationRecord
  searchkick case_sensitive: true
end

Note: If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.

Change import batch size

class Product < ApplicationRecord
  searchkick batch_size: 200 # defaults to 1000
end

Create index without importing

Product.reindex(import: false)

Use a different id

class Product < ApplicationRecord
  def search_document_id
    custom_id
  end
end

Lazy searching

products = Product.search("carrots", execute: false)
products.each { ... } # search not executed until here

Add request parameters, like search_type and query_cache

Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})

Set options across all models

Searchkick.model_options = {
  batch_size: 200
}

Reindex conditionally

class Product < ApplicationRecord
  searchkick callbacks: false

  # add the callbacks manually
  after_commit :reindex, if: -> (model) { model.previous_changes.key?("name") } # use your own condition
end

Reindex all models - Rails only

rake searchkick:reindex:all

Turn on misspellings after a certain number of characters

Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi

Note: With this option, if the query length is the same as prefix_length, misspellings are turned off

Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha

Elasticsearch 6 to 7 Upgrade

Install Searchkick 4
Upgrade your Elasticsearch cluster

Elasticsearch Gotchas

Consistency

Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the refresh method to have it show up immediately.

product.save!
Product.search_index.refresh

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

class Product < ApplicationRecord
  searchkick settings: {number_of_shards: 1}
end

For convenience, this is set by default in the test environment.

History

View the changelog.

Thanks

Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

Report bugs
Fix bugs and submit pull requests
Write, clarify, or fix documentation
Suggest or add new features

To get started with development:

git clone https://github.com/ankane/searchkick.git
cd searchkick
bundle install
bundle exec rake test

Feel free to open an issue to get feedback on your idea before spending too much time on it.

Searchkick

Contents

Getting Started

Querying

Results

Boosting

Get Everything

Pagination

Partial Matches

Exact Matches

Phrase Matches

Stemming and Language

Synonyms

Dynamic Synonyms

Elasticsearch 7.3+

Elasticsearch < 7.3

Misspellings

Bad Matches

Emoji

Indexing

To Reindex, or Not to Reindex

Reindex

No need to reindex

Strategies

Associations

Intelligent Search

Personalized Results

Instant Search / Autocomplete

Here’s how to make it work with Rails

Suggestions

Aggregations

Highlight

Similar Items

Geospatial Searches

Boost By Distance

Geo Shapes

Inheritance

Debugging Queries

Testing

Parallel Tests

Minitest

RSpec

Factory Bot

GitHub Actions

Deployment

Elastic Cloud

Heroku

Amazon Elasticsearch Service

Self-Hosted and Other

Data Protection

Automatic Failover

Lograge

Performance

JSON Generation

Persistent HTTP Connections

Searchable Fields

Filterable Fields

Parallel Reindexing

Refresh Interval

Queuing

Routing

Partial Reindexing

Performant Conversions

Advanced

Advanced Mapping

Advanced Search

Elasticsearch Gem

Multi Search

Multiple Models

Multi-Tenancy

Scroll API

Deep Paging

Nested Data

Reference

Elasticsearch 6 to 7 Upgrade

Elasticsearch Gotchas

Consistency

Inconsistent Scores

History

Thanks