Scheddy

Scheddy is a batteries-included task scheduler for Rails. It is intended as a replacement for cron and cron-like functionality (including job queue specific schedulers), with some useful differences.

Flexible scheduling. Handles fixed times (Monday at 9am), intervals (every 15 minutes), and tiny intervals (every 5 seconds).
Tiny intervals are great for scheduling workload specific jobs (database field next_run_at).
Catch up missed tasks. Designed for environments with frequent deploys. Also useful in dev where the scheduler isn't always running.
Job-queue agnostic. Works great with various ActiveJob adapters and non-ActiveJob queues too.
Minimal dependencies. Uses your existing database (or no database at all). Redis not required either.
Built-in cluster support by running 2+ Scheddy instances. (Database required in this case.)
Tasks and their schedules are versioned as part of your code.

Installation

Add to your application's Gemfile:

gem "scheddy"

After running bundle install, add the migrations to your app:

bin/rails scheddy:install:migrations
bin/rails db:migrate

FYI, if all tasks set track_runs false and running only a single Scheddy instance, the migrations are optional.

Usage

Scheddy is configured with a straightforward DSL.

For clarity, Scheddy's units of work are referred to as Tasks. This is to differentiate them from background queue Jobs, like those run via ActiveJob. Scheddy's tasks have no relation to rake tasks.

Start by creating config/initializers/scheddy.rb:

Scheddy.config do

  ## Fixed times
  task 'monday reports' do
    run_at '0 9 * * mon'  # cron syntax
    # run_at 'monday 9am' # use fugit's natural language parsing
    # track_runs false    # defaults to true for run_at() jobs
    perform do
      ReportJob.perform_later
    end
  end

  task 'tuesday reports' do
    run_when day: :tue, hour: 9..16, minute: [0,30]
      # a native ruby syntax is also supported
      #   :day    - day of week
      #   :month
      #   :date   - day of month
      #   :hour
      #   :minute
      #   :second
      # all values default to '*' (except second, which defaults to 0)
    # track_runs false    # defaults to true for run_when() jobs
    perform do
      AnotherReportJob.perform_later
    end
  end

  ## Intervals
  task 'send welcome emails' do
    run_every 30.minutes
    # track_runs false    # when run_every is >= 15.minutes, defaults to true; else to false
    perform do
      User.where(welcome_email_at: nil).find_each(batch_size: 100) do |user|
        WelcomeMailer.with(user: user).welcome_email.deliver_later
      end
    end
  end

  task 'heartbeat' do
    run_every 300  # seconds may be used instead
    perform 'HeartbeatJob.perform_later'  # a string to eval may be used too
  end

  # Use tiny intervals for lightweight scanning for ready-to-work records
  task 'disable expired accounts' do
    run_every 15.seconds
    logger_tag 'expired-scan'  # tag log lines with an alternate value; nil disables tagging
    perform do
      Subscription.expired.pluck(:id).each do |id|
        DisableAccountJob.perform_later id
      end
    end
  end

end

Fixed times: `run_at` and `run_when`

Fixed time tasks are comparable to cron-style scheduling. Times will be interpreted according to the Rails default TZ.

By default run_at and run_when will automatically catch up missed tasks. Scheddy does this by maintaining a record of the last run. If one or more runs was missed, it will run once immediately. Multiple misses will still only be run once. Set track_runs false to disable catch-ups.

Intervals: `run_every`

Intervals are similar to cron style */5 syntax, but one key difference is the cycle is calculated based on Scheddy's startup time.

To avoid all tasks running at once, interval tasks are given an initial random delay of no more than the interval length itself. For example, a task running at 15 second intervals will be randomly delayed 0-14 seconds for first run. It will then continue running every 15 seconds.

By default, tiny interval tasks (those under 15 minutes) do not track last run or perform catch-ups, but longer interval tasks (>= 15 minutes) do. This is because tiny intervals will re-run soon anyway and it reduces database activity. Set track_runs true|false to override.

Additional notes

Units of work

Notice that all these examples delegate the actual work to an external job. This is the recommended approach, but is not strictly required.

In general, bite-sized bits of work are fine in Scheddy, but bigger chunks of work usually belong in a background queue. In general, when timeliness is key (running right on time) or scheduling a background job is more costly than doing the work directly, then performing work inside the Scheddy task may be appropriate.

Database transactions are valid. These can increase use of database connections from the pool. Ensure Rails is configured appropriately.

Threading and execution

Each task runs in its own thread which helps ensure all tasks perform on time. However, Scheddy is not intended as a job executor and doesn't have a robust mechanism for retrying failed jobs--that belongs to your background job queue.

A given task will only ever be executed once at a time. Mostly relevant when using tiny intervals, if a prior execution is still going when the next execution is scheduled to start, Scheddy will skip the next execution and log an error message to that effect.

Running 2+ Scheddy instances in a cluster

Running multiple Scheddy instances will automatically form a cluster with one leader and however many standbys. Tasks are only executed by the leader.

Clean and unclean handoffs are both supported, taking 1-2 and 4-5 minutes respectively. After a handoff, task executions will be caught up by the new leader, same as if a single instance had been stopped and restarted.

Note that the prior section's promise that a task will only be executed once at a time cannot be guaranteed in the case of a handoff between Scheddy instances. Since tasks are intended to run very quickly, and long tasks should run via ActiveJob instead, this should not be an issue in practice.

Task context

Tasks may receive an optional context to check if they need to stop for pending shutdown or to know the deadline for completing work before the next cycle would begin.

Deadlines (finish_before) are mostly useful if there is occasionally a large block of work combined with tiny intervals. It may be necessary to calculate your own buffer when comparing against finish_before (1 second is shown below). As already mentioned, Scheddy is smart enough to skip the next cycle if the prior cycle is still running, so handling deadlines is entirely optional.

task 'iterating task' do
  run_every 15.seconds
  perform do |context|
    Model.where(...).find_each do |model|
      SomeJob.perform_later model.id if model.run_job?
      break if context.stop? # the scheduler has requested to shutdown
      break if context.finish_before < 1.second.from_now # the next cycle is imminent
    end
  end
end

Rails reloader

Each task's block is run inside the Rails reloader. In development mode, any classes referenced inside the block will be reloaded automatically to your latest code, just like the Rails dev-server itself.

It's possible to also make the task work reloadable by using a proxy class for the task itself. If your tasks are a bit bigger, organizing them into app/tasks/ might be worthwhile anyway.

# config/initializers/scheddy.rb
Scheddy.config do
  task 'weekly report' do
    run_at 'friday 9am'
    perform 'WeeklyReportTask.perform'
  end
end

# app/tasks/weekly_report_task.rb
class WeeklyReportTask
  def self.perform
    ReportJob.perform_later
  end
end

Running Scheddy

Depending on your ruby setup, one of the following should do:

  scheddy start
  # OR
  bundle exec scheddy start

You can also check your tasks configuration with:

  scheddy tasks
  # OR
  bundle exec scheddy tasks

When running 2+ Scheddy's, a handoff may be initiated with:

  scheddy stepdown
  # OR
  bundle exec scheddy stepdown

In production

Scheddy runs as its own process. During app deployments, it is recommended to shutdown the old Scheddy instance before starting the new one.

Since Scheddy has the ability to catch up missed tasks, it is often viable to run just a single Scheddy instance, especially when that instance is automatically restarted by systemd, dockerd, Kubernetes, or whatever supervisory system you use.

It is also possible to run 2+ Scheddy instances as a cluster. One instance will be selected as the leader and execute tasks. The other instance(s) will operate as standbys, ready to take over if the leader steps down or fails.

In development (and `Procfile` in production)

Assuming you're using Procfile.dev or Procfile for development, add:

scheddy: bundle exec scheddy start

Signals and shutdown

Scheddy will shutdown upon receiving an INT, QUIT, or TERM signal.

There is a default 45 second wait for tasks to complete, which should be more than enough for the tiny types of tasks at hand. Tasks may also check for when to stop work part way through. This may be useful in iterators processing large numbers of items. See Task Context above.

Error handling

Scheddy's default error handler uses the Rails Errors API introduced in Rails 7. If your exception tracker of choice doesn't implement this API, or if using Rails 6.x, set your own error handler.

Note that the default handler is responsible for exception logging, so you must perform your own logging if wanted. If the handler is set to nil, exceptions will be silenced.

Scheddy.config do
  error_handler do |exception, task|
    # displaying task.name is the most likely use of task
    name = "task '#{task.name}'" if task  # task might be nil
    logger.error "Exception in Scheddy #{name}: #{e.inspect}"
    # report the exception here
  end

  error_handler ->(exception){
    # passing a proc instead of a block is also allowed
    # the task arg can be left out if it won't be used
  }

  error_handler nil  # silence & don't report
end

Compatibility

Used in production on Rails 7.0+. Gemspec is set to Rails 6.0+, but such is not well tested.

Contributing

Pull requests are welcomed.

License

MIT licensed.