Class: Hatchet::Reaper

Inherits:
Object
  • Object
show all
Defined in:
lib/hatchet/reaper.rb,
lib/hatchet/reaper/app_age.rb,
lib/hatchet/reaper/reaper_throttle.rb

Overview

Delete apps

Delete a single app:

@reaper.destroy_with_log(id: id, name: name, reason: "console")

Clear out all apps older than HATCHET_ALIVE_TTL_MINUTES:

@reaper.destroy_older_apps

If you need to clear up space or wait for space to be cleared up then:

@reaper.clean_old_or_sleep

Notes:

  • The class uses a file mutex so that multiple processes on the same machine do not attempt to run the reaper at the same time.

Defined Under Namespace

Classes: AlreadyDeletedError, AppAge, ReaperThrottle

Constant Summary collapse

HATCHET_APP_LIMIT =

the number of apps hatchet keeps around

Integer(ENV["HATCHET_APP_LIMIT"] || 20)
DEFAULT_REGEX = 
/^#{Regexp.escape(Hatchet::APP_PREFIX)}[a-f0-9]+/
TTL_MINUTES = 
ENV.fetch("HATCHET_ALIVE_TTL_MINUTES", "7").to_i
MUTEX_FILE =

Protect against parallel deletion on the same machine via concurrent processes

Does not protect against distributed systems on different machines trying to delete the same applications

File.open(File.join(Dir.tmpdir(), "hatchet_reaper_mutex"), File::CREAT)

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(api_rate_limit:, regex: DEFAULT_REGEX, io: STDOUT, hatchet_app_limit: HATCHET_APP_LIMIT, initial_sleep: 10) ⇒ Reaper

Returns a new instance of Reaper.



40
41
42
43
44
45
46
47
 
# File 'lib/hatchet/reaper.rb', line 40

def initialize(api_rate_limit: , regex: DEFAULT_REGEX, io: STDOUT, hatchet_app_limit:  HATCHET_APP_LIMIT, initial_sleep: 10)
  @io = io
  @apps = []
  @regex = regex
  @limit = hatchet_app_limit
  @api_rate_limit = api_rate_limit
  @reaper_throttle = ReaperThrottle.new(initial_sleep: initial_sleep)
end

Instance Attribute Details

#hatchet_app_limit ⇒ Object

Returns the value of attribute hatchet_app_limit.



38
39
40
 
# File 'lib/hatchet/reaper.rb', line 38

def hatchet_app_limit
  @hatchet_app_limit
end

#io ⇒ Object

Returns the value of attribute io.



38
39
40
 
# File 'lib/hatchet/reaper.rb', line 38

def io
  @io
end

Instance Method Details

#destroy_all(force_refresh: @apps.empty?) ⇒ Object

No guardrails, will delete all apps that match the hatchet namespace



111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
# File 'lib/hatchet/reaper.rb', line 111

def destroy_all(force_refresh: @apps.empty?)
  MUTEX_FILE.flock(File::LOCK_EX)

  refresh_app_list if force_refresh

  while app = @apps.pop
    begin
      destroy_with_log(name: app["name"], id: app["id"], reason: "destroy all")
    rescue AlreadyDeletedError => e
      handle_conflict(
        conflict_message: e.message,
        strategy: :refresh_api_and_continue
      )
    end
  end
ensure
  MUTEX_FILE.flock(File::LOCK_UN)
end

#destroy_older_apps(minutes: TTL_MINUTES, force_refresh: @apps.empty?, on_conflict: :refresh_api_and_continue) ⇒ Object

Destroys apps that are older than the given argument (expecting integer minutes)

This method might be running concurrently on multiple processes or multiple machines.

When a duplicate destroy is detected we can move forward with a conflict strategy:

  • ‘:refresh_api_and_continue`: Sleep to see if another process will clean up everything for us and then re-populate apps from the API and continue.

  • ‘:stop_if_under_limit`: Sleep to allow other processes to continue. Then if apps list is under the limit, assume someone else is already cleaning up for us and that we’re good to move ahead to try to create an app. Otherwise if we’re at or over the limit sleep, refresh the app list, and continue attempting to delete apps.



79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/hatchet/reaper.rb', line 79

def destroy_older_apps(minutes: TTL_MINUTES, force_refresh: @apps.empty?, on_conflict: :refresh_api_and_continue)
  MUTEX_FILE.flock(File::LOCK_EX)

  refresh_app_list if force_refresh

  while app = @apps.pop
    age = AppAge.new(created_at: app["created_at"], ttl_minutes: minutes)
    if !age.can_delete?
      @apps.push(app)
      break
    else
      begin
        destroy_with_log(
          id: app["id"],
          name: app["name"],
          reason: "app age (#{age.in_minutes}m) is older than #{minutes}m"
        )
      rescue AlreadyDeletedError => e
        if handle_conflict(
          strategy: on_conflict,
          conflict_message: e.message,
        ) == :stop
          break
        end
      end
    end
  end
ensure
  MUTEX_FILE.flock(File::LOCK_UN)
end

#destroy_with_log(name:, id:, reason:) ⇒ Object 



184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
 
# File 'lib/hatchet/reaper.rb', line 184

def destroy_with_log(name:, id:, reason: )
  message = "Destroying #{name.inspect}: #{id}, (#{@apps.length}/#{@limit}) reason: #{reason}"

  @api_rate_limit.call.app.delete(id)

  io.puts message
rescue Excon::Error::NotFound => e
  body = e.response.body
  request_id = e.response.headers["Request-Id"]
  if body =~ /Couldn\'t find that app./
    message = "Duplicate destroy attempted #{name.inspect}: #{id}, status: 404, request_id: #{request_id}"
    raise AlreadyDeletedError.new(message)
  else
    raise e
  end
rescue Excon::Error::Forbidden => e
  request_id = e.response.headers["Request-Id"]
  message = "Duplicate destroy attempted #{name.inspect}: #{id}, status: 403, request_id: #{request_id}"
  raise AlreadyDeletedError.new(message)
end

#sleep_if_over_limit(reason:) ⇒ Object 



49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/hatchet/reaper.rb', line 49

def sleep_if_over_limit(reason: )
  if @apps.length >= @limit
    age = AppAge.new(created_at: @apps.last["created_at"], ttl_minutes: TTL_MINUTES)
    @reaper_throttle.call(max_sleep: age.sleep_for_ttl) do |sleep_for|
      io.puts <<-EOM.strip_heredoc
        WARNING: Hatchet app limit reached (#{@apps.length}/#{@limit})
        All known apps are younger than #{TTL_MINUTES} minutes.
        Sleeping (#{sleep_for}s)

        Reason: #{reason}
      EOM

      sleep(sleep_for)
    end
  end
end