Class: RightScale::InstanceState

Inherits:
Object
  • Object
show all
Defined in:
lib/instance/instance_state.rb

Overview

Manages instance state

Defined Under Namespace

Classes: PlannedVolumeState

Constant Summary collapse

CONFIG_YAML_FILE = 
File.normalize_path(File.join(RightScale::Platform.filesystem.right_link_static_state_dir, 'features.yml'))
CONFIG = 
if File.exists?(CONFIG_YAML_FILE)
  RightSupport::Config.features(CONFIG_YAML_FILE)
else
  RightSupport::Config.features({})
end
RECORDED_STATES =

States that are recorded in a standard fashion and audited when transitioned to

%w{ booting operational stranded decommissioning }
SUCCESSFUL_STATES =

States that cause the system MOTD/banner to indicate that everything is OK

%w{ operational }
FAILED_STATES =

States that cause the system MOTD/banner to indicate that something is wrong

%w{ stranded }
INITIAL_STATE =

Initial state prior to booting

'pending'
FINAL_STATE =

Final state when shutting down that is recorded in a non-standard fashion

'decommissioned'
STATES =

Valid internal states

RECORDED_STATES + [FINAL_STATE]
STATE_DIR =

Path to JSON file where current instance state is serialized

AgentConfig.agent_state_dir
STATE_FILE = 
File.join(STATE_DIR, 'state.js')
LOGIN_POLICY_FILE =

Path to JSON file where authorized login users are defined

File.join(STATE_DIR, 'login_policy.js')
BOOT_LOG_FILE =

Path to boot log

File.join(RightScale::Platform.filesystem.log_dir, 'install')
DECOMMISSION_LOG_FILE =

Path to decommission log

File.join(RightScale::Platform.filesystem.log_dir, 'decommission')
FORCE_SHUTDOWN_DELAY =

Number of seconds to wait for cloud to shutdown instance

180
MAX_RECORD_STATE_RETRIES =

Maximum number of retries to record state with RightNet

5
RETRY_RECORD_STATE_DELAY =

Number of seconds between attempts to record state

5
LAST_COMMUNICATION_STORAGE_INTERVAL =

Minimum interval in seconds for persistent storage of last communication

2

Class Method Summary collapse

Class Method Details

.decommission_type ⇒ Object

(String) Type of decommission currently in progress or nil



122
123
124
125
126
127
128
 
# File 'lib/instance/instance_state.rb', line 122

def self.decommission_type
  if @value == 'decommissioning' || @value == 'decommissioned'
    @decommission_type
  else
    raise RightScale::Exceptions::WrongState.new("Unexpected call to InstanceState.decommission_type for current state #{@value.inspect}")
  end
end

.decommission_type=(decommission_type) ⇒ Object

Set decommission type and set state to ‘decommissioning’

Parameters

decommission_type(String)

One of RightScale::ShutdownRequest::LEVELS or nil

Return

result(String)

new decommission type

Raise

RightScale::Exceptions::Application

Cannot update in read-only mod



277
278
279
280
281
282
283
284
 
# File 'lib/instance/instance_state.rb', line 277

def self.decommission_type=(decommission_type)
  unless RightScale::ShutdownRequest::LEVELS.include?(decommission_type)
    raise RightScale::ShutdownRequest::InvalidLevel.new("Unexpected decommission_type: #{decommission_type}")
  end
  @decommission_type = decommission_type
  self.value = 'decommissioning'
  @decommission_type
end

.identity ⇒ Object

(String) Instance agent identity



104
105
106
 
# File 'lib/instance/instance_state.rb', line 104

def self.identity
  @identity
end

.init(identity, read_only = false) ⇒ Object

Set instance id with given id Load persisted state if any, compare instance ids and force boot if instance ID is different or if reboot flagged For reboot detection relying on rightboot script in linux and shutdown notification in windows to update the reboot flag in the state file

Parameters

identity(String)

Instance identity

read_only(Boolean)

Whether only allowed to read the instance state, defaults to false

Return

true

Always return true



142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
 
# File 'lib/instance/instance_state.rb', line 142

def self.init(identity, read_only = false)
  @identity = identity
  @read_only = read_only
  @startup_tags = []
  @log_level = Logger::INFO
  @initial_boot = false
  @reboot = false
  @resource_uid = nil
  @last_recorded_value = nil
  @record_retries = 0
  @record_request = nil
  @record_timer = nil
  @last_communication = 0
  @planned_volume_state = nil
  @decommission_type = nil

  Log.notify(lambda { |l| @log_level = l }) unless @read_only

  Sender.instance.message_received { message_received } unless @read_only

  # need to grab the current resource uid whether there is a state file or not.
  @resource_uid = current_resource_uid

  dir = File.dirname(STATE_FILE)
  FileUtils.mkdir_p(dir) unless File.directory?(dir)
  if File.file?(STATE_FILE)
    state = RightScale::JsonUtilities::read_json(STATE_FILE)
    Log.debug("Initializing instance #{identity} with #{state.inspect}")

    # Initial state reconciliation: use recorded state and boot timestamp to determine how we last stopped.
    # There are four basic scenarios to worry about:
    #  1) first run          -- Agent is starting up for the first time after a fresh install
    #  2) reboot/restart     -- Agent already ran; agent ID not changed; reboot detected: transition back to booting
    #  3) bundled boot       -- Agent already ran; agent ID changed: transition back to booting
    #  4) decommission/crash -- Agent exited anyway; ID not changed; no reboot; keep old state entirely
    #  5) ec2 restart        -- Agent already ran; agent ID changed; instance ID is the same; transition back to booting
    if state['identity'] && state['identity'] != identity && !@read_only
      @last_recorded_value = state['last_recorded_value']
      self.value = 'booting'
      # if the current resource_uid is the same as the last
      # observed resource_uid, then this is a restart,
      # otherwise this is a bundle
      old_resource_uid = state["last_observed_resource_uid"]
      if @resource_uid && @resource_uid == old_resource_uid
        # CASE 5 -- identity has changed; ec2 restart
        Log.debug("Restart detected; transitioning state to booting")
        @reboot = true
      else
        # CASE 3 -- identity has changed; bundled boot
        Log.debug("Bundle detected; transitioning state to booting")
      end
    elsif state['reboot'] && !@read_only
      # CASE 2 -- rebooting flagged by rightboot script in linux or by shutdown notification in windows
      Log.debug("Reboot detected; transitioning state to booting")
      @last_recorded_value = state['last_recorded_value']
      self.value = 'booting'
      @reboot = true
    else
      # CASE 4 -- restart without reboot; continue with retries if recorded state does not match
      @value = state['value']
      @reboot = state['reboot']
      @startup_tags = state['startup_tags']
      @log_level = state['log_level']
      @last_recorded_value = state['last_recorded_value']
      @record_retries = state['record_retries']
      @decommission_type = state['decommission_type'] if (@value == 'decommissioning' || @value == 'decommissioned')
      if @value != @last_recorded_value && RECORDED_STATES.include?(@value) &&
         @record_retries < MAX_RECORD_STATE_RETRIES && !@read_only
        record_state
      else
        @record_retries = 0
      end
      update_logger
    end
  else
    # CASE 1 -- state file does not exist; initial boot, create state file
    Log.debug("Initializing instance #{identity} with booting")
    @last_recorded_value = INITIAL_STATE
    self.value = 'booting'
    @initial_boot = true
  end

  if File.file?()
    @login_policy = RightScale::JsonUtilities::read_json() rescue nil #corrupt file here is not important enough to fail
  else
    @login_policy = nil
  end
  Log.debug("Existing login users: #{@login_policy.users.length} recorded") if @login_policy

  #Ensure MOTD is up to date
  update_motd

  true
end

.initial_boot? ⇒ Boolean

Is this the initial boot?

Return

res(Boolean)

Whether this is the instance first boot

Returns:

  • (Boolean) 


298
299
300
 
# File 'lib/instance/instance_state.rb', line 298

def self.initial_boot?
  res = @initial_boot
end

.last_recorded_value ⇒ Object

(String) One of STATES



94
95
96
 
# File 'lib/instance/instance_state.rb', line 94

def self.last_recorded_value
  @last_recorded_value
end

.log_file(state) ⇒ Object

Log file to be used for given instance state

Parameters

state(String)

Instance state, one of STATES

Return

log(String)

Log file path

nil

Log file should not be changed



455
456
457
458
459
460
 
# File 'lib/instance/instance_state.rb', line 455

def self.log_file(state)
  log_file = case state
    when 'booting'         then BOOT_LOG_FILE
    when 'decommissioning' then DECOMMISSION_LOG_FILE
  end
end

.log_level ⇒ Object

Log level

Return

log_level(Const)

One of Logger::DEBUG…Logger::FATAL



393
394
395
 
# File 'lib/instance/instance_state.rb', line 393

def self.log_level
  @log_level
end

.login_policy ⇒ Object

(LoginPolicy) The most recently enacted login policy



109
110
111
 
# File 'lib/instance/instance_state.rb', line 109

def self.
  @login_policy
end

.login_policy=(login_policy) ⇒ Object

Record set of authorized login users

Parameters

login_users(Array) set of authorized login users

Return

login_users(Array) authorized login users



439
440
441
442
443
444
445
 
# File 'lib/instance/instance_state.rb', line 439

def self.=()
  @login_policy = .dup
  File.open(, 'w') do |f|
    f.write(@login_policy.to_json)
  end
  
end

.message_received ⇒ Object

Update the time this instance last received a message thus demonstrating that it is still connected

Return

true

Always return true



315
316
317
318
319
320
321
 
# File 'lib/instance/instance_state.rb', line 315

def self.message_received
  now = Time.now.to_i
  if (now - @last_communication) > LAST_COMMUNICATION_STORAGE_INTERVAL
    @last_communication = now
    store_state
  end
end

.observe(&observer) ⇒ Object

Callback given observer on all state transitions

Block

Given block should take one argument which will be the transitioned to state

Return

true

Always return true



404
405
406
407
408
 
# File 'lib/instance/instance_state.rb', line 404

def self.observe(&observer)
  @observers ||= []
  @observers << observer
  true
end

.planned_volume_state ⇒ Object

Queries most recent state of planned volume mappings.

Return

result(Array)

persisted mappings or empty



117
118
119
 
# File 'lib/instance/instance_state.rb', line 117

def self.planned_volume_state
  @planned_volume_state ||= PlannedVolumeState.new
end

.reboot? ⇒ Boolean

Are we rebooting? (needed for RightScripts)

Return

res(Boolean)

Whether this instance was rebooted

Returns:

  • (Boolean) 


306
307
308
 
# File 'lib/instance/instance_state.rb', line 306

def self.reboot?
  res = @reboot
end

.record_request ⇒ Object

(IdempotentRequest) Current record state request



99
100
101
 
# File 'lib/instance/instance_state.rb', line 99

def self.record_request
  @record_request
end

.resource_uid ⇒ Object

Instance AWS id for EC2 instances

Return

resource_uid(String)

Instance AWS ID on EC2, equivalent on other cloud when available



290
291
292
 
# File 'lib/instance/instance_state.rb', line 290

def self.resource_uid
  resource_uid = @resource_uid
end

.shutdown(user_id, skip_db_update, kind) ⇒ Object

Ask core agent to shut ourselves down for soft termination Do not specify the last recorded state since does not matter at this point and no need to risk request failure Add a timer to force shutdown if do not hear back from the cloud or the request hangs

Parameters

user_id(Integer)

ID of user that triggered soft-termination

skip_db_update(Boolean)

Whether to re-query instance state after call to Ec2 to terminate was made

kind(String)

‘terminate’, ‘stop’ or ‘reboot’

Return

true

Always return true



335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
 
# File 'lib/instance/instance_state.rb', line 335

def self.shutdown(user_id, skip_db_update, kind)
  payload = {:agent_identity => @identity, :state => FINAL_STATE, :user_id => user_id, :skip_db_update => skip_db_update, :kind => kind}
  Sender.instance.send_retryable_request("/state_recorder/record", payload) do |r|
    res = OperationResult.from_results(r)
    case kind
    when 'reboot'
      RightScale::Platform.controller.reboot unless res.success?
    when 'terminate', 'stop'
      Sender.instance.send_push("/registrar/remove", {:agent_identity => @identity, :created_at => Time.now.to_i})
      RightScale::Platform.controller.shutdown unless res.success?
    else
      Log.error("InstanceState.shutdown() kind was unexpected: #{kind}")
    end
  end
  case kind
  when 'reboot'
    EM.add_timer(FORCE_SHUTDOWN_DELAY) { RightScale::Platform.controller.reboot }
  when 'terminate', 'stop'
    EM.add_timer(FORCE_SHUTDOWN_DELAY) { RightScale::Platform.controller.shutdown }
  else
    Log.error("InstanceState.shutdown() kind was unexpected: #{kind}")
  end
end

.startup_tags ⇒ Object

Tags retrieved on startup

Return

tags(Array)

List of tags retrieved on startup



385
386
387
 
# File 'lib/instance/instance_state.rb', line 385

def self.startup_tags
  @startup_tags
end

.startup_tags=(val) ⇒ Object

Set startup tags

Parameters

val(Array)

List of tags

Return

val(Array)

List of tags

Raise

RightScale::Exceptions::Application

Cannot update in read-only mode

Raises:

  • (RightScale::Exceptions::Application) 


369
370
371
372
373
374
375
376
377
378
379
 
# File 'lib/instance/instance_state.rb', line 369

def self.startup_tags=(val)
  raise RightScale::Exceptions::Application, "Not allowed to modify instance state in read-only mode" if @read_only
  if @startup_tags.nil? || @startup_tags != val
    @startup_tags = val
    # FIX: storing state on change to ensure the most current set of tags is available to
    #      cook (or other processes that load instance state) when it is launched.  Would
    #      be better to communicate state via other means.
    store_state
  end
  val
end

.update_logger ⇒ Object

Point logger to log file corresponding to current instance state

Return

true

Always return true



414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
 
# File 'lib/instance/instance_state.rb', line 414

def self.update_logger
  previous_level = nil
  if @current_logger
    previous_level = @current_logger.level
    Log.remove_logger(@current_logger)
    @current_logger = nil
  end
  if file = log_file(@value)
    dir = File.dirname(file)
    FileUtils.mkdir_p(dir) unless File.directory?(dir)
    @current_logger = ::Logger.new(file)
    @current_logger.level = previous_level if previous_level
    Log.add_logger(@current_logger)
  end
  true
end

.value ⇒ Object

(String) One of STATES



89
90
91
 
# File 'lib/instance/instance_state.rb', line 89

def self.value
  @value
end

.value=(val) ⇒ Object

Set instance state

Parameters

val(String) One of STATES

Return

val(String) new state

Raise

RightScale::Exceptions::Application

Cannot update in read-only mode

RightScale::Exceptions::Argument

Invalid new value

Raises:

  • (RightScale::Exceptions::Application) 


248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
 
# File 'lib/instance/instance_state.rb', line 248

def self.value=(val)
  previous_val = @value || INITIAL_STATE
  raise RightScale::Exceptions::Application, "Not allowed to modify instance state in read-only mode" if @read_only
  raise RightScale::Exceptions::Argument, "Invalid instance state #{val.inspect}" unless STATES.include?(val)
  Log.info("Transitioning state from #{previous_val} to #{val}")
  @reboot = false if val != :booting
  @value = val
  @decommission_type = nil unless (@value == 'decommissioning' || @value == 'decommissioned')

  update_logger
  update_motd
  broadcast_wall unless (previous_val == val)
  record_state if RECORDED_STATES.include?(val)
  store_state
  @observers.each { |o| o.call(val) } if @observers

  val
end