Class: CloudCrowd::Node

Inherits:
Sinatra::Default
  • Object
show all
Defined in:
lib/cloud_crowd/node.rb

Overview

A Node is a Sinatra/Thin application that runs a single instance per-machine It registers with the central server, receives WorkUnits, and forks off Workers to process them. The actions are:

get /heartbeat

Returns 200 OK to let monitoring tools know the server’s up.

post /work

The central server hits /work to dispatch a WorkUnit to this Node.

Constant Summary collapse

DEFAULT_PORT =

A Node’s default port. You only run a single node per machine, so they can all use the same port without any problems.

9063
SCRAPE_UPTIME =

A list of regex scrapers, which let us extract the one-minute load average and the amount of free memory on different flavors of UNIX.

/\d+\.\d+/
SCRAPE_LINUX_MEMORY = 
/MemFree:\s+(\d+) kB/
SCRAPE_MAC_MEMORY = 
/Pages free:\s+(\d+)./
SCRAPE_MAC_PAGE = 
/page size of (\d+) bytes/
MONITOR_INTERVAL =

The interval at which the node monitors the machine’s load and memory use (if configured to do so in config.yml).

3
OVERLOADED_MESSAGE =

The response sent back when this node is overloaded.

'Node Overloaded'

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(port = DEFAULT_PORT) ⇒ Node

When creating a node, specify the port it should run on.



62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/cloud_crowd/node.rb', line 62

def initialize(port=DEFAULT_PORT)
  require 'json'
  @server           = CloudCrowd.central_server
  @host             = Socket.gethostname
  @enabled_actions  = CloudCrowd.actions.keys
  @asset_store      = AssetStore.new
  @port             = port || DEFAULT_PORT
  @overloaded       = false
  @max_load         = CloudCrowd.config[:max_load]
  @min_memory       = CloudCrowd.config[:min_free_memory]
  start unless test?
end

Instance Attribute Details

#asset_store ⇒ Object (readonly)

Returns the value of attribute asset_store.



30
31
32
 
# File 'lib/cloud_crowd/node.rb', line 30

def asset_store
  @asset_store
end

#enabled_actions ⇒ Object (readonly)

Returns the value of attribute enabled_actions.



30
31
32
 
# File 'lib/cloud_crowd/node.rb', line 30

def enabled_actions
  @enabled_actions
end

#host ⇒ Object (readonly)

Returns the value of attribute host.



30
31
32
 
# File 'lib/cloud_crowd/node.rb', line 30

def host
  @host
end

#port ⇒ Object (readonly)

Returns the value of attribute port.



30
31
32
 
# File 'lib/cloud_crowd/node.rb', line 30

def port
  @port
end

#server ⇒ Object (readonly)

Returns the value of attribute server.



30
31
32
 
# File 'lib/cloud_crowd/node.rb', line 30

def server
  @server
end

Instance Method Details

#check_in(critical = false) ⇒ Object

Checking in with the central server informs it of the location and configuration of this Node. If it can’t check-in, there’s no point in starting.



88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/cloud_crowd/node.rb', line 88

def check_in(critical=false)
  @server["/node/#{@host}"].put(
    :port             => @port,
    :busy             => @overloaded,
    :max_workers      => CloudCrowd.config[:max_workers],
    :enabled_actions  => @enabled_actions.join(',')
  )
rescue Errno::ECONNREFUSED
  puts "Failed to connect to the central server (#{@server.to_s})."
  raise SystemExit if critical
end

#check_out ⇒ Object

Before exiting, the Node checks out with the central server, releasing all of its WorkUnits for other Nodes to handle



102
103
104
 
# File 'lib/cloud_crowd/node.rb', line 102

def check_out
  @server["/node/#{@host}"].delete
end

#free_memory ⇒ Object

The current amount of free memory in megabytes.



120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/cloud_crowd/node.rb', line 120

def free_memory
  case RUBY_PLATFORM
  when /darwin/
    stats = `vm_stat`
    @mac_page_size ||= stats.match(SCRAPE_MAC_PAGE)[1].to_f / 1048576.0
    stats.match(SCRAPE_MAC_MEMORY)[1].to_f * @mac_page_size
  when /linux/
    `cat /proc/meminfo`.match(SCRAPE_LINUX_MEMORY)[1].to_f / 1024.0
  else
    raise NotImplementedError, "'min_free_memory' is not yet implemented on your platform"
  end
end

#load_average ⇒ Object

The current one-minute load average.



115
116
117
 
# File 'lib/cloud_crowd/node.rb', line 115

def load_average
  `uptime`.match(SCRAPE_UPTIME).to_s.to_f
end

#overloaded? ⇒ Boolean

Is the node overloaded? If configured, checks if the load average is greater than ‘max_load’, or if the available RAM is less than ‘min_free_memory’.

Returns:

  • (Boolean) 


109
110
111
112
 
# File 'lib/cloud_crowd/node.rb', line 109

def overloaded?
  (@max_load && load_average > @max_load) ||
  (@min_memory && free_memory < @min_memory)
end

#start ⇒ Object

Starting up a Node registers with the central server and begins to listen for incoming WorkUnits.



77
78
79
80
81
82
83
 
# File 'lib/cloud_crowd/node.rb', line 77

def start
  trap_signals
  start_server
  monitor_system if @max_load || @min_memory
  check_in(true)
  @server_thread.join
end