Class: Redis::ScriptManager

Inherits:
Object
  • Object
show all
Defined in:
lib/redis/script_manager.rb,
lib/redis/script_manager/version.rb

Overview

Utility to efficiently manage SCRIPT LOAD, EVAL, and EVALSHA over all Redis/Lua scripts.

Defined Under Namespace

Classes: Configuration

Constant Summary collapse

VERSION = 
'0.0.6'.freeze

Class Method Summary collapse

Class Method Details

._statsd_increment(metric) ⇒ Object

Reports a single count on the requested metric to statsd (if any).

Parameters:

  • metric —

    String



216
217
218
219
220
 
# File 'lib/redis/script_manager.rb', line 216

def self._statsd_increment(metric)
  if configuration.statsd
    configuration.statsd.increment(configuration.stats_prefix+metric)
  end
end

._statsd_timing(metric, timing) ⇒ Object

Reports a timing on the requested metric to statsd (if any).

Parameters:

  • metric —

    String



226
227
228
229
230
 
# File 'lib/redis/script_manager.rb', line 226

def self._statsd_timing(metric,timing)
  if configuration.statsd
    configuration.statsd.timing(configuration.stats_prefix+metric,timing)
  end
end

.configuration ⇒ Object 



234
235
236
 
# File 'lib/redis/script_manager.rb', line 234

def self.configuration
  @configuration ||= Configuration.new
end

.configuration=(configuration) ⇒ Object

Sets the current Redis::ScriptManager::Configuration.



240
241
242
 
# File 'lib/redis/script_manager.rb', line 240

def self.configuration=(configuration)
  @configuration = configuration
end

.configure {|configuration| ... } ⇒ Object

Yields the current Redis::ScriptManager::Configuration, supports the typical gem configuration pattern:

Redis::ScriptManager.configure do |config|
  config.statsd             = $statsd
  config.stats_prefix       = 'foo'
  config.minify_lua         = true
  config.max_tiny_lua       = 1235
  config.preload_shas       = true
  config.preload_cache_size = 10
end

Yields:



256
257
258
 
# File 'lib/redis/script_manager.rb', line 256

def self.configure
  yield configuration
end

.eval_gently(redis, lua, keys = [], args = []) ⇒ Object

Efficiently evaluates a Lua script on Redis.

Makes a best effort to moderate bandwidth by leveraging EVALSHA and managing the SCRIPT LOAD state.

and :script

list in the lua script.

the lua script.

on redis

Parameters:

  • redis —

    a Redis to call, must respond to :eval, :evalsha,

  • lua —

    a String, the Lua script to execute on redis

  • keys (defaults to: []) —

    a list of String, the keys which will bind to the KEYS

  • args (defaults to: []) —

    a list of arguments which will bind to ARGV list in

Returns:

  • the return result of evaluating lua against keys and args



32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/redis/script_manager.rb', line 32

def self.eval_gently(redis,lua,keys=[],args=[])
  [:eval,:evalsha,:script].each do |method|
    if !redis.respond_to?(method) || !redis.respond_to?(:script)
      raise ArgumentError, "bogus redis #{redis}, no #{method}"
    end
  end
  if !lua.is_a?(String)
    raise ArgumentError, "bogus lua #{lua}"
  end
  if !keys.is_a?(Array)
    raise ArgumentError, "bogus keys #{keys}: non-array"
  end
  keys_classes = keys.map(&:class).uniq
  if [] != keys_classes - [String,Symbol]
    raise ArgumentError, "bogus keys #{keys}: bad types in #{keys_classes}"
  end
  if !args.is_a?(Array)
    raise ArgumentError, "bogus args #{args}"
  end
  if keys.size < 1
    raise ArgumentError, 'Twemproxy intolerant of 0 keys in EVAL or EVALSHA'
  end
  #
  # Per http://redis.io/commands/eval:
  #
  #   Redis does not need to recompile the script every time as it
  #   uses an internal caching mechanism, however paying the cost of
  #   the additional bandwidth may not be optimal in many contexts."
  #
  # So, where the bandwidth imposed by uploading a short script is
  # small, we will just use EVAL.  Also, in MONITOR streams it can
  # be nice to see the Lua instead of a bunch of noisy shas.
  #
  # However, where the script is long we try to conserve bandwidth
  # by trying EVALSHA first.  In case our script has never been
  # uploaded before, or if the Redis suffered a FLUSHDB or SCRIPT
  # FLUSH, we catch NOSCRIPT error, recover with SCRIPT LOAD, and
  # repeat the EVALSHA.
  #
  # Caveat: if all of this is wrapped in a Redis pipeline, then the
  # first EVALSHA returns a Redis::Future, not a result.  We won't
  # know whether it throws an error until after the pipeline - and
  # after we've committed to the rest of our stream of commands.
  # Thus, we can't recover from a NOSCRIPT error.
  #
  # To be safe in this pipelined-but-not-in-script-database edge
  # case, we stick with simple EVAL when we detect we are running
  # within a pipeline.
  #
  if configuration.do_minify_lua
    lua = minify_lua(lua)
  end
  if lua.size < configuration.max_tiny_lua
    _statsd_increment("eval")
    return redis.eval(lua,keys,args)
  end
  if configuration.do_preload
    #
    # I tested RLEC in ali-staging with this script:
    #
    #   sha = Redis.current.script(:load,'return redis.call("get",KEYS[1])')
    #   100.times.map { |n| "foo-#{n}" }.each do |key|
    #     set = Redis.current.set(key,key)
    #     get = Redis.current.evalsha(sha,[key])
    #     puts "%7s %7s %7s" % [key,get,set]
    #   end
    #
    # Sure enough, all 100 EVALSHA worked.  Since ali-staging is a
    # full-weight sharded RLEC, this tells me that SCRIPT LOAD
    # propagates to all Redis shards in RLEC.
    #
    # Thus, we can trust that any script need only be sent down a
    # particular Redis connection once.  Thereafter we can assume
    # EVALSHA will work.
    #
    # I do not know if the same thing will happen in Redis Cluster,
    # and I am just about certain that the more primitive
    # Twemproxy-over-array-of-plain-Redis will *not* propagate
    # SCRIPT this way.
    #
    # Here is an twemproxy issue which tracks this question
    # https://github.com/twitter/twemproxy/issues/68.
    #
    # This implementation is meant to transmit each script to each
    # Redis no more than once per process, and thereafter be
    # pure-EVALSHA.
    #
    sha                = Digest::SHA1.hexdigest(lua)
    sha_connection_key = [redis.object_id,sha]
    if !@preloaded_shas.include?(sha_connection_key)
      _statsd_increment("preloaded_shas.cache_miss")
      new_sha = redis.script(:load,lua)
      if !new_sha.is_a?(Redis::Future) && sha != new_sha
        raise RuntimeError, "mismatch #{sha} vs #{new_sha} for lua #{lua}"
      end
      @preloaded_shas  << sha_connection_key
    else
      _statsd_increment("preloaded_shas.cache_hit")
    end
    result   = redis.evalsha(sha,keys,args)
    if configuration.preload_cache_size < @preloaded_shas.size
      #
      # To defend against unbound cache size, at a predetermined
      # limit throw away half of them.
      #
      # It is benign to re-load a script, just a performance blip.
      #
      # If these caches are running away in size, we have a worse
      # time: redis.info.used_memory_lua could be growing without
      # bound.
      #
      _statsd_increment("preloaded_shas.cache_purge")
      num_to_keep      = @preloaded_shas.size / 2
      @preloaded_shas  = @preloaded_shas.to_a.sample(num_to_keep)
    end
    cache_size         = @preloaded_shas.size
    _statsd_timing("preloaded_shas.cache_size",cache_size)
    return result
  end
  if in_pipeline?(redis)
    _statsd_increment("pipeline_eval")
    return redis.eval(lua,keys,args)
  end
  sha = Digest::SHA1.hexdigest(lua)
  begin
    _statsd_increment("evalsha1")
    result = redis.evalsha(sha,keys,args)
    if result.is_a?(Redis::Future)
      #
      # We should have detected this above where we checked whether
      # redis.client was a pipeline.
      #
      raise "internal error: unexpected Redis::Future from evalsha"
    end
    result
  rescue Redis::CommandError => ex
    if nil != ex.to_s.index('NOSCRIPT') # some vulerability to change in msg
      _statsd_increment("script_load")
      new_sha = redis.script(:load,lua)
      if sha != new_sha
        raise RuntimeError, "mismatch #{sha} vs #{new_sha} for lua #{lua}"
      end
      _statsd_increment("evalsha2")
      return redis.evalsha(sha,keys,args)
    end
    raise ex
  end
end

.in_pipeline?(redis) ⇒ Boolean

Returns true if redis is currently in a pipeline, false otherwise.

Returns:

  • (Boolean) —

    true if redis is currently in a pipeline, false otherwise



183
184
185
186
187
188
189
190
191
192
193
 
# File 'lib/redis/script_manager.rb', line 183

def self.in_pipeline?(redis)
  #
  # redis-rb 4.0 added support for the redis-server command
  # CLIENT, and in so doing re-named the accessor for the lower
  # level client from :client to :_client.
  #
  #   htps://github.com/redis/redis-rb/blob/master/CHANGELOG.md#40
  #
  client = redis.respond_to?(:_client) ? redis._client : redis.client
  client.is_a?(Redis::Pipeline) # thanks @marshall
end

.minify_lua(lua) ⇒ Object

To save bandwidth, minify the Lua code.



203
204
205
206
207
208
209
 
# File 'lib/redis/script_manager.rb', line 203

def self.minify_lua(lua)
  lua
    .split("\n")
    .map    { |l| l.gsub(/\s*--[^'"]*$/,'').strip } # rest-of-line comments
    .reject { |l| /^\s*$/ =~ l }                    # blank lines
    .join("\n")
end

.purge_preloaded_shas ⇒ Object

for test, clean state



197
198
199
 
# File 'lib/redis/script_manager.rb', line 197

def self.purge_preloaded_shas # for test, clean state
  @preloaded_shas = Set[]
end