Class: ExternalIndex

Inherits:
External::Base show all
Defined in:
lib/external_index.rb

Overview

Provides array-like access to index data kept on disk. Index data is defined by a packing format (see Array#pack) like ‘II’, which would represent two integers; in this case each member of the ExternalIndex would be a two-integer array.

All directives except ‘@’ and ‘X’ are allowed, in any combination.

– not implemented – dclone, flatten, flatten!, frozen?, pack, quote, to_yaml, transpose, yaml_initialize

be careful accession io directly. for peformance reasons there is no check to make sure io is in register (ie pos is at a frame boundary, ie io.length % frame_size == 0) In addition, note that length must be adjusted manually in most io operations (truncate is the exception). Thus if you change the file length by any means, the file length must be reset.

Constant Summary collapse

DEFAULT_BUFFER_SIZE =

The default buffer size (8Mb)

8 * 2**20

Constants inherited from External::Base

External::Base::TEMPFILE_BASENAME

Instance Attribute Summary collapse

Attributes inherited from External::Base

#io

Attributes included from External::Chunkable

#default_blksize

Attributes included from External::Enumerable

#enumerate_to_a

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from External::Base

#close, #closed?, #dup, #empty?, #eql?, #flush, #inspect, open, #slice, #to_ary

Methods included from External::Chunkable

#chunk, #default_span, range_begin_and_end, #reverse_chunk, split_range, split_span

Methods included from External::Enumerable

#all?, #any?, #collect, #detect, #each_with_index, #entries, #find, #find_all, #include?, #map, #member?, #select

Constructor Details

#initialize(io = nil, options = {}) ⇒ ExternalIndex

Returns a new instance of ExternalIndex.



105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
 
# File 'lib/external_index.rb', line 105

def initialize(io=nil, options={})
  super(io)
  
  options = {
    :format => "I",
    :nil_value => nil,
    :buffer_size => DEFAULT_BUFFER_SIZE
  }.merge(options)

  # set the format, frame, and frame size
  format = options[:format]
  @frame = 0
  @frame_size = 0
  @process_in_bulk = true
  
  scanner = StringScanner.new(format)
  if scanner.skip(/\d+/)
    # skip leading numbers ... they are normally ignored
    # by pack and unpack but you could raise an error.
  end

  bulk_directive = nil
  while directive = scanner.scan(/./)
    size = ExternalIndex.directive_size(directive)
    raise ArgumentError.new("cannot determine size of: '#{directive}'") if size == nil
    
    # scan for a multiplicity factor
    multiplicity = (scanner.scan(/\d+/) || 1).to_i
    @frame += multiplicity
    @frame_size += size * multiplicity
    
    # if the bulk directive changes, 
    # processing in bulk is impossible
    if bulk_directive == nil
      bulk_directive = directive
    elsif bulk_directive != directive
      @process_in_bulk = false
    end
  end
  
  # The "a" and "A" directives cannot be
  # processed in bulk.
  if ['a','A'].include?(bulk_directive)
    @process_in_bulk = false
  end
  
  # Repetitive formats like "I", "II", "I2I", 
  # etc can be packed and unpacked in bulk.
  @format = process_in_bulk ? "#{bulk_directive}*" : format
  
  # set the buffer size
  self.buffer_size = options[:buffer_size]
  
  # set the nil value to an array of zeros, or
  # to the specified nil value.  If a nil value
  # was specified, ensure it is of the correct
  # frame size and can be packed
  nil_value = if options[:nil_value] == nil 
    self.class.default_nil_value(format, frame)
  else
    options[:nil_value]
  end
  
  begin 
    @nil_value = nil_value.pack(format)
    unless nil_value.length == frame && @nil_value.unpack(format) == nil_value
      raise "" # just to invoke the rescue block
    end
  rescue
    raise ArgumentError, 
      "unacceptable nil value '#{nil_value}': the nil value must " +
      "be in frame and packable using the format '#{format}'"
  end
end

Instance Attribute Details

#format ⇒ Object (readonly)

The format of the indexed data. Format may be optimized from the original input format in cases like ‘III’ where bulk processing is useful.



87
88
89
 
# File 'lib/external_index.rb', line 87

def format
  @format
end

#frame ⇒ Object (readonly)

The number of elements in each entry, ex: (‘I’ => 1, ‘III’ => 3).

frame is calculated from format.



91
92
93
 
# File 'lib/external_index.rb', line 91

def frame
  @frame
end

#frame_size ⇒ Object (readonly)

The number of bytes required for each entry; frame_size is calculated from format.



95
96
97
 
# File 'lib/external_index.rb', line 95

def frame_size
  @frame_size
end

#process_in_bulk ⇒ Object (readonly)

A flag indicating whether or not the format was optimized to pack/unpack entries in bulk; proccess_in_bulk is automatically set according to format.



100
101
102
 
# File 'lib/external_index.rb', line 100

def process_in_bulk
  @process_in_bulk
end

Class Method Details

.[](*argv) ⇒ Object

Initializes a new ExternalIndex using an array-like [] syntax. The last argument may be an options hash (this is ok since ExternalIndex cannot store a Hash anyhow).



27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/external_index.rb', line 27

def [](*argv)
  options = argv.last.kind_of?(Hash) ? argv.pop : {}
  index = new(nil, options)
  
  normalized_args = argv.collect do |item| 
    item.nil? ? index.nil_value : item
  end.flatten
  index.unframed_write(normalized_args)
  
  # reset the position of the IO under this initialize
  index.pos = 0
  index
end

.default_nil_value(format, frame) ⇒ Object

Returns an array of zeros in the specified frame.



79
80
81
 
# File 'lib/external_index.rb', line 79

def default_nil_value(format, frame)
  Array.new(frame, 0)
end

.directive_size(directive) ⇒ Object

Returns the number of bytes required to pack an item in an array using the directive (see Array.pack for more details). All directives return a size except the positioning directives ‘@’ and ‘X’; these and all other unknown directives return nil.

Directives          N bytes
------------------------------
AaBbCcHhUwxZ     |   1
nSsv             |   2
M                |   3
eFfgIiLlNPpV     |   4
m                |   5
u                |   6
DdEGQq           |   8
@X               |   nil



64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/external_index.rb', line 64

def directive_size(directive)
  case directive
  when /^[eFfgIiLlNPpV]$/ then 4
  when /^[DdEGQq]$/ then 8
  when /^[AaBbCcHhUwxZ]$/ then 1
  when /^[nSsv]$/ then 2
  when 'M' then 3
  when 'm' then 5
  when 'u' then 6
  else
    nil
  end
end

.read(fd, options = {}) ⇒ Object

Opens and reads the file into an array.



42
43
44
45
46
47
 
# File 'lib/external_index.rb', line 42

def read(fd, options={})
  return [] if fd.nil?
  open(fd, "r", options) do |index|
    index.read(nil, 0)
  end
end

Instance Method Details

#+(another) ⇒ Object

def *(arg)

not_implemented

end



243
244
245
 
# File 'lib/external_index.rb', line 243

def +(another)
  dup.concat(another)
end

#<<(array) ⇒ Object

Differs from the Array << in that multiple entries can be shifted on at once.



253
254
255
256
 
# File 'lib/external_index.rb', line 253

def <<(array)
  unframed_write(array, length)
  self
end

#<=>(another) ⇒ Object 



258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/external_index.rb', line 258

def <=>(another)
  return 0 if self.object_id == another.object_id
  
  case another
  when Array
    if another.length < self.length
      # if another is equal to the matching subset of self,
      # then self is obviously the longer array and wins.
      result = (self.to_a(another.length) <=> another)
      result == 0 ? 1 : result
    else
      self.to_a <=> another
    end
  when ExternalIndex
    self.io.sort_compare(another.io, (buffer_size/2).ceil)
  else
    raise TypeError.new("can't convert from #{another.class} to ExternalIndex or Array")
  end
end

#==(another) ⇒ Object 



278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
 
# File 'lib/external_index.rb', line 278

def ==(another)
  return true if super
  
  case another
  when Array
    return false if self.length != another.length
    self.to_a == another
    
  when ExternalIndex
    return false if self.length != another.length || self.index_attrs != another.index_attrs
    return true  if (self.io.sort_compare(another.io, (buffer_size/2).ceil)) == 0
 
    self.to_a == another.to_a
  else
    false
  end      
end

#[](one, two = nil) ⇒ Object

Element Reference — Returns the entry at index, or returns an array starting at start and continuing for length entries, or returns an array specified by range. Negative indices count backward from the end of self (-1 is the last element). Returns nil if the index (or starting index) is out of range.

index = ExternalIndex[1,2,3,4,5]
index[2]                   #=> [3]
index[6]                   #=> nil
index[1, 2]                #=> [[2],[3]]
index[1..3]                #=> [[2],[3],[4]]
index[4..7]                #=> [[5]]
index[6..10]               #=> nil
index[-3, 3]               #=> [[3],[4],[5]]

# special cases
index[5]                   #=> nil
index[5, 1]                #=> []
index[5..10]               #=> []

Note that entries are returned in frame.



316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
 
# File 'lib/external_index.rb', line 316

def [](one, two = nil)
  one = convert_to_int(one)
  
  case one
  when Fixnum
    
    # normalize the index
    if one < 0
      one += length 
      return nil if one < 0
    end

    if two == nil 
      at(one)            # read one, no frame
    else
      two = convert_to_int(two)
      return nil if two < 0 || one > length
      return []  if two == 0 || one == length
      
      read(two, one)     # read length, framed
    end
    
  when Range
    raise TypeError, "can't convert Range into Integer" unless two == nil
    total = length
    start, length = split_range(one, total)

    # (identical to those above...)
    return nil if start < 0 || start > total
    return []  if length < 0 || start == total
    
    read(length + 1, start)  # read length, framed
    
  when nil
    raise TypeError, "no implicit conversion from nil to integer"
  when Bignum
    # special case, RangeError not TypeError is raised by Array
    raise RangeError, "can't convert #{one.class} into Integer"
  else
    raise TypeError, "can't convert #{one.class} into Integer"
  end
end

#[]=(*args) ⇒ Object

Element Assignment — Sets the entry at index, or replaces a subset starting at start and continuing for length entries, or replaces a subset specified by range. A negative indices will count backward from the end of self. Inserts elements if length is zero. If nil is used in the second and third form, deletes elements from self. An IndexError is raised if a negative index points past the beginning of self. See also push, and unshift.

index = ExternalIndex.new("", :format => 'I')
index.nil_value                       # => [0]
index[4] = [4]; index                 # => [[0], [0], [0], [0], [4]]
index[0, 3] = [[1], [2], [3]]; index  # => [[1], [2], [3], [0], [4]]
index[1..2] = [[5], [6]]; index       # => [[1], [5], [6], [0], [4]]
index[0, 2] = [[7]]; index            # => [[7], [6], [0], [4]]
index[0..2] = [[8]]; index            # => [[8], [4]]
index[-1]   = [9]; index              # => [[8], [9]]
index[1..-1] = nil; index             # => [[8]]

Differences from Array#[]=

ExternalIndex#[]= can only take entries in frame. This means that for individual assignments, a framed array must be given; in the case of [start, length] and range insertions, an array of framed arrays must be given. Nils are allowed in both cases, and are treated the same as in Array (although insertions replace nil with the nil_value for self).

index = ExternalIndex.new("", :format => 'II')
index.nil_value                       # => [0,0]

index[0] = [1,2]; index               # => [[1,2]]
index[1] = nil; index                 # => [[1,2], [0,0]]

index[0,2] = [[1,2],[3,4]]; index     # => [[1,2], [3,4]]
index[1..3] = [[5,6],[7,8]]; index    # => [[1,2], [5,6], [7,8]]
index[0,3] = nil; index               # => []

Another ExternalIndex with the same frame, format, and nil_value (ie index_attrs) may be used as an input to [start, length] and range insertions.

Performance

Range insertions may require a full copy/rewrite of an ExternalIndex io. For very large instances, this obviously can be quite slow; the cases to watch out for are:

  • insertion of self into self (worst case)

  • insertion of values with lengths that do not match the insertion length

For example:

index = ExternalIndex.new("")
index[0,1] = index
index[0,3] = [[1], [2]]
index[0...3] = [[1], [2], [3], [4]]

– TODO – cleanup error messages so they are more meaningful and helpful, esp for frame errors ++

Raises:

  • (ArgumentError) 


416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
 
# File 'lib/external_index.rb', line 416

def []=(*args)
  raise ArgumentError, "wrong number of arguments (1 for 2)" if args.length < 2
  
  one, two, value = args
  if args.length == 2
    value = two 
    two = nil
  end

  one = convert_to_int(one)
  case one
  when Fixnum
    if one < 0
      one += length
      raise IndexError, "index #{one} out of range" if one  < 0
    end
    
    if two == nil
      # simple insertion 
      unframed_write(value == nil ? nil_value : value, one)
    else
      two = convert_to_int(two)
      raise IndexError, "negative length (#{two})" if two < 0
      
      value = convert_to_ary(value)
      case
      when self == value
        # special case when insertion is self (no validation needed)
        # A whole copy of self is required because the insertion 
        # can overwrite the tail of self.  As such this can be a
        # worst-case scenario-slow and expensive procedure.
        copy_beg = (one + two) * frame_size
        copy_end = io.length
      
        io.copy do |copy|
          # truncate io
          io.truncate(one * frame_size)
          io.pos = io.length
      
          # pad as needed
          pad_to(one) if one > length
      
          # write the copy of self
          io.insert(copy)
      
          # copy the tail of the insertion
          io.insert(copy, copy_beg..copy_end)
        end
      
      when value.length == two
        # optimized insertion, when insertion is the correct length
        write(value, one)
      
      else
        # range insertion: requires copy and rewrite of the tail 
        # of the ExternalIndex, after the insertion.
        # WARN - can be slow when the tail is large
        copy_beg = (one + two) * frame_size
        copy_end = io.length
      
        io.copy("r", copy_beg..copy_end) do |copy|
          # pad as needed
          pad_to(one) if one > length
      
          # write inserted value
          io.pos = one * frame_size
          write(value)
      
          # truncate io
          io.truncate(io.pos)
      
          # copy the tail of the insertion
          io.insert(copy)
        end
      end
    end

  when Range
    raise TypeError, "can't convert Range into Integer" unless two == nil
    start, length, total = split_range(one)
    
    raise RangeError, "#{one} out of range" if start < 0
    self[start, length < 0 ? 0 : length + 1] = value

  when nil
    raise TypeError, "no implicit conversion from nil to integer"
  else
    raise TypeError, "can't convert #{one.class} into Integer"
  end
end

#another ⇒ Object

Returns another instance of self.class, initialized with the current options of self.



227
228
229
 
# File 'lib/external_index.rb', line 227

def another
  self.class.new(nil, options)
end

#at(index) ⇒ Object

Returns entry at index



516
517
518
519
520
521
522
523
 
# File 'lib/external_index.rb', line 516

def at(index)
  if index >= length || (index < 0 && index < -length)
    nil
  else
    str = readbytes(1, index)
    str == nil ? nil : str.unpack(format)
  end
end

#buffer_size ⇒ Object

Returns the buffer size of self (equal to io.default_blksize and default_blksize * frame_size). Buffer size specifies the memory available for io perform external operations.



183
184
185
 
# File 'lib/external_index.rb', line 183

def buffer_size
  self.io.default_blksize
end

#buffer_size=(buffer_size) ⇒ Object

Sets the buffer size of self (as well as io.default_blksize and self.default_blksize). See buffer_size.

Raises:

  • (ArgumentError) 


189
190
191
192
193
194
 
# File 'lib/external_index.rb', line 189

def buffer_size=(buffer_size)
  raise ArgumentError.new("buffer size must be > 0") if buffer_size <= 0
  
  @default_blksize = (buffer_size/frame_size).ceil
  self.io.default_blksize = buffer_size
end

#clear ⇒ Object

Removes all elements from self.



526
527
528
529
 
# File 'lib/external_index.rb', line 526

def clear
  io.truncate(0)
  self
end

#compact ⇒ Object

Returns a copy of self with all nil entries removed. Nil entries are those which equal nil_value.

potentially expensive



535
536
537
538
539
540
541
542
 
# File 'lib/external_index.rb', line 535

def compact
  another = self.another
  nil_array = self.nil_value
  each do |array|
    another << array unless array == nil_array
  end
  another
end

#concat(another) ⇒ Object

Appends the entries in another to self. Another may be an array of entries (in frame), or another ExternalIndex with corresponding index_attrs.

potentially expensive especially if another is very large, or if it must be loaded into memory to be concatenated, ie when cached? = true.



555
556
557
558
559
560
561
562
563
564
565
566
 
# File 'lib/external_index.rb', line 555

def concat(another)
  case another
  when Array
    write(another, length)
  when ExternalIndex 
    check_index(another)
    io.concat(another.io)
  else 
    raise TypeError.new("can't convert #{another.class} into ExternalIndex or Array")
  end
  self
end

#default_blksize=(value) ⇒ Object

Returns the default_blksize of self. See buffer_size.



197
198
199
200
 
# File 'lib/external_index.rb', line 197

def default_blksize=(value)
  @default_blksize = value
  self.io.default_blksize = value * frame_size
end

#each(&block) ⇒ Object

Calls block once for each entry in self, passing that entry as a parameter.



581
582
583
584
585
586
587
588
589
590
 
# File 'lib/external_index.rb', line 581

def each(&block) # :yield: entry
  self.pos = 0
  chunk do |offset, length|
    # special treatment for 1, because then read(1) => [...] rather
    # than [[...]].  when frame > 1, each will iterate over the 
    # element rather than pass it to the block directly
    read(length).each(&block)
  end
  self
end

#each_index(&block) ⇒ Object

Same as each, but passes the index of the entry instead of the entry itself.



593
594
595
596
 
# File 'lib/external_index.rb', line 593

def each_index(&block) # :yield: index
  0.upto(length-1, &block)
  self
end

#first(n = nil) ⇒ Object

Returns the first n entries (default 1)



609
610
611
 
# File 'lib/external_index.rb', line 609

def first(n=nil)
  n.nil? ? self[0] : self[0,n]
end

#index_attrs ⇒ Object

An array of the index attributes of self: [frame, format, nil_value]



214
215
216
 
# File 'lib/external_index.rb', line 214

def index_attrs
  [frame, format, nil_value]
end

#last(n = nil) ⇒ Object

Returns the last n entries (default 1)



650
651
652
653
654
655
656
 
# File 'lib/external_index.rb', line 650

def last(n=nil)
  return self[-1] if n.nil?

  start = length-n
  start = 0 if start < 0
  self[start, n]
end

#length ⇒ Object

Returns the number of entries in self



659
660
661
 
# File 'lib/external_index.rb', line 659

def length
  io.length/frame_size
end

#nil_value(unpacked = true) ⇒ Object

Returns the string value used for nils. Specify unpacked to show the unpacked array value.

index = ExternalIndex.new 
index.nil_value            # => [0]
index.nil_value(false)     # => "\000\000\000\000"



209
210
211
 
# File 'lib/external_index.rb', line 209

def nil_value(unpacked=true)
  unpacked ? @nil_value.unpack(format) : @nil_value
end

#nitems ⇒ Object

Returns the number of non-nil entries in self. Nil entries

are those which equal nil_value. May be zero.



665
666
667
668
669
670
671
672
673
674
 
# File 'lib/external_index.rb', line 665

def nitems
  # TODO - seems like this could be optimized 
  # to run without unpacking each item...
  count = self.length
  nil_array = self.nil_value
  each do |array|
    count -= 1 if array == nil_array
  end
  count
end

#options ⇒ Object

Returns initialization options for the current settings of self.



219
220
221
222
223
 
# File 'lib/external_index.rb', line 219

def options
  { :format => process_in_bulk ? format[0,1] * frame : format,
    :nil_value => nil_value,
    :buffer_size => buffer_size}
end

#pos ⇒ Object

Returns the current position of self (ie io.pos/frame_size). pos is often used as the default location for IO-like operations like read or write.



791
792
793
 
# File 'lib/external_index.rb', line 791

def pos
  io.pos/frame_size
end

#pos=(pos) ⇒ Object

Sets the current position of the index. Positions can be set beyond the actual length of the index, similar to an IO. Negative positions are counted back from the end of the index (just as they are in an array), but naturally raise an error if they count back to a position less than zero.

index = ExternalIndex[[1],[2],[3]]
index.length                          # => 3

index.pos = 2; index.pos              # => 2
index.pos = 10; index.pos             # => 10

index.pos = -1; index.pos             # => 2
index.pos = -10; index.pos            # !> ArgumentError



810
811
812
813
814
815
816
817
 
# File 'lib/external_index.rb', line 810

def pos=(pos)
  if pos < 0
    raise ArgumentError.new("position out of bounds: #{pos}") if pos < -length
    pos += length 
  end
  
  io.pos = (pos * frame_size)
end

#push(*array) ⇒ Object

Append — Pushes the given entry(s) on to the end of self. This expression returns self, so several appends may be chained together. Pushed entries must be in frame.



691
692
693
694
 
# File 'lib/external_index.rb', line 691

def push(*array)
  write(array, length)
  self
end

#read(n = nil, pos = nil) ⇒ Object

Reads n entries from the specified position (ie, read is basically readbytes, then unpack). By default all remaining entries will be read; single entries are returned in frame, multiple entries are returned in an array.

index = ExternalIndex[[1],[2],[3]]
index.pos                       # => 0
index.read                      # => [[1],[2],[3]]
index.read(1,0)                 # => [[1]]
index.read(10,1)                # => [[2],[3]]

The behavior of read when no entries can be read echos that of IO; when n is nil, an empty array is returned; when n is specified, nil will be returned.

index.pos = 3
index.read                      # => []
index.read(1)                   # => nil



888
889
890
891
 
# File 'lib/external_index.rb', line 888

def read(n=nil, pos=nil)
  str = readbytes(n, pos)
  str == nil ? nil : unpack(str)
end

#readbytes(n = nil, pos = nil) ⇒ Object

Reads the packed byte string for n entries from the specified position. By default all remaining entries will be read.

index = ExternalIndex[[1],[2],[3]]
index.pos                              # => 0
index.readbytes.unpack("I*")           # => [1,2,3]
index.readbytes(1,0).unpack("I*")      # => [1]
index.readbytes(10,1).unpack("I*")     # => [2,3]

The behavior of readbytes when no entries can be read echos that of IO; when n is nil, an empty string is returned; when n is specified, nil will be returned.

index.pos = 3
index.readbytes                        # => ""
index.readbytes(1)                     # => nil



836
837
838
839
840
841
842
 
# File 'lib/external_index.rb', line 836

def readbytes(n=nil, pos=nil)
  # set the io position to the specified index
  self.pos = pos unless pos == nil

  # read until the end if no n is given
  n == nil ? io.read : io.read(n * frame_size)
end

#reverse_each(&block) ⇒ Object

Same as each, but traverses self in reverse order.



713
714
715
716
717
718
719
720
721
 
# File 'lib/external_index.rb', line 713

def reverse_each(&block)
  reverse_chunk do |offset, length|
    # special treatment for 1, because then read(1) => [...] rather
    # than [[...]].  when frame > 1, each will iterate over the 
    # element rather than pass it to the block directly
    read(length, offset).reverse_each(&block)
  end
  self
end

#size ⇒ Object

Alias for length



732
733
734
 
# File 'lib/external_index.rb', line 732

def size
  length
end

#to_a ⇒ Object

Converts self to an array, or returns the cache if cached?.



745
746
747
 
# File 'lib/external_index.rb', line 745

def to_a
  length == 0 ? [] : read(length, 0)
end

#unframed_write(array, pos = nil) ⇒ Object

Same as write, except the input entries are unframed. Multiple entries can be provided in a single array, so long as the total number of elements is divisible into entries of the correct frame.

index = ExternalIndex[]
index.unframed_write([2,3], 1)
index.pos = 0; 
index.unframed_write([1])
index.read(3, 0)                # => [[1],[2],[3]]



932
933
934
935
936
937
938
939
940
941
942
943
944
945
 
# File 'lib/external_index.rb', line 932

def unframed_write(array, pos=nil)
  case array
  when Array
    check_unframed_array(array)
    prepare_write_to_pos(pos)
    write_unframed_array(array)
  when ExternalIndex
    check_index(array)
    prepare_write_to_pos(pos)
    write_index(array)
  else  
    raise ArgumentError.new("could not convert #{array.class} to Array or ExternalIndex")
  end
end

#unpack(str) ⇒ Object

Unpacks the given string into an array of index values. Entries are returned in frame.

index = ExternalIndex[[1],[2],[3]]
index.format                          # => 'I*'
index.unpack( [1].pack('I*') )        # => [[1]] 
index.unpack( [1,2,3].pack('I*') )    # => [[1],[2],[3]]
index.unpack("")                      # => []



853
854
855
856
857
858
859
860
861
862
863
864
865
866
 
# File 'lib/external_index.rb', line 853

def unpack(str)
  case
  when process_in_bulk
    # multiple entries, bulk processing (faster)
    results = []
    str.unpack(format).each_slice(frame) {|s| results << s}
    results
  else
    # multiple entries, individual unpacking (slower)
    Array.new(str.length/frame_size) do |i|
      str[i*frame_size, frame_size].unpack(format)
    end
  end
end

#values_at(*selectors) ⇒ Object

Returns a copy of self containing the entries corresponding to the given selector(s). The selectors may be either integer indices or ranges.

potentially expensive



771
772
773
774
775
776
777
778
 
# File 'lib/external_index.rb', line 771

def values_at(*selectors)
  another = self.another
  selectors.each do |s| 
    entries = self[s]
    another << (entries == nil ? nil_value : entries.flatten)
  end
  another
end

#write(array, pos = nil) ⇒ Object

Writes the framed entries into self starting at the specified position. By default writing begins at the current position. The array can have multiple entries so long as each is in the correct frame.

index = ExternalIndex[]
index.write([[2],[3]], 1)
index.pos = 0; 
index.write([[1]])
index.read(3, 0)                # => [[1],[2],[3]]

write may accept an ExternalIndex if it has the same index_attrs as self.



906
907
908
909
910
911
912
913
914
915
916
917
918
919
 
# File 'lib/external_index.rb', line 906

def write(array, pos=nil)
  case array
  when Array
    check_framed_array(array)
    prepare_write_to_pos(pos)
    write_framed_array(array)
  when ExternalIndex
    check_index(array)
    prepare_write_to_pos(pos)
    write_index(array)
  else  
    raise ArgumentError, "could not convert #{array.class} to Array or ExternalIndex"
  end
end