Class: Kaitai::Struct::Stream

Inherits:
Object
  • Object
show all
Defined in:
lib/kaitai/struct/struct.rb

Overview

Kaitai::Struct::Stream is an implementation of Kaitai Stream API for Ruby. It’s implemented as a wrapper for generic IO objects.

It provides a wide variety of simple methods to read (parse) binary representations of primitive types, such as integer and floating point numbers, byte arrays and strings, and also provides stream positioning / navigation methods with unified cross-language and cross-toolkit semantics.

Typically, end users won’t access Kaitai Stream class manually, but would describe a binary structure format using .ksy language and then would use Kaitai Struct compiler to generate source code in desired target language. That code, in turn, would use this class and API to do the actual parsing job.

Defined Under Namespace

Classes: UnexpectedDataError

Constant Summary collapse

@@big_endian =

Test endianness of the platform

[0x0102].pack('s') == [0x0102].pack('n')

Stream positioning collapse

Integer numbers collapse

Floating point numbers collapse

Unaligned bit values collapse

Byte arrays collapse

Byte array processing collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(arg) ⇒ Stream

Constructs new Kaitai Stream object.

Parameters:

  • arg (String, IO) —

    if String, it will be used as byte array to read data from; if IO, if will be used literally as source of data



97
98
99
100
101
102
103
104
105
106
 
# File 'lib/kaitai/struct/struct.rb', line 97

def initialize(arg)
  if arg.is_a?(String)
    @_io = StringIO.new(arg)
  elsif arg.is_a?(IO)
    @_io = arg
  else
    raise TypeError.new('can be initialized with IO or String only')
  end
  align_to_byte
end

Class Method Details

.bytes_strip_right(bytes, pad_byte) ⇒ Object 



426
427
428
429
430
431
432
433
 
# File 'lib/kaitai/struct/struct.rb', line 426

def self.bytes_strip_right(bytes, pad_byte)
  new_len = bytes.length
  while new_len > 0 and bytes.getbyte(new_len - 1) == pad_byte
    new_len -= 1
  end

  bytes[0, new_len]
end

.bytes_terminate(bytes, term, include_term) ⇒ Object 



435
436
437
438
439
440
441
442
443
 
# File 'lib/kaitai/struct/struct.rb', line 435

def self.bytes_terminate(bytes, term, include_term)
  new_len = 0
  max_len = bytes.length
  while bytes.getbyte(new_len) != term and new_len < max_len
    new_len += 1
  end
  new_len += 1 if include_term and new_len < max_len
  bytes[0, new_len]
end

.open(filename) ⇒ Object

Convenience method to create a Kaitai Stream object, opening a local file with a given filename.

Parameters:

  • filename (String) —

    local file to open



112
113
114
 
# File 'lib/kaitai/struct/struct.rb', line 112

def self.open(filename)
  self.new(File.open(filename, 'rb:ASCII-8BIT'))
end

.process_rotate_left(data, amount, group_size) ⇒ String

Performs a circular left rotation shift for a given buffer by a given amount of bits, using groups of groupSize bytes each time. Right circular rotation should be performed using this procedure with corrected amount.

Parameters:

  • data (String) —

    source data to process

  • amount (Fixnum) —

    number of bits to shift by

  • group_size (Fixnum) —

    number of bytes per group to shift

Returns:

  • (String) —

    copy of source array with requested shift applied

Raises:

  • (NotImplementedError) 


502
503
504
505
506
507
508
509
510
511
512
513
514
515
 
# File 'lib/kaitai/struct/struct.rb', line 502

def self.process_rotate_left(data, amount, group_size)
  raise NotImplementedError.new("unable to rotate group #{group_size} bytes yet") unless group_size == 1

  mask = group_size * 8 - 1
  anti_amount = -amount & mask

  # NB: actually, left bit shift (<<) in Ruby would have required
  # truncation to type_bits size (i.e. something like "& 0xff" for
  # group_size == 8), but we can skip this one, because later these
  # number would be packed with Array#pack, which will do truncation
  # anyway

  data.bytes.map { |x| (x << amount) | (x >> anti_amount) }.pack('C*')
end

.process_xor_many(data, key) ⇒ String

Performs a XOR processing with given data, XORing every byte of input with a key array, repeating key array many times, if necessary (i.e. if data array is longer than key array). Uses pure Ruby implementation suggested by [Thomas Leitner](github.com/gettalong), borrowed from github.com/fny/xorcist/blob/master/bin/benchmark

Parameters:

  • data (String) —

    data to process

  • key (String) —

    array of bytes to XOR with

Returns:

  • (String) —

    processed data



478
479
480
481
482
483
484
485
486
487
488
489
490
491
 
# File 'lib/kaitai/struct/struct.rb', line 478

def self.process_xor_many(data, key)
  out = data.dup
  kl = key.length
  ki = 0
  i = 0
  max = data.length
  while i < max
    out.setbyte(i, data.getbyte(i) ^ key.getbyte(ki))
    ki += 1
    ki = 0 if ki >= kl
    i += 1
  end
  out
end

.process_xor_one(data, key) ⇒ String

Performs a XOR processing with given data, XORing every byte of input with a single given value. Uses pure Ruby implementation suggested by [Thomas Leitner](github.com/gettalong), borrowed from github.com/fny/xorcist/blob/master/bin/benchmark

Parameters:

  • data (String) —

    data to process

  • key (Fixnum) —

    value to XOR with

Returns:

  • (String) —

    processed data



457
458
459
460
461
462
463
464
465
466
 
# File 'lib/kaitai/struct/struct.rb', line 457

def self.process_xor_one(data, key)
  out = data.dup
  i = 0
  max = data.length
  while i < max
    out.setbyte(i, data.getbyte(i) ^ key)
    i += 1
  end
  out
end

.resolve_enum(enum_map, value) ⇒ Object

Resolves value using enum: if the value is not found in the map, we’ll just use literal value per se.



522
523
524
 
# File 'lib/kaitai/struct/struct.rb', line 522

def self.resolve_enum(enum_map, value)
  enum_map[value] || value
end

Instance Method Details

#align_to_byte ⇒ Object 



289
290
291
292
 
# File 'lib/kaitai/struct/struct.rb', line 289

def align_to_byte
  @bits_left = 0
  @bits = 0
end

#close ⇒ Object

Closes underlying IO object.



118
119
120
 
# File 'lib/kaitai/struct/struct.rb', line 118

def close
  @_io.close
end

#ensure_fixed_contents(expected) ⇒ String

Unused since Kaitai Struct Compiler v0.9+ - compatibility with older versions.

Reads next len bytes from the stream and ensures that they match expected fixed byte array. If they differ, throws a UnexpectedDataError runtime exception.

Parameters:

  • expected (String) —

    contents to be expected

Returns:

  • (String) —

    read bytes as byte array, which are guaranteed to equal to expected

Raises:



419
420
421
422
423
424
 
# File 'lib/kaitai/struct/struct.rb', line 419

def ensure_fixed_contents(expected)
  len = expected.bytesize
  actual = @_io.read(len)
  raise UnexpectedDataError.new(actual, expected) if actual != expected
  actual
end

#eof? ⇒ true, false

Check if stream pointer is at the end of stream.

Returns:

  • (true, false) —

    true if we are located at the end of the stream



130
 
# File 'lib/kaitai/struct/struct.rb', line 130

def eof?; @_io.eof? and @bits_left == 0; end

#pos ⇒ Fixnum

Get current position of a stream pointer.

Returns:

  • (Fixnum) —

    pointer position, number of bytes from the beginning of the stream



140
 
# File 'lib/kaitai/struct/struct.rb', line 140

def pos; @_io.pos; end

#read_bits_int(n) ⇒ Object

Unused since Kaitai Struct Compiler v0.9+ - compatibility with older versions.



325
326
327
 
# File 'lib/kaitai/struct/struct.rb', line 325

def read_bits_int(n)
  read_bits_int_be(n)
end

#read_bits_int_be(n) ⇒ Object 



294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/kaitai/struct/struct.rb', line 294

def read_bits_int_be(n)
  res = 0

  bits_needed = n - @bits_left
  @bits_left = -bits_needed % 8

  if bits_needed > 0
    # 1 bit  => 1 byte
    # 8 bits => 1 byte
    # 9 bits => 2 bytes
    bytes_needed = ((bits_needed - 1) / 8) + 1 # `ceil(bits_needed / 8)`
    buf = read_bytes(bytes_needed)
    buf.each_byte { |byte|
      res = res << 8 | byte
    }

    new_bits = res
    res = res >> @bits_left | @bits << bits_needed
    @bits = new_bits # will be masked at the end of the function
  else
    res = @bits >> -bits_needed # shift unneeded bits out
  end

  mask = (1 << @bits_left) - 1 # `@bits_left` is in range 0..7
  @bits &= mask

  res
end

#read_bits_int_le(n) ⇒ Object 



329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
 
# File 'lib/kaitai/struct/struct.rb', line 329

def read_bits_int_le(n)
  res = 0
  bits_needed = n - @bits_left

  if bits_needed > 0 then
    # 1 bit  => 1 byte
    # 8 bits => 1 byte
    # 9 bits => 2 bytes
    bytes_needed = ((bits_needed - 1) / 8) + 1 # `ceil(bits_needed / 8)`
    buf = read_bytes(bytes_needed)
    i = 0
    buf.each_byte { |byte|
      res |= byte << (i * 8)
      i += 1
    }

    new_bits = res >> bits_needed
    res = res << @bits_left | @bits
    @bits = new_bits
  else
    res = @bits
    @bits >>= n
  end

  @bits_left = -bits_needed % 8

  mask = (1 << n) - 1 # no problem with this in Ruby (arbitrary precision integers)
  res &= mask
  return res
end

#read_bytes(n) ⇒ String

Reads designated number of bytes from the stream.

Parameters:

  • n (Fixnum) —

    number of bytes to read

Returns:

  • (String) —

    read bytes as byte array

Raises:

  • (EOFError) —

    if there were less bytes than requested available in the stream



370
371
372
373
374
375
376
377
378
379
 
# File 'lib/kaitai/struct/struct.rb', line 370

def read_bytes(n)
  r = @_io.read(n)
  if r
    rl = r.bytesize
  else
    rl = 0
  end
  raise EOFError.new("attempted to read #{n} bytes, got only #{rl}") if rl < n
  r
end

#read_bytes_full ⇒ String

Reads all the remaining bytes in a stream as byte array.

Returns:

  • (String) —

    all remaining bytes in a stream as byte array



384
385
386
 
# File 'lib/kaitai/struct/struct.rb', line 384

def read_bytes_full
  @_io.read
end

#read_bytes_term(term, include_term, consume_term, eos_error) ⇒ Object 



388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
 
# File 'lib/kaitai/struct/struct.rb', line 388

def read_bytes_term(term, include_term, consume_term, eos_error)
  r = ''
  loop {
    if @_io.eof?
      if eos_error
        raise EOFError.new("end of stream reached, but no terminator #{term} found")
      else
        return r
      end
    end
    c = @_io.getc
    if c.ord == term
      r << c if include_term
      @_io.seek(@_io.pos - 1) unless consume_term
      return r
    end
    r << c
  }
end

#read_f4be ⇒ Object

Big-endian



265
266
267
 
# File 'lib/kaitai/struct/struct.rb', line 265

def read_f4be
  read_bytes(4).unpack('g')[0]
end

#read_f4le ⇒ Object

Little-endian



277
278
279
 
# File 'lib/kaitai/struct/struct.rb', line 277

def read_f4le
  read_bytes(4).unpack('e')[0]
end

#read_f8be ⇒ Object 



269
270
271
 
# File 'lib/kaitai/struct/struct.rb', line 269

def read_f8be
  read_bytes(8).unpack('G')[0]
end

#read_f8le ⇒ Object 



281
282
283
 
# File 'lib/kaitai/struct/struct.rb', line 281

def read_f8le
  read_bytes(8).unpack('E')[0]
end

#read_s1 ⇒ Object

Signed



155
156
157
 
# File 'lib/kaitai/struct/struct.rb', line 155

def read_s1
  read_bytes(1).unpack('c')[0]
end

#read_s2be ⇒ Object

.….….….….….….….….….….….….….….….….….… Big-endian .….….….….….….….….….….….….….….….….….…



163
164
165
 
# File 'lib/kaitai/struct/struct.rb', line 163

def read_s2be
  to_signed(read_u2be, SIGN_MASK_16)
end

#read_s2le ⇒ Object

.….….….….….….….….….….….….….….….….….… Little-endian .….….….….….….….….….….….….….….….….….…



185
186
187
 
# File 'lib/kaitai/struct/struct.rb', line 185

def read_s2le
  to_signed(read_u2le, SIGN_MASK_16)
end

#read_s4be ⇒ Object 



167
168
169
 
# File 'lib/kaitai/struct/struct.rb', line 167

def read_s4be
  to_signed(read_u4be, SIGN_MASK_32)
end

#read_s4le ⇒ Object 



189
190
191
 
# File 'lib/kaitai/struct/struct.rb', line 189

def read_s4le
  to_signed(read_u4le, SIGN_MASK_32)
end

#read_s8be ⇒ Object 



172
173
174
 
# File 'lib/kaitai/struct/struct.rb', line 172

def read_s8be
  read_bytes(8).unpack('q')[0]
end

#read_s8le ⇒ Object 



194
195
196
 
# File 'lib/kaitai/struct/struct.rb', line 194

def read_s8le
  read_bytes(8).unpack('q')[0]
end

#read_u1 ⇒ Object

Unsigned



207
208
209
 
# File 'lib/kaitai/struct/struct.rb', line 207

def read_u1
  read_bytes(1).unpack('C')[0]
end

#read_u2be ⇒ Object

.….….….….….….….….….….….….….….….….….… Big-endian .….….….….….….….….….….….….….….….….….…



215
216
217
 
# File 'lib/kaitai/struct/struct.rb', line 215

def read_u2be
  read_bytes(2).unpack('n')[0]
end

#read_u2le ⇒ Object

.….….….….….….….….….….….….….….….….….… Little-endian .….….….….….….….….….….….….….….….….….…



238
239
240
 
# File 'lib/kaitai/struct/struct.rb', line 238

def read_u2le
  read_bytes(2).unpack('v')[0]
end

#read_u4be ⇒ Object 



219
220
221
 
# File 'lib/kaitai/struct/struct.rb', line 219

def read_u4be
  read_bytes(4).unpack('N')[0]
end

#read_u4le ⇒ Object 



242
243
244
 
# File 'lib/kaitai/struct/struct.rb', line 242

def read_u4le
  read_bytes(4).unpack('V')[0]
end

#read_u8be ⇒ Object 



224
225
226
 
# File 'lib/kaitai/struct/struct.rb', line 224

def read_u8be
  read_bytes(8).unpack('Q')[0]
end

#read_u8le ⇒ Object 



247
248
249
 
# File 'lib/kaitai/struct/struct.rb', line 247

def read_u8le
  read_bytes(8).unpack('Q')[0]
end

#seek(x) ⇒ Object

Set stream pointer to designated position.

Parameters:

  • x (Fixnum) —

    new position (offset in bytes from the beginning of the stream)



135
 
# File 'lib/kaitai/struct/struct.rb', line 135

def seek(x); @_io.seek(x); end

#size ⇒ Fixnum

Get total size of the stream in bytes.

Returns:

  • (Fixnum) —

    size of the stream in bytes



145
 
# File 'lib/kaitai/struct/struct.rb', line 145

def size; @_io.size; end