Class: Python::Pickle::Deserializer Private

Inherits:
Object
  • Object
show all
Defined in:
lib/python/pickle/deserializer.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Handles deserializing a stream of Python Pickle instructions.

Constant Summary collapse

OBJECT_CLASS =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

The Python object class.

PyClass.new('__builtins__','object')

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(constants: nil, extensions: nil, buffers: nil) ⇒ Deserializer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Initializes the deserializer.

Parameters:

  • extensions (Hash{Integer => Object}) (defaults to: nil)

    A Hash of registered extension IDs and their Objects.

  • constants (Hash{String => Hash{String => Class,Method}}) (defaults to: nil)

    An optional mapping of custom Python constant names to Ruby classes or methods.

  • buffers (Enumerable, nil) (defaults to: nil)

    An enumerable list of out-of-band buffers.



65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
# File 'lib/python/pickle/deserializer.rb', line 65

def initialize(constants: nil, extensions: nil, buffers: nil)
  @meta_stack = []
  @stack = []
  @memo  = []

  @constants = {
    # Python 2.x
    'copy_reg' => {
      '_reconstructor' => method(:copyreg_reconstructor)
    },

    '__builtin__' => {
      'object'    => OBJECT_CLASS,
      'set'       => Set,
      'bytearray' => ByteArray
    },

    # Python 3.x
    'builtins' => {
      'object'    => OBJECT_CLASS,
      'set'       => Set,
      'bytearray' => ByteArray
    }
  }
  @constants.merge!(constants) if constants

  @extensions = {}
  @extensions.merge!(extensions) if extensions

  @buffers = buffers.each if buffers
end

Instance Attribute Details

#buffersEnumerator? (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

An enumerable list of out-of-band buffers.

Returns:

  • (Enumerator, nil)


47
48
49
# File 'lib/python/pickle/deserializer.rb', line 47

def buffers
  @buffers
end

#constantsHash{String => Hash{String => Class,Method}} (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Mapping of Python constants to Ruby classes and methods.

Returns:

  • (Hash{String => Hash{String => Class,Method}})


37
38
39
# File 'lib/python/pickle/deserializer.rb', line 37

def constants
  @constants
end

#extensionsHash{Integer => Object} (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Mapping of Python Pickle extension codes to Ruby objects.

Returns:

  • (Hash{Integer => Object})


42
43
44
# File 'lib/python/pickle/deserializer.rb', line 42

def extensions
  @extensions
end

#memoArray (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The memo dictionary.

Returns:

  • (Array)


32
33
34
# File 'lib/python/pickle/deserializer.rb', line 32

def memo
  @memo
end

#meta_stackArray (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The meta-stack for saving/restoring #stack.

Returns:

  • (Array)


22
23
24
# File 'lib/python/pickle/deserializer.rb', line 22

def meta_stack
  @meta_stack
end

#stackArray (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The object stack.

Returns:

  • (Array)


27
28
29
# File 'lib/python/pickle/deserializer.rb', line 27

def stack
  @stack
end

Instance Method Details

#copyreg_reconstructor(klass, super_class, init_arg) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Implements Python's copyreg._reconstructor function for Python Pickle protocol 0 compatibility.

Parameters:

  • class (PyClass, Class)

    The Python or Ruby class to be initialized.

  • super_class (PyClass)

    The Python super-class of the class.

  • init_arg (Array, nil)

    The argument(s) that will be passed to the class'es new method.



467
468
469
# File 'lib/python/pickle/deserializer.rb', line 467

def copyreg_reconstructor(klass,super_class,init_arg)
  klass.new(*init_arg)
end

#execute(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a Python Pickle instruction.



120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
# File 'lib/python/pickle/deserializer.rb', line 120

def execute(instruction)
  case instruction
  when Instructions::Proto,
       Instructions::Frame
    # no-op
  when Instructions::Get,
       Instructions::BinGet,
       Instructions::LongBinGet
    execute_get(instruction)
  when Instructions::MARK     then execute_mark
  when Instructions::POP_MARK then execute_pop_mark
  when Instructions::DUP      then execute_dup
  when Instructions::Put,
       Instructions::BinPut
    execute_put(instruction)
  when Instructions::POP     then execute_pop
  when Instructions::MEMOIZE then execute_memoize
  when Instructions::Ext1,
       Instructions::Ext2,
       Instructions::Ext4
    execute_ext(instruction)
  when Instructions::NONE     then execute_none
  when Instructions::NEWTRUE  then execute_newtrue
  when Instructions::NEWFALSE then execute_newfalse
  when Instructions::Float,
       Instructions::BinFloat,
       Instructions::Int,
       Instructions::BinInt1,
       Instructions::Long,
       Instructions::Long1,
       Instructions::Long4,
       Instructions::BinBytes,
       Instructions::ShortBinBytes,
       Instructions::BinBytes8,
       Instructions::String,
       Instructions::BinString,
       Instructions::ShortBinString,
       Instructions::BinUnicode,
       Instructions::ShortBinUnicode,
       Instructions::BinUnicode8
    @stack.push(instruction.value)
  when Instructions::ByteArray8      then execute_byte_array8(instruction)
  when Instructions::EMPTY_LIST      then execute_empty_list
  when Instructions::EMPTY_TUPLE     then execute_empty_tuple
  when Instructions::TUPLE           then execute_tuple
  when Instructions::EMPTY_DICT      then execute_empty_dict
  when Instructions::EMPTY_SET       then execute_empty_set
  when Instructions::FROZENSET       then execute_frozenset
  when Instructions::APPEND          then execute_append
  when Instructions::APPENDS         then execute_appends
  when Instructions::ADDITEMS        then execute_additems
  when Instructions::LIST            then execute_list
  when Instructions::TUPLE1          then execute_tuple1
  when Instructions::TUPLE2          then execute_tuple2
  when Instructions::TUPLE3          then execute_tuple3
  when Instructions::DICT            then execute_dict
  when Instructions::Global          then execute_global(instruction)
  when Instructions::STACK_GLOBAL    then execute_stack_global
  when Instructions::Inst            then execute_inst(instruction)
  when Instructions::OBJ             then execute_obj
  when Instructions::NEWOBJ          then execute_newobj
  when Instructions::NEWOBJ_EX       then execute_newobj_ex
  when Instructions::REDUCE          then execute_reduce
  when Instructions::BUILD           then execute_build
  when Instructions::SETITEM         then execute_setitem
  when Instructions::SETITEMS        then execute_setitems
  when Instructions::NEXT_BUFFER     then execute_next_buffer
  when Instructions::READONLY_BUFFER then execute_readonly_buffer
  when Instructions::STOP
    return :halt, @stack.pop
  else
    raise(NotImplementedError,"instruction is currently not fully supported: #{instruction.inspect}")
  end
end

#execute_additemsObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a ADDITEMS instruction.

Since:

  • 0.2.0



391
392
393
394
395
396
397
398
399
400
401
402
# File 'lib/python/pickle/deserializer.rb', line 391

def execute_additems
  items = pop_meta_stack
  set   = @stack.last

  unless set.kind_of?(Set)
    raise(DeserializationError,"cannot add items #{items.inspect} to a non-Set object: #{set.inspect}")
  end

  items.each do |item|
    set << item
  end
end

#execute_appendObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an APPEND instruction.



356
357
358
359
360
361
362
363
364
365
# File 'lib/python/pickle/deserializer.rb', line 356

def execute_append
  item = @stack.pop
  list = @stack.last

  unless (list.kind_of?(Array) || list.kind_of?(Set))
    raise(DeserializationError,"cannot append element #{item.inspect} onto a non-Array: #{list.inspect}")
  end

  list << item
end

#execute_appendsObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an APPENDS instruction.



370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
# File 'lib/python/pickle/deserializer.rb', line 370

def execute_appends
  items = pop_meta_stack
  list  = @stack.last

  case list
  when Array
    list.concat(items)
  when Set
    items.each do |item|
      list << item
    end
  else
    raise(DeserializationError,"cannot append elements #{items.inspect} onto a non-Array: #{list.inspect}")
  end
end

#execute_buildObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a BUILD instruction.



588
589
590
591
592
593
594
595
596
597
598
599
# File 'lib/python/pickle/deserializer.rb', line 588

def execute_build
  arg    = @stack.pop
  object = @stack.last

  if object.respond_to?(:__setstate__)
    object.__setstate__(arg)
  elsif object.kind_of?(Hash)
    object.merge!(arg)
  else
    raise(DeserializationError,"cannot execute BUILD on an object that does not define a __setstate__ method or is not a Hash: #{object.inspect}")
  end
end

#execute_byte_array8(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a BYTEARRAY8 instruction.



297
298
299
# File 'lib/python/pickle/deserializer.rb', line 297

def execute_byte_array8(instruction)
  @stack.push(ByteArray.new(instruction.value))
end

#execute_dictObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a DICT instruction.



442
443
444
445
446
447
448
449
450
451
452
# File 'lib/python/pickle/deserializer.rb', line 442

def execute_dict
  pairs    = pop_meta_stack
  new_dict = {}

  until pairs.empty?
    key, value = pairs.pop(2)
    new_dict[key] = value
  end

  @stack.push(new_dict)
end

#execute_dupObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a DUP instruction.



224
225
226
# File 'lib/python/pickle/deserializer.rb', line 224

def execute_dup
  @stack.push(@stack.last)
end

#execute_empty_dictObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an EMPTY_DICT instruction.



326
327
328
# File 'lib/python/pickle/deserializer.rb', line 326

def execute_empty_dict
  @stack.push({})
end

#execute_empty_listObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the EMPTY_LIST instruction.



304
305
306
# File 'lib/python/pickle/deserializer.rb', line 304

def execute_empty_list
  @stack.push([])
end

#execute_empty_setObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an EMPTY_SET instruction.

Since:

  • 0.2.0



335
336
337
# File 'lib/python/pickle/deserializer.rb', line 335

def execute_empty_set
  @stack.push(Set.new)
end

#execute_empty_tupleObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the EMPTY_TUPLE instruction.



311
312
313
# File 'lib/python/pickle/deserializer.rb', line 311

def execute_empty_tuple
  @stack.push(Tuple.new)
end

#execute_ext(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a EXT1, EXT2, or EXT4 instruction.

Parameters:

Raises:



264
265
266
267
268
269
270
271
# File 'lib/python/pickle/deserializer.rb', line 264

def execute_ext(instruction)
  ext_id = instruction.value
  object = @extensions.fetch(ext_id) do
    raise(DeserializationError,"unknown extension ID: #{ext_id.inspect}")
  end

  @stack.push(object)
end

#execute_frozensetObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a FROZENSET instruction.

Since:

  • 0.2.0



344
345
346
347
348
349
350
351
# File 'lib/python/pickle/deserializer.rb', line 344

def execute_frozenset
  items = pop_meta_stack

  set = Set.new(items)
  set.freeze

  @stack.push(set)
end

#execute_get(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a GET, BINGET, or LONG_BINGET instruction.

Parameters:



201
202
203
204
205
# File 'lib/python/pickle/deserializer.rb', line 201

def execute_get(instruction)
  index = instruction.value

  @stack.push(@memo[index])
end

#execute_global(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a GLOBAL instruction.

Parameters:



497
498
499
500
501
502
503
# File 'lib/python/pickle/deserializer.rb', line 497

def execute_global(instruction)
  namespace = instruction.namespace
  name      = instruction.name
  constant  = resolve_constant(namespace,name)

  @stack.push(constant)
end

#execute_inst(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an INST instruction.

Since:

  • 0.2.0



520
521
522
523
524
525
526
527
528
# File 'lib/python/pickle/deserializer.rb', line 520

def execute_inst(instruction)
  namespace = instruction.namespace
  name      = instruction.name
  py_class  = resolve_constant(namespace,name)
  args      = pop_meta_stack
  py_object = py_class.new(*args)

  @stack.push(py_object)
end

#execute_listObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a LIST instruction.



407
408
409
410
# File 'lib/python/pickle/deserializer.rb', line 407

def execute_list
  elements = pop_meta_stack
  @stack.push(elements)
end

#execute_markObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a MARK instruction.



210
211
212
# File 'lib/python/pickle/deserializer.rb', line 210

def execute_mark
  push_meta_stack
end

#execute_memoizeObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the MEMOIZE instruction.



251
252
253
# File 'lib/python/pickle/deserializer.rb', line 251

def execute_memoize
  @memo.push(@stack.last)
end

#execute_newfalseObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWFALSE instruction.



290
291
292
# File 'lib/python/pickle/deserializer.rb', line 290

def execute_newfalse
  @stack.push(false)
end

#execute_newobjObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWOBJ instruction.



545
546
547
548
549
550
# File 'lib/python/pickle/deserializer.rb', line 545

def execute_newobj
  py_class, args = @stack.pop(2)
  py_object = py_class.new(*args)

  @stack.push(py_object)
end

#execute_newobj_exObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWOBJ_EX instruction.



555
556
557
558
559
560
561
562
563
564
565
566
# File 'lib/python/pickle/deserializer.rb', line 555

def execute_newobj_ex
  py_class, args, kwargs = @stack.pop(3)
  py_object = if kwargs
                kwargs = kwargs.transform_keys(&:to_sym)

                py_class.new(*args,**kwargs)
              else
                py_class.new(*args)
              end

  @stack.push(py_object)
end

#execute_newtrueObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWTRUE instruction.



283
284
285
# File 'lib/python/pickle/deserializer.rb', line 283

def execute_newtrue
  @stack.push(true)
end

#execute_next_bufferObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEXT_BUFFER instruction.

Since:

  • 0.2.0



637
638
639
640
641
642
643
# File 'lib/python/pickle/deserializer.rb', line 637

def execute_next_buffer
  unless @buffers
    raise(DeserializationError,"pickle stream includes a NEXT_BUFFER instruction, but no buffers were provided")
  end

  @stack.push(@buffers.next)
end

#execute_noneObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NONE instruction.



276
277
278
# File 'lib/python/pickle/deserializer.rb', line 276

def execute_none
  @stack.push(nil)
end

#execute_objObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an OBJ instruction.

Since:

  • 0.2.0



535
536
537
538
539
540
# File 'lib/python/pickle/deserializer.rb', line 535

def execute_obj
  py_class, *args = pop_meta_stack
  py_object = py_class.new(*args)

  @stack.push(py_object)
end

#execute_popObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the POP instruction.



244
245
246
# File 'lib/python/pickle/deserializer.rb', line 244

def execute_pop
  @stack.pop
end

#execute_pop_markObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a POP_MARK instruction.



217
218
219
# File 'lib/python/pickle/deserializer.rb', line 217

def execute_pop_mark
  pop_meta_stack
end

#execute_put(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a PUT, BINPUT, or LONG_BINPUT instruction.

Parameters:



234
235
236
237
238
239
# File 'lib/python/pickle/deserializer.rb', line 234

def execute_put(instruction)
  index = instruction.value
  value = @stack.last

  @memo[index] = value
end

#execute_readonly_bufferObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a READONLY_BUFFER instruction.

Since:

  • 0.2.0



650
651
652
653
# File 'lib/python/pickle/deserializer.rb', line 650

def execute_readonly_buffer
  buffer = @stack.last
  buffer.freeze
end

#execute_reduceObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a REDUCE instruction.



571
572
573
574
575
576
577
578
579
580
581
582
583
# File 'lib/python/pickle/deserializer.rb', line 571

def execute_reduce
  callable, arg = @stack.pop(2)
  object = case callable
           when PyClass, Class
             callable.new(*arg)
           when Method
             callable.call(*arg)
           else
             raise(DeserializationError,"cannot execute REDUCE on a non-class: #{callable.inspect}")
           end

  @stack.push(object)
end

#execute_setitemObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a SETITEM instruction.



604
605
606
607
608
609
610
611
612
613
# File 'lib/python/pickle/deserializer.rb', line 604

def execute_setitem
  key, value = @stack.pop(2)
  dict = @stack.last

  unless dict.kind_of?(Hash)
    raise(DeserializationError,"cannot set key (#{key.inspect}) and value (#{value.inspect}) into non-Hash: #{dict.inspect}")
  end

  dict[key] = value
end

#execute_setitemsObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a SETITEMS instruction.



618
619
620
621
622
623
624
625
626
627
628
629
630
# File 'lib/python/pickle/deserializer.rb', line 618

def execute_setitems
  pairs = pop_meta_stack
  dict  = @stack.last

  unless dict.kind_of?(Hash)
    raise(DeserializationError,"cannot set key value pairs (#{pairs.inspect}) into non-Hash: #{dict.inspect}")
  end

  until pairs.empty?
    key, value = pairs.pop(2)
    dict[key] = value
  end
end

#execute_stack_globalObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a STACK_GLOBAL instruction.



508
509
510
511
512
513
# File 'lib/python/pickle/deserializer.rb', line 508

def execute_stack_global
  namespace, name = @stack.pop(2)
  constant = resolve_constant(namespace,name)

  @stack.push(constant)
end

#execute_tupleObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE instruction.



318
319
320
321
# File 'lib/python/pickle/deserializer.rb', line 318

def execute_tuple
  items = Tuple.new(pop_meta_stack)
  @stack.push(items)
end

#execute_tuple1Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE1 instruction.



415
416
417
418
419
# File 'lib/python/pickle/deserializer.rb', line 415

def execute_tuple1
  new_tuple = Tuple.new(@stack.pop(1))

  @stack.push(new_tuple)
end

#execute_tuple2Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE2 instruction.



424
425
426
427
428
# File 'lib/python/pickle/deserializer.rb', line 424

def execute_tuple2
  new_tuple = Tuple.new(@stack.pop(2))

  @stack.push(new_tuple)
end

#execute_tuple3Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE3 instruction.



433
434
435
436
437
# File 'lib/python/pickle/deserializer.rb', line 433

def execute_tuple3
  new_tuple = Tuple.new(@stack.pop(3))

  @stack.push(new_tuple)
end

#pop_meta_stackArray

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Pops a previous stack off of #meta_stack and restores #stack.

Returns:

  • (Array)

    The current #stack values will be returned.



111
112
113
114
115
# File 'lib/python/pickle/deserializer.rb', line 111

def pop_meta_stack
  items  = @stack
  @stack = (@meta_stack.pop || [])
  return items
end

#push_meta_stackObject

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Pushes the #stack onto the #meta_stack.



100
101
102
103
# File 'lib/python/pickle/deserializer.rb', line 100

def push_meta_stack
  @meta_stack.push(@stack)
  @stack = []
end

#resolve_constant(namespace, name) ⇒ Class, ...

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Resolves a constant that exists in a Python namespace.

Parameters:

  • namespace (String)

    The namespace name.

  • name (String)

    The name of the constant within the namespace.

Returns:

  • (Class, PyClass, Method, nil)

    The resolved class or method.



483
484
485
486
487
488
489
# File 'lib/python/pickle/deserializer.rb', line 483

def resolve_constant(namespace,name)
  constant = if (mod = @constants[namespace])
               mod[name]
             end

  return constant || PyClass.new(namespace,name)
end