Class: XZ::StreamReader

Inherits:
Stream
  • Object
show all
Defined in:
lib/xz/stream_reader.rb

Overview

An IO-like reader class for XZ-compressed data, allowing you to access XZ-compressed data as if it was a normal IO object, but please note you can’t seek in the data–this doesn’t make much sense anyway. Where would you want to seek? The plain or the XZ data?

A StreamReader object actually wraps another IO object it reads the compressed data from; you can either pass this IO object directly to the ::new method, effectively allowing you to pass any IO-like thing you can imagine (just ensure it is readable), or you can pass a path to a file to ::open, in which case StreamReader will open the path using Ruby’s File class internally. If you use ::open’s block form, the method will take care of properly closing both the liblzma stream and the File instance correctly.

Instance Attribute Summary collapse

Attributes inherited from Stream

#external_encoding, #internal_encoding, #lineno

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Stream

#<<, #advise, #close, #close_read, #close_write, #closed?, #each, #each_byte, #each_char, #each_codepoint, #eof, #finish, #finished?, #getbyte, #getc, #gets, #lzma_code, #pos, #print, #printf, #putc, #puts, #readbyte, #readchar, #readline, #reopen, #set_encoding, #to_io, #write

Constructor Details

#initialize(delegate_io, memory_limit: XZ::LibLZMA::UINT64_MAX, flags: [:tell_unsupported_check], external_encoding: nil, internal_encoding: nil) ⇒ StreamReader

Creates a new instance that is wrapped around the given IO object.

Parameters

Positional parameters

delegate_io

The underlying IO object to read the compressed data from. This IO object has to have been opened in binary mode, otherwise you are likely to receive exceptions indicating that the compressed data is corrupt.

Keyword arguments

memory_limit (UINT64_MAX)

If not XZ::LibLZMA::UINT64_MAX, makes liblzma use no more memory than memory_limit bytes.

flags ([:tell_unsupported_check])

Additional flags passed to liblzma (an array). Possible flags are:

:tell_no_check

Spit out a warning if the archive hasn’t an integrity checksum.

:tell_unsupported_check

Spit out a warning if the archive has an unsupported checksum type.

:concatenated

Decompress concatenated archives.

external_encoding (Encoding.default_external)

Assume the decompressed data inside the XZ is encoded in this encoding. Defaults to Encoding.default_external, which in turn defaults to the environment.

internal_encoding (Encoding.default_internal)

Request that the data found in the XZ file (which is assumed to be in the encoding specified by external_encoding) to be transcoded into this encoding. Defaults to Encoding.default_internal, which defaults to nil, which means to not transcode anything.

Return value

The newly created instance.

Remarks

The strings returned from the reader will be in the encoding specified by the internal_encoding parameter. If that parameter is nil (default), then they will be in the encoding specified by external_encoding.

This method used to accept a block in earlier versions. Since version 1.0.0, this behaviour has been removed to synchronise the API with Ruby’s own GzipReader.open.

This method doesn’t close the underlying IO or the liblzma stream. You need to call #finish or #close manually; see ::open for a method that takes a block to automate this.

Example

file = File.open("compressed.txt.xz", "rb") # Note binary mode
xz = XZ::StreamReader.open(file)
puts xz.read #=> I love Ruby
xz.close # closes both `xz' and `file'

file = File.open("compressed.txt.xz", "rb") # Note binary mode
xz = XZ::StreamReader.open(file)
puts xz.read #=> I love Ruby
xz.finish # closes only `xz'
file.close # Now close `file' manually

Raises:

  • (ArgumentError) 


169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
 
# File 'lib/xz/stream_reader.rb', line 169

def initialize(delegate_io, memory_limit: XZ::LibLZMA::UINT64_MAX, flags: [:tell_unsupported_check], external_encoding: nil, internal_encoding: nil)
  super(delegate_io)
  raise(ArgumentError, "When specifying the internal encoding, the external encoding must also be specified") if internal_encoding && !external_encoding
  raise(ArgumentError, "Memory limit out of range") unless memory_limit > 0 && memory_limit <= XZ::LibLZMA::UINT64_MAX

  @memory_limit = memory_limit
  @readbuf = String.new
  @readbuf.force_encoding(Encoding::BINARY)

  if external_encoding
    encargs = []
    encargs << external_encoding
    encargs << internal_encoding if internal_encoding
    set_encoding(*encargs)
  end

  @allflags = flags.reduce(0) do |val, flag|
    flag = XZ::LibLZMA::LZMA_DECODE_FLAGS[flag] || raise(ArgumentError, "Unknown flag #{flag}")
    val | flag
  end

  res = XZ::LibLZMA.lzma_stream_decoder(@lzma_stream.to_ptr,
                                    @memory_limit,
                                    @allflags)
  XZ::LZMAError.raise_if_necessary(res)
end

Instance Attribute Details

#memory_limit ⇒ Object (readonly)

The memory limit configured for this lzma decoder.



45
46
47
 
# File 'lib/xz/stream_reader.rb', line 45

def memory_limit
  @memory_limit
end

Class Method Details

.open(filename, **args) ⇒ Object

call-seq:

open(filename [, kw]) → stream_reader
open(filename [, kw]){|sr| ...} → stream_reader

Open the given file and wrap a new instance around it with ::new. If you use the block form, both the internally created File instance and the liblzma stream will be closed automatically for you.

Parameters

filename

Path to the file to open.

sr (block argument)

The created StreamReader instance.

See ::new for a description of the keyword parameters.

Return value

The newly created instance.

Remarks

Starting with version 1.0.0, the block form also returns the newly created instance rather than the block’s return value. This is in line with Ruby’s own GzipReader.open API.

Example

# Normal usage
XZ::StreamReader.open("myfile.txt.xz") do |xz|
  puts xz.read #=> I love Ruby
end

# If you really need the File instance created internally:
file = nil
XZ::StreamReader.open("myfile.txt.xz") do |xz|
  puts xz.read #=> I love Ruby
  file = xz.finish # prevents closing
end
file.close # Now close it manually

# Or just don't use the block form:
xz = XZ::StreamReader.open("myfile.txt.xz")
puts xz.read #=> I love Ruby
file = xz.finish
file.close # Don't forget to close it manually (or use xz.close instead of xz.finish above).



90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/xz/stream_reader.rb', line 90

def self.open(filename, **args)
  file = File.open(filename, "rb")
  reader = new(file, **args)

  if block_given?
    begin
      yield(reader)
    ensure
      # Close both delegate IO and reader.
      reader.close unless reader.finished?
    end
  end

  reader
end

Instance Method Details

#eof? ⇒ Boolean

Returns true if:

  1. The underlying IO has reached EOF, and

  2. liblzma has returned everything it could make out of that.

Returns:

  • (Boolean) 


303
304
305
 
# File 'lib/xz/stream_reader.rb', line 303

def eof?
  @delegate_io.eof? && @readbuf.empty?
end

#inspect ⇒ Object

Human-readable description



308
309
310
 
# File 'lib/xz/stream_reader.rb', line 308

def inspect
  "<#{self.class} pos=#{@pos} bufsize=#{@readbuf.bytesize} finished=#{@finished} closed=#{closed?} io=#{@delegate_io.inspect}>"
end

#read(length = nil, outbuf = String.new) ⇒ Object

Mostly like IO#read. The length parameter refers to the amount of decompressed bytes to read, not the amount of bytes to read from the compressed data. That is, if you request a read of 50 bytes, you will receive a string with a maximum length of 50 bytes, regardless of how many bytes this was in compressed form.

Return values are as per IO#read.



203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
 
# File 'lib/xz/stream_reader.rb', line 203

def read(length = nil, outbuf = String.new)
  return "".force_encoding(Encoding::BINARY) if length == 0 # Shortcut; retval as per IO#read.

  # Note: Querying the underlying IO as early as possible allows to
  # have Ruby's own IO exceptions to bubble up.
  if length
    return nil if eof? # In line with IO#read
    outbuf.force_encoding(Encoding::BINARY) # As per IO#read docs

    # The user's request is in decompressed bytes, so it doesn't matter
    # how much is actually read from the compressed file.
    if @delegate_io.eof?
      data   = ""
      action = XZ::LibLZMA::LZMA_FINISH
    else
      data   = @delegate_io.read(XZ::CHUNK_SIZE)
      action = @delegate_io.eof? ? XZ::LibLZMA::LZMA_FINISH : XZ::LibLZMA::LZMA_RUN
    end

    lzma_code(data, action) { |decompressed| @readbuf << decompressed }

    # If the requested amount has been read, return it.
    # Also return if EOF has been reached. Note that
    # String#slice! will clear the string to an empty one
    # if `length' is greater than the string length.
    # If EOF is not yet reached, try reading and decompresing
    # more data.
    if @readbuf.bytesize >= length || @delegate_io.eof?
      result = @readbuf.slice!(0, length)
      @pos += result.bytesize
      return outbuf.replace(result)
    else
      return read(length, outbuf)
    end
  else
    # Read the entire file and decompress it into memory, returning it.
    while chunk = @delegate_io.read(XZ::CHUNK_SIZE)
      action = @delegate_io.eof? ? XZ::LibLZMA::LZMA_FINISH : XZ::LibLZMA::LZMA_RUN
      lzma_code(chunk, action) { |decompressed| @readbuf << decompressed }
    end

    @pos += @readbuf.bytesize

    # Apply encoding conversion.
    # First, tag the read data with the external encoding.
    @readbuf.force_encoding(@external_encoding)

    # Now, transcode it to the internal encoding if that was requested.
    # Otherwise return it with the external encoding as-is.
    if @internal_encoding
      @readbuf.encode!(@internal_encoding, **@transcode_options)
      outbuf.force_encoding(@internal_encoding)
    else
      outbuf.force_encoding(@external_encoding)
    end

    outbuf.replace(@readbuf)
    @readbuf.clear
    @readbuf.force_encoding(Encoding::BINARY) # Back to binary mode for further reading

    return outbuf
  end
end

#rewind ⇒ Object

Abort the current decompression process and reset everything to the start so that reading from this reader will start over from the beginning of the compressed data.

The delegate IO has to support the #rewind method. Otherwise like IO#rewind.



273
274
275
276
277
278
279
280
281
282
283
 
# File 'lib/xz/stream_reader.rb', line 273

def rewind
  super

  @readbuf.clear
  res = XZ::LibLZMA.lzma_stream_decoder(@lzma_stream.to_ptr,
                                    @memory_limit,
                                    @allflags)
  XZ::LZMAError.raise_if_necessary(res)

  0 # Mimic IO#rewind's return value
end

#ungetbyte(obj) ⇒ Object

Like IO#ungetbyte.



286
287
288
289
290
291
292
 
# File 'lib/xz/stream_reader.rb', line 286

def ungetbyte(obj)
  if obj.respond_to? :chr
    @readbuf.prepend(obj.chr)
  else
    @readbuf.prepend(obj.to_s)
  end
end

#ungetc(str) ⇒ Object

Like IO#ungetc.



295
296
297
 
# File 'lib/xz/stream_reader.rb', line 295

def ungetc(str)
  @readbuf.prepend(str)
end