Class: TorchData::DataPipes::Iter::Util::CSVParser

Inherits:
IterDataPipe
  • Object
show all
Defined in:
lib/torchdata/data_pipes/iter/util/csv_parser.rb

Instance Method Summary collapse

Constructor Details

#initialize(source_datapipe, delimiter: ",") ⇒ CSVParser

Returns a new instance of CSVParser.



8
9
10
11
12
 
# File 'lib/torchdata/data_pipes/iter/util/csv_parser.rb', line 8

def initialize(source_datapipe, delimiter: ",")
  @source_datapipe = source_datapipe
  @helper = PlainTextReaderHelper.new
  @fmtparams = {col_sep: delimiter}
end

Instance Method Details

#each(&block) ⇒ Object 



14
15
16
17
18
19
20
21
22
 
# File 'lib/torchdata/data_pipes/iter/util/csv_parser.rb', line 14

def each(&block)
  @source_datapipe.each do |path, file|
    stream = @helper.skip_lines(file)
    stream = @helper.decode(stream)
    stream = CSV.parse(stream, **@fmtparams)
    stream = @helper.as_tuple(stream)
    @helper.return_path(stream, path: path).each(&block)
  end
end