Class: ETL::Processor::FtpDownloaderProcessor

Inherits:
Processor show all
Defined in:
lib/etl/processor/ftp_downloader_processor.rb

Overview

Custom processor to download files via FTP

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(control, configuration) ⇒ FtpDownloaderProcessor

configuration options include:

  • host - hostname or IP address of FTP server (required)

  • port - port number for FTP server (default: 21)

  • remote_dir - remote path on FTP server (default: /)

  • files - list of files to download from FTP server (default: [])

  • username - username for FTP server authentication (default: anonymous)

  • password - password for FTP server authentication (default: nil)

  • local_dir - local output directory to save downloaded files (default: ”)

As an example you might write something like the following in your control process file:

pre_process :ftp_downloader, {
  :host => 'ftp.sec.gov',
  :path => 'edgar/Feed/2007/QTR2',
  :files => ['20070402.nc.tar.gz', '20070403.nc.tar.gz', '20070404.nc.tar.gz', 
             '20070405.nc.tar.gz', '20070406.nc.tar.gz'],
  :local_dir => '/data/sec/2007/04',
}

The above example will anonymously download via FTP the first week’s worth of SEC filing feed data from the second quarter of 2007 and download the files to the local directory /data/sec/2007/04.



36
37
38
39
40
41
42
43
44
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 36

def initialize(control, configuration)
  @host = configuration[:host]
  @port = configuration[:port] || 21
  @remote_dir = configuration[:remote_dir] || '/'
  @files = configuration[:files] || []
  @username = configuration[:username] || 'anonymous'
  @password = configuration[:password]
  @local_dir = configuration[:local_dir] || ''
end

Instance Attribute Details

#files ⇒ Object (readonly)

Returns the value of attribute files.



13
14
15
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 13

def files
  @files
end

#host ⇒ Object (readonly)

Returns the value of attribute host.



10
11
12
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 10

def host
  @host
end

#local_dir ⇒ Object (readonly)

Returns the value of attribute local_dir.



15
16
17
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 15

def local_dir
  @local_dir
end

#port ⇒ Object (readonly)

Returns the value of attribute port.



11
12
13
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 11

def port
  @port
end

#remote_dir ⇒ Object (readonly)

Returns the value of attribute remote_dir.



12
13
14
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 12

def remote_dir
  @remote_dir
end

#username ⇒ Object (readonly)

Returns the value of attribute username.



14
15
16
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 14

def username
  @username
end

Instance Method Details

#process ⇒ Object 



46
47
48
49
50
51
52
53
54
 
# File 'lib/etl/processor/ftp_downloader_processor.rb', line 46

def process
  Net::FTP.open(@host) do |conn|
    conn.connect(@host, @port)
    conn.(@username, @password)
    @files.each do |f|
      conn.getbinaryfile(remote_file(f), local_file(f))
    end
  end
end