rest-ftp-daemon
A pretty simple but configurable and efficient FTP-client daemon, driven through a RESTful API. Create transfer jobs by POSTing a simple JSON structure, be notified of their completion, watch their status on a dedicated dashboard.
Features
System and process features
- environment-aware configuration in a YAML file
- daemon process is tagged with its name and environment in process lists
- global dashboard directly served within the daemon HTTP interface
File management ans transferts
- allow authentication in FTP target in a standard URI-format
- static path pointers in configuration to abstract local mounts or remote FTPs (endpoint tokens)
- local source path and local/remote target path can use patterns to match multiple files (
/dir/file*.jpg
) - several file transfer protocols supported: FTPs, FTPes, sFTP
Job management
- highly parrallel job processing using dedicated worker threads with their own context
- jobs are taken into account as soon as they are submitted
- each job carry its own attributes: build subdirectories (mkdir), overwrite target file, priority weight
- dynamic evaluation of priorities, honoring any change on context until the job is picked
- automatically clean-up jobs after a configurable amount of time (failed, finished)
Realtime status reporting
- realtime transfer status reporting, with progress and errors
- periodic update notifications sent along with transfer status and progress to an arbitrary URL (JSON resource POSTed)
Status
Though it may need more robust tests, this gem has been used successfully in production for a while without glitches at France Télévisions.
Expected features in a short-time range :
- Provide swagger-style API documentation
- Authenticate API clients
- Allow more transfer protocols (sFTP, HTTP POST etc)
- Expose JSON status of workers on
GET /jobs/
for automated monitoring
Installation
With Ruby (version 2.1 or higher) and rubygems properly installed, you only need to issue :
gem install rest-ftp-daemon
If that is not the case yet, see section Debian install preparation.
Usage
You must provide a configuration file for the daemon to start, either
explicitly using option --config
or implicitly at /etc/rest-ftp-daemon.yml
.
(A sample file is provided see --help
for more info about it.)
You can then simply start the daemon on the standard port, or on a specific port using -p
$ rest-ftp-daemon -p 3000 start
Check that the daemon is running and exposes a JSON status structure on http://localhost:3000/status
.
The dashboard will provide a global view on http://localhost:3000/
If the daemon appears to exit quickly when launched, it may be caused by logfiles that can't be written (check files permissions or owner).
Launcher options :
Param | Short | Default | Description |
---|---|---|---|
-p | --port | (automatic) | Port to listen for API requests |
-e | production | Environment name | |
--dev | Equivalent to -e development | ||
-w | --workers | 1 | Number of workers spawned at launch |
-d | --daemonize | false | Wether to send the daemon to background |
-f | --foreground | false | Wether to keep the daemon running in the shell |
-P | --pid | (automatic) | Path of the file containing the PID |
-u | --user | (none) | User to run the daemon as |
-g | --group | (none) | Group of the user to run the daemon as |
-h | --help | Show info about the current version and available options | |
-v | --version | Show the current version |
Examples
Start a job to transfer a file named "file.iso" to a local FTP server
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
'{"source":"~/file.iso","target":"ftp://anonymous@localhost/incoming/dest2.iso"}' "http://localhost:3000/jobs"
Start a job using endpoint tokens
First define nas
ans ftp1
in the configuration file :
defaults: &defaults
development:
<<: *defaults
endpoints:
nas: "~/"
ftp1: "ftp://anonymous@localhost/incoming/"
Those tokens will be expanded when the job is run:
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
'{"source":"~/file.dmg","priority":"3","target":"ftp://anonymous@localhost/incoming/dest4.dmg","notify":"http://requestb.in/1321axg1"}' "http://localhost:3000/jobs"
Get info about a job with ID="q89j.1"
Both parameters q89j.1
and 1
will be accepted as ID in the API. Requests below are equivalent:
GET http://localhost:3000/jobs/q89j.1
GET http://localhost:3000/jobs/1
API Documentation
API documentation is maintained on Apiary
Configuration
Most of the configuration options live in a YAML configuration file, containing two main sections:
defaults
section should be left as-is and will be used is no other environment-specific value is provided.production
section can receive personalized settings according to your environment-specific setup and paths.
Configuration priority is defined as follows (from most important to last resort):
- command-line parameters
- config file defaults section
- config file environment section
- application internal defaults
As a starting point, rest-ftp-daemon.yml.sample
is an example config file that can be copied into the expected location /etc/rest-ftp-daemon.yml
.
Default administrator credentials are admin/admin
. Please change the password in this configuration file before starting any kind of production.
Logging
The application will log to paths specified in the configuration file, if any.
Separate logging paths can be provided for the Thin webserver, API related messages, and workers related messages.
Providing empty values as paths, will simply activate logging to STDOUT
.
Job cleanup
Job queue can be set to automatically cleanup after a certain delay. Entries are removed from the queue when they have been idle (updated_at) for more than X seconds, and in any of the following statuses:
- failed (conchita.clean_failed)
- finished (conchita.clean_finished)
- queued, (conchita.clean_queued)
Cleanup is done on a regular basis, every (conchita.timer) seconds.
TODO for this document
- Update Apiary documentation
- Update Apiary documentation
- Update Apiary documentation
- Update Apiary documentation !
- Document /status
- Document /routes
- Document mkdir and overwrite options
- Document counters
Debian install preparation
This project is available as a rubygem, requires Ruby 2.1 and rubygems installed.
You may use rbenv
and ruby-build
to get the right Ruby version. If this is your case, ensure that ruby-build definitions are up-to-date and include ruby-2.1.0
# apt-get install ruby-build rbenv
# ruby-build --definitions | grep '2.1'
Otherwise, you way have to update ruby-build to include Ruby 2.1.0 definitions. On Debian, 2.1.0 is not included in Wheezy and appears in Jessie's version of the package.
Use a dedicated user for the daemon, switch to this user and enable rbenv
# adduser --disabled-password --gecos "" rftpd
# su rftpd -l
# echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
# echo 'eval "$(rbenv init -)"' >> ~/.bashrc
Install the right ruby version and activate it
# rbenv install 2.1.0
# rbenv local 2.1.0
# rbenv rehash
Update RubyGems and install the gem from rubygems.org
# gem update --system
# gem install rest-ftp-daemon --no-ri --no-rdoc
# rbenv rehash
# rest-ftp-daemon start
Known bugs
As this project is based on SettingsLogic, which in turns uses Syck YAML parser, configuration merge from "defaults" section and environment-specific section is broken. A sub-tree defined for a specific environment, will overwrite the corresponding subtree from "defaults".
If you get
fatal error: 'openssl/ssl.h' file not found when installing
eventmachine``` on OSX El Capitan, you can try with:gem install eventmachine -v '1.0.8' -- --with-cppflags=-I/usr/local/opt/openssl/include bundle install
Contributing
Contributions are more than welcome, be it for documentation, features, tests, refactoring, you name it. If you are unsure of where to start, the Code Climate report will provide you with improvement directions. And of course, if in doubt, do not hesitate to open an issue. (Please note that this project has adopted a code of conduct.)
If you want your contribution to adopted in the smoothest and fastest way, don't forget to:
- provide sufficient documentation in you commit and pull request
- add proper testing (we know full grown solid test coverage is still lacking and need to up the game)
- use the RuboCop guidelines provided (there are all sorts of editor integration plugins available)
So,
- Fork the project
- Create your feature branch (
git checkout -b my-new-feature
) Code
- add proper tests if adding a feature
- run the tests using
rake
- check for RuboCop style guide violations
Commit your changes
Push to the branch (
git push origin my-new-feature
)Create new Pull Request
About
Thanks to https://github.com/berkshelf/berkshelf-api for parts and ideas used in this project
This project has been initiated and originally written by Bruno MEDICI Consultant (http://bmconseil.com/)