Crabbs

A simple web crawler

Description

A simple web crawler for single domains that oupts a site map as json

Installation

gem install crabbs

Usage

crabbs -u http://example.com

Please reference to crabbs --help for further options.

Contributing

1. Fork it ( https://github.com/[my-github-username]/crabbs/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request