Tolq Spieker
Tolq Spieker is a javascript friendly crawler. It uses selenium and capybara to crawl a site.
Installation and usage
$ gem install spieker
$ spieker <url>
You can also use the library from your apps to do whatever you please.
TODO
- Be awesome in parallel
- Use poltergeist instead, does not work well by default
How it works
Spieker makes a request to the URL you provide. It scrapes all links, filters any remote or non resource links, and crawls those on its own terms. It tracks visited links in memory. When using the binary, it very verbosely outputs this to STDOUT.
Changelog
See Changelog
Contribute
- fork
- test
- pull request
Contributors
License
See License