Tolq Spieker

Tolq Spieker is a javascript friendly crawler. It uses selenium and capybara to crawl a site.

Installation and usage

$ gem install spieker
$ spieker <url>

You can also use the library from your apps to do whatever you please.

TODO

• Be awesome in parallel
• Use poltergeist instead, does not work well by default

How it works

Spieker makes a request to the URL you provide. It scrapes all links, filters any remote or non resource links, and crawls those on its own terms. It tracks visited links in memory. When using the binary, it very verbosely outputs this to STDOUT.

Changelog

See Changelog

Contribute

• fork
• test
• pull request

Contributors

• Timon Vonk

License

See License