smoke

smoke is a Ruby based DSL that allows you to query web services such as YQL, RSS / Atom and JSON or XML in an elegant manner.

These "services" can then be re-represented, sorted and filtered. Data can be collected from multiple sources, sorted via a common property, chopped up and refried.

Then you can output as a plain ruby object or one of your other favourites (JSON, YAML, XML)

Examples of use

The examples directory has something to get you running straight away
I powered my entire site using smoke, until further documentation exists, this is probably a good place to start.
Read further details in the rdoc documentation or the wiki

Media

Presentation from Webjam11 in Perth
Presentation from Melbourne #roro
Early screencast to get developer / peer feedback

Install

gem install smoke

The concept

The concept comes from using Yahoo Pipes to make web based mash ups: Get a list of tv shows for my torrent client, compile a recipe book or make tools to give me a list of albums that artists in my music library are about to be released.

Using with web services

Smoke has its own special rack based library to expose your sources web services. Checkout rack-smoke for more details

How or what to contribute

Test everything you do
Add a way to output (XML, anyone?)
Examples of queries you'd like to be able to do (email / github message them to me)

API Examples

YQL

# This will use yahoo search to get an array of search results about Ruby
Smoke.yql(:ruby) do
  select  :all
  from    "search.web"
  where   :query, "ruby"

  discard :title, /tuesday/i
end

Smoke.yql(:python) do
  select  :all
  from    "search.web"
  where   :query, "python"
end

Join sources and use them together

Smoke.join(:ruby, :python)

or even

Smoke.join(:python, :ruby) do
  emit do
    sort :title
    rename :shit_name => :title
  end
end

Define a source allowing for variables to be injected later

Source definition:

Smoke.feed :delicious do
  prepare do
    url "http://feeds.delicious.com/v2/rss/#{username}?count=15"
  end
end

Execution:

Smoke.delicious.username("bschwarz").output

TODO (working on, just mental notes)

Items returned from smoke to be Hashie "rich" objects
Output as a web feed (atom, rss)
- How to push through values like "author"?
- Passing modified time headers through to the atom formatter

For wiki pages (docs, later)

Document all sources with their irrespective differential methods
How to use path
YQL Definitions
Tranformations
Insert
Joining
Variable injection
Sort, Reverse
Keep, Discard
Truncate
Manually setting the content type for a url