Tokenizer

DESCRIPTION

Tokenizer is a simple tokenizer able to split easy sentences up into tokens.

Implemented Features

How to Use it

Since the package is not completed use ruby -I bin/tokenize to start Tokenizer. After “God mode activated” you can type in a sentence you want to have tokenized.