AnyStyle

AnyStyle is a fast and smart parser of bibliographic references. Originally inspired by parsCit and FreeCite, AnyStyle uses machine learning algorithms and aims to make it easy to train models with data that's relevant to you.

Using AnyStyle on the command line

$ [sudo] gem install anystyle-cli
$ anystyle --help
$ anystyle help find
$ anystyle help parse

See anystyle-cli for more details.

Using AnyStyle in Ruby

Install the anystyle gem.

$ [sudo] gem install anystyle

Now you can use the static Parser and Finder instances by calling the AnyStyle.parse or AnyStyle.find methods. For example:

require 'anystyle'

pp AnyStyle.parse 'Derrida, J. (1967). L’écriture et la différence (1 éd.). Paris: Éditions du Seuil.'
#-> [{
#  :author=>[{:family=>"Derrida", :given=>"J."}],
#  :date=>["1967"],
#  :title=>["L’écriture et la différence"],
#  :edition=>["1"],
#  :location=>["Paris"],
#  :publisher=>["Éditions du Seuil"],
#  :language=>"fr",
#  :scripts=>["Common", "Latin"],
#  :type=>"book"
#}]

You can also create your own AnyStyle::Parser or AnyStyle::Finder with custom options.

Using AnyStyle on the web

AnyStyle is available at anystyle.io.

The web application is open source and you're welcome to host your own instance!

Improving results for your data

Training

You can train custom Finder and Parser models. To do this, you need to prepare your own data sets for training. You can create your own data from scratch or build on AnyStyle's default sets. The default parser model uses the core data set. And though the finder model sources aren't available in their entirety, due to copyright restrictions, you can find several tagged documents here.

When you have compiled a data set for training, you will be ready to create your own model:

$ anystyle train training-data.xml custom.mod

This will save your new model as custom.mod. To use your model instead of AnyStyle's default, use the -P or --parser-model flag and, respectively, -F or --finder-model to use a custom finder model. For instance, the command below will parse a file bib.txt with the custom model and print the result to STDOUT in JSON format:

$ anystyle -P custom.mod -f json parse bib.txt -

When training your own models, it's good practice to check their quality using a second data set. For example, to check your custom model using AnyStyle's manually curated gold data set:

$ anystyle -P x.mod check ./res/parser/gold.xml
Checking gold.xml.................   1 seq  0.06%   3 tok  0.01%  3s

This command prints sequence and token error rates. Here, sequence errors are the number of references tagged differently by the parser as compared to the curated input; the number of token errors is the total number of words in these references. In the example above, one reference was wrong (out of 1,700 at the time), because a total of three words had a different tag.

When working with training data, it's a good idea to use the Wapiti::Dataset API in Ruby: it supports standard set operators and makes it easy to combine or compare data sets.

Natural Languages used in AnyStyle

The core data set contains the manually marked-up references which comprise AnyStyle's default parser model. If your references include non-English documents, the distribution of natural languages in this corpus is relevant.

Language	n
ENGLISH	965
FRENCH	54
GERMAN	26
ITALIAN	11
Others	9

Not reliably determined	449
(but mainly English)

(Measured using cld and AnyStyle version 1.3.13)

There is a strong prevalence of English-language documents with the conventions used in English-language bibliographies, with some representation of other European languages. The languages used reflect those used in scientific publishing as well as the maintainers' competencies. If you are working with documents in languages other than English, you might consider training the model with some examples in the relevant languages.

AnyStyle works with references written in any Latin script, including most European languages, languages such as Indonesian and Malaysian, as well as romanized Arabic, Chinese and Japanese. It also supports non-Latin alphabets such as Cyrillic, although no examples of these appear in the default training sets. Languages written in syllabaries or complex symbols which don't use white space to separate tokens aren't compatible with AnyStyle's approach: this includes Chinese, Japanese, Arabic, and Indian languages.

Dictionary Adapters

During the statistical analysis of reference strings, AnyStyle relies on a large feature dictionary; by default, AnyStyle creates a persistent Ruby hash in the folder of the anystyle-data Gem. This uses up about 2MB of disk space and keeps the entire dictionary in memory. If you prefer a smaller memory footprint, you can use AnyStyle's GDBM dictionary. GDBM bindings are part of the Ruby standard library and supported on all platforms, though you may need to install GDBM before installing Ruby.

If you don't want to use the persistent Ruby hash nor GBDM, you can store your dictionary in memory or use a Redis. The best way to change the default dictionary adapter is by adjusting AnyStyle's default configuration (when using the static parser instances you must set the default before using the parser):

AnyStyle::Dictionary.defaults[:adapter] = :ruby
#-> Use a persistent Ruby hash;
#-> slower start-up than GDBM but no extra dependency

AnyStyle::Dictionary.defaults[:adapter] = :hash
#-> Use in-memory dictionary; slow start-up but uses no space on disk

require 'anystyle/dictionary/gdbm'
AnyStyle::Dictionary.defaults[:adapter] = :gdbm

To use Redis, install the redis and redis/namespace (optional) Gems and configure AnyStyle to use the Redis adapter:

AnyStyle::Dictionary.defaults[:adapter] = :redis

# Adjust the Redis-specifi configuration
require 'anystyle/dictionary/redis'
AnyStyle::Dictionary::Redis.defaults[:host] = 'localhost'
AnyStyle::Dictionary::Redis.defaults[:port] = 6379

About AnyStyle

Contributing

The AnyStyle source code is hosted on GitHub. You can check out a copy of the latest code using Git:

$ git clone https://github.com/inukshuk/anystyle.git

If you've found a bug or have a question, please report the issue or, for extra credit, clone the AnyStyle repository, write a failing example, fix the bug and submit a pull request.

Credits

AnyStyle is a volunteer effort and you're encourage to join! Over the years the main contributors have been:

License

AnyStyle is distributed under a BSD-style license. See LICENSE for details.