Class: Linguist::Classifier
- Inherits:
-
Object
- Object
- Linguist::Classifier
- Defined in:
- lib/linguist/classifier.rb
Overview
Language bayesian classifier.
Class Method Summary collapse
-
.call(blob, possible_languages) ⇒ Object
Public: Use the classifier to detect language of the blob.
-
.classify(db, tokens, languages = nil) ⇒ Object
Public: Guess language of data.
-
.train!(db, language, data) ⇒ Object
Public: Train classifier that data is a certain language.
Instance Method Summary collapse
-
#classify(tokens, languages) ⇒ Object
Internal: Guess language of data.
-
#initialize(db = {}) ⇒ Classifier
constructor
Internal: Initialize a Classifier.
-
#language_probability(language) ⇒ Object
Internal: Probably of a language occurring - P©.
-
#token_probability(token, language) ⇒ Object
Internal: Probably of token in language occurring - P(F | C).
-
#tokens_probability(tokens, language) ⇒ Object
Internal: Probably of set of tokens in a language occurring - P(D | C).
Constructor Details
#initialize(db = {}) ⇒ Classifier
Internal: Initialize a Classifier.
82 83 84 85 86 87 88 |
# File 'lib/linguist/classifier.rb', line 82 def initialize(db = {}) @tokens_total = db['tokens_total'] @languages_total = db['languages_total'] @tokens = db['tokens'] @language_tokens = db['language_tokens'] @languages = db['languages'] end |
Class Method Details
.call(blob, possible_languages) ⇒ Object
Public: Use the classifier to detect language of the blob.
blob - An object that quacks like a blob. possible_languages - Array of Language objects
Examples
Classifier.call(FileBlob.new("path/to/file"), [
Language["Ruby"], Language["Python"]
])
Returns an Array of Language objects, most probable first.
18 19 20 21 22 23 |
# File 'lib/linguist/classifier.rb', line 18 def self.call(blob, possible_languages) language_names = possible_languages.map(&:name) classify(Samples.cache, blob.data, language_names).map do |name, _| Language[name] # Return the actual Language objects end end |
.classify(db, tokens, languages = nil) ⇒ Object
Public: Guess language of data.
db - Hash of classifier tokens database. data - Array of tokens or String data to analyze. languages - Array of language name Strings to restrict to.
Examples
Classifier.classify(db, "def hello; end")
# => [ 'Ruby', 0.90], ['Python', 0.2], ... ]
Returns sorted Array of result pairs. Each pair contains the String language name and a Float score.
76 77 78 79 |
# File 'lib/linguist/classifier.rb', line 76 def self.classify(db, tokens, languages = nil) languages ||= db['languages'].keys new(db).classify(tokens, languages) end |
.train!(db, language, data) ⇒ Object
Public: Train classifier that data is a certain language.
db - Hash classifier database object language - String language of data data - String contents of file
Examples
Classifier.train(db, 'Ruby', "def hello; end")
Returns nothing.
Set LINGUIST_DEBUG=1 or =2 to see probabilities per-token or per-language. See also #dump_all_tokens, below.
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
# File 'lib/linguist/classifier.rb', line 39 def self.train!(db, language, data) tokens = Tokenizer.tokenize(data) db['tokens_total'] ||= 0 db['languages_total'] ||= 0 db['tokens'] ||= {} db['language_tokens'] ||= {} db['languages'] ||= {} tokens.each do |token| db['tokens'][language] ||= {} db['tokens'][language][token] ||= 0 db['tokens'][language][token] += 1 db['language_tokens'][language] ||= 0 db['language_tokens'][language] += 1 db['tokens_total'] += 1 end db['languages'][language] ||= 0 db['languages'][language] += 1 db['languages_total'] += 1 nil end |
Instance Method Details
#classify(tokens, languages) ⇒ Object
Internal: Guess language of data
data - Array of tokens or String data to analyze. languages - Array of language name Strings to restrict to.
Returns sorted Array of result pairs. Each pair contains the String language name and a Float score.
97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
# File 'lib/linguist/classifier.rb', line 97 def classify(tokens, languages) return [] if tokens.nil? tokens = Tokenizer.tokenize(tokens) if tokens.is_a?(String) scores = {} debug_dump_all_tokens(tokens, languages) if verbosity >= 2 languages.each do |language| scores[language] = tokens_probability(tokens, language) + language_probability(language) debug_dump_probabilities(tokens, language, scores[language]) if verbosity >= 1 end scores.sort { |a, b| b[1] <=> a[1] }.map { |score| [score[0], score[1]] } end |
#language_probability(language) ⇒ Object
Internal: Probably of a language occurring - P©
language - Language to check.
Returns Float between 0.0 and 1.0.
143 144 145 |
# File 'lib/linguist/classifier.rb', line 143 def language_probability(language) Math.log(@languages[language].to_f / @languages_total.to_f) end |
#token_probability(token, language) ⇒ Object
Internal: Probably of token in language occurring - P(F | C)
token - String token. language - Language to check.
Returns Float between 0.0 and 1.0.
130 131 132 133 134 135 136 |
# File 'lib/linguist/classifier.rb', line 130 def token_probability(token, language) if @tokens[language][token].to_f == 0.0 1 / @tokens_total.to_f else @tokens[language][token].to_f / @language_tokens[language].to_f end end |
#tokens_probability(tokens, language) ⇒ Object
Internal: Probably of set of tokens in a language occurring - P(D | C)
tokens - Array of String tokens. language - Language to check.
Returns Float between 0.0 and 1.0.
118 119 120 121 122 |
# File 'lib/linguist/classifier.rb', line 118 def tokens_probability(tokens, language) tokens.inject(0.0) do |sum, token| sum += Math.log(token_probability(token, language)) end end |