Class: UHFerret::Ferret

Inherits:
Object
  • Object
show all
Defined in:
lib/uhferret.rb

Overview

UHFerret::Ferret holds a reference to a list of documents, and provides methods to manage this list of documents, compute and
retrieve similarities between documents.

Instance Method Summary collapse

Constructor Details

#initialize(&block) ⇒ Ferret

Constructs an instance of Ferret.

block

optional block is used to add files etc during construction.



38
39
40
41
42
 
# File 'lib/uhferret.rb', line 38

def initialize &block
  @ferret = Uhferret_lib::DocumentList.new
  self.instance_eval(&block) if block_given?
  @ferret_run = false
end

Instance Method Details

#[](index) ⇒ Object

Return document in document list at given index position.

Raises an IndexError if index is not valid.



105
106
107
108
109
 
# File 'lib/uhferret.rb', line 105

def [](index)
  check_index index

  @ferret.getDocument index
end

#add(filename, type = TextDocument, id = 0) ⇒ Object

Add given filename to list of documents. The type of document can be given as:

  • UHFerret::TextDocument, for natural language documents
  • UHFerret::CodeDocument, for c-style computer programs Option third argument specifies the group_id for this document. The group_id can be used to suppress comparisons in some kinds of output.
  • If a pdf or word-processed document is added, it must first be converted to text. Ferret tries to do this, attaching .txt to the end of the filename.


54
55
56
57
58
59
60
61
62
 
# File 'lib/uhferret.rb', line 54

def add(filename, type = TextDocument, id = 0)
  if Utils.is_pdf_document?(filename)
    filename = Utils.convert_pdf_document filename
  elsif Utils.is_wp_document?(filename)
    filename = Utils.convert_wp_document filename
  end
  @ferret.AddDocument(filename, type, (id.zero? ? @ferret.GetNewGroupId : id))
  @ferret_run = false
end

#add_list_from_file(filename, type = TextDocument) ⇒ Object

Add list of files specified in given filename The type of documents can be given as:

  • UHFerret::TextDocument, for natural language documents
  • UHFerret::CodeDocument, for c-style computer programs


68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/uhferret.rb', line 68

def add_list_from_file(filename, type = TextDocument)
  within_group = false
  current_id = 0

  IO.foreach(filename) do |line|
    line.strip!
    if line.upcase == "START GROUP"
      within_group = true
      current_id = @ferret.GetNewGroupId
    elsif line.upcase == "END GROUP"
      within_group = false
    elsif File.readable? line
      add(line, type, (within_group ? current_id : 0))
    end
  end

  @ferret_run = false
end

#containment(doc_1, doc_2) ⇒ Object

Return the containment of doc_1 in doc_2.

Raises an ArgumentError if ferret has not been 'run' before, and an IndexError if the document indices are not valid.



159
160
161
162
163
164
165
 
# File 'lib/uhferret.rb', line 159

def containment(doc_1, doc_2)
  check_ferret_has_run :containment
  check_index doc_1
  check_index doc_2

  @ferret.ComputeContainment(doc_1, doc_2)
end

#distinct_trigrams_count ⇒ Object

Return the total number of distinct trigrams in set of documents.

Raises an ArgumentError if ferret has not been 'run' before calling.



197
198
199
200
201
 
# File 'lib/uhferret.rb', line 197

def distinct_trigrams_count
  check_ferret_has_run :distinct_trigrams_count

  @ferret.GetTotalTrigramCount
end

#each ⇒ Object

Apply provided block to each document in the document list.



112
113
114
115
116
 
# File 'lib/uhferret.rb', line 112

def each
  @ferret.Size.times do |i|
    yield @ferret.getDocument(i)
  end
end

#each_pair ⇒ Object

Apply provided block to each pair of compared document indices, in descending order of resemblance.

Raises an ArgumentError if ferret has not been 'run' before.



132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/uhferret.rb', line 132

def each_pair
  check_ferret_has_run :each_pair

  if @sorted_pairs == []
    # extract all valid document pairs
    @ferret.Size.times do |i|
      (i+1).upto(@ferret.Size-1) do |j|
        @sorted_pairs << [i, j]
      end
    end
    # sort into descending order of resemblance
    @sorted_pairs.sort! do |pair_a, pair_b|
      @ferret.ComputeResemblance(pair_b[0], pair_b[1]) <=>
      @ferret.ComputeResemblance(pair_a[0], pair_a[1])
    end
  end

  # apply block to each pair in sorted order
  @sorted_pairs.each do |pair|
    yield(pair[0], pair[1])
  end
end

#num_pairs ⇒ Object

Return the number of pairs of documents compared.



124
125
126
 
# File 'lib/uhferret.rb', line 124

def num_pairs
  @ferret.NumberOfPairs
end

#output_all_comparisons ⇒ Object

outputs a table of all comparisons, suitable for loading into a spreadsheet



317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
 
# File 'lib/uhferret.rb', line 317

def output_all_comparisons
  # -- output headings
  size.times do |i|
    print ", #{self[i].filename}"
  end
  puts
  # -- output comparisons
  size.times do |i|
    print self[i].filename
    size.times do |j|
      print ", #{resemblance(i, j)}"
    end
    puts
  end
end

#output_html_similarity_table ⇒ Object

outputs similarity table as a html page, sorted in order of similarity



258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
 
# File 'lib/uhferret.rb', line 258

def output_html_similarity_table
  puts <<BODY
<html><body>
<h1>Ferret: Table of Comparisons</h1>
<p>Return to <a href="/ferret/home">Ferret home page</a>.</p>
<table border=1><tbody><tr><th>Index</th><th>Document 1</th><th>Document 2</th><th>Similarity</th><th>View</th></tr>
BODY
  idx = 0
  each_pair do |i, j|
    unless self[i].group_id == self[j].group_id
      idx += 1
      break if idx > MAX_TABLE_SIZE

      puts <<ROW
    <tr>
    <td> #{idx} </td>
    <td> #{format_file(self[i].pathname)} </td>
    <td> #{format_file(self[j].pathname)} </td>
    <td> #{format("%0.3f", resemblance(i, j))} </td>
    <td><a href="/ferret/report?upload=#{Dir.pwd}&file1=#{self[i].pathname}&file2=#{self[j].pathname}" target="_blank"\>View</a></td>
    </tr>
ROW
    end
  end
  puts "</tbody></table></p>"

  puts <<TAIL
<hr>
<p>Return to <a href="/ferret/home">Ferret home page.</a>
<hr><font size=-1>Generated by Ferret, Copyright 2012 University of Hertfordshire</font>
</body></html>
TAIL
end

#output_similarity_table(full_path = false) ⇒ Object

displays each pair of documents, sorted in order of similarity



239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/uhferret.rb', line 239

def output_similarity_table(full_path = false)
  puts "Number of documents: #{size}"
  puts "Number of distinct trigrams: #{distinct_trigrams_count}"
  each_pair do |i, j|
    unless self[i].group_id == self[j].group_id
      if full_path
        puts "#{self[i].pathname} ; #{self[j].pathname} ; \
      #{trigram_matches(i, j)} ; #{trigram_count(i)} ; #{trigram_count(j)} ; \
      #{resemblance(i, j)}"
      else
        puts "#{self[i].filename} ; #{self[j].filename} ; \
      #{trigram_matches(i, j)} ; #{trigram_count(i)} ; #{trigram_count(j)} ; \
      #{resemblance(i, j)}"
        end
    end
  end
end

#output_trigram_list ⇒ Object

outputs a list of trigrams with the document indices in which they appear, indices are space separated



294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
 
# File 'lib/uhferret.rb', line 294

def output_trigram_list
  begin
    tuples = @ferret.GetTupleSet
    tuples.Begin
    while tuples.HasMore
      print @ferret.MakeTrigramString(tuples.GetToken(0),
                                      tuples.GetToken(1),
                                      tuples.GetToken(2))
      print "  FILES:[ "
      doc_indices = tuples.GetDocumentsForCurrentTuple
      doc_indices.size.times do |i|
        print "#{doc_indices[i]} "
      end
      print " ]"
      puts
      tuples.GetNext
    end
  rescue Exception => ex
    puts "Error in writing trigram list: #{ex}"
  end
end

#resemblance(doc_1, doc_2) ⇒ Object

Return the resemblance of doc_1 and doc_2.

Raises an ArgumentError if ferret has not been 'run' before, and an IndexError if the document indices are not valid.



171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/uhferret.rb', line 171

def resemblance(doc_1, doc_2)
  check_ferret_has_run :resemblance
  check_index doc_1
  check_index doc_2

  if doc_1 == doc_2 
    return 1.0
  else
    @ferret.ComputeResemblance([doc_1, doc_2].min, [doc_1, doc_2].max)
  end
end

#run ⇒ Object

Run ferret on the current document list. You must run ferret before retrieving measures of containment or resemblance.

Raises an ArgumentError if there are not at least two documents in the document list.



92
93
94
95
96
97
98
99
100
 
# File 'lib/uhferret.rb', line 92

def run
  if @ferret.Size >= 2
    @ferret.RunFerret
    @ferret_run = true
    @sorted_pairs = []
  else
    raise ArgumentError.new("UHFerret needs at least two documents to run")
  end
end

#size ⇒ Object

Return the number of documents in the document list.



119
120
121
 
# File 'lib/uhferret.rb', line 119

def size
  @ferret.Size
end

#trigram_count(index) ⇒ Object

Return the number of trigrams in given document index.

Raises an ArgumentError if ferret has not been 'run' before, and an IndexError if the document index is not valid.



187
188
189
190
191
192
 
# File 'lib/uhferret.rb', line 187

def trigram_count index
  check_ferret_has_run :trigram_count
  check_index index

  @ferret.CountTrigrams index
end

#trigram_matches(doc_1, doc_2) ⇒ Object

Return the number of matching trigrams in given two document indices.

Raises an ArgumentError if ferret has not been 'run' before, and an IndexError if the document indices are not valid.



207
208
209
210
211
212
213
 
# File 'lib/uhferret.rb', line 207

def trigram_matches(doc_1, doc_2)
  check_ferret_has_run :trigram_matches
  check_index doc_1
  check_index doc_2

  @ferret.CountMatches(doc_1, doc_2)
end

#xml_output(output_file, doc_1, doc_2) ⇒ Object

Write an XML report of the given two document indices into given filename.

Raises an ArgumentError if ferret has not been 'run' before, and an IndexError if the document indices are not valid.



219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
 
# File 'lib/uhferret.rb', line 219

def xml_output(output_file, doc_1, doc_2)
  check_ferret_has_run :xml_output
  check_index doc_1
  check_index doc_2

  File.open(output_file, "w") do |file|
    file.puts "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>"
    file.puts "<?xml-stylesheet type=\"text/xsl\" href=\"uhferret.xsl\" ?>"
    file.puts "<uhferret>"

    file.puts "<common-trigrams>#{trigram_matches(doc_1, doc_2)}</common-trigrams>"
    file.puts "<similarity>#{resemblance(doc_1, doc_2)}</similarity>"
    write_xml_document(file, doc_1, doc_2)
    write_xml_document(file, doc_2, doc_1)

    file.puts "</uhferret>"
  end
end