simple text search in ruby using ferret

ferret is a lightweight text search engine for ruby, a bit like lucene but with less (ie no) java.

i've been looking at it today as part of my named entity extraction prototype which needs to be able to fuzzily match one short string against a list of other short strings.

let's go through an example, it's the only way my brain works sorry. making a ferret index is simple; we'll just make a memory based index for this demo.

1 2	require 'ferret' index = Ferret::Index::Index.new()

next we'll add a handful of places in africa and europe to our index. each document we add is simply a hash with whatever fields we want to be able to search or return

african_places = ['Ain Sefra','Algiers','South Algiers','Batna','Batni']
african_places.each do |place|
  index << { :continent => 'africa', :name => place }
end
european_places = ['Paris','London','Batna']
european_places.each do |place|
  index << { :continent => 'europe', :name => place }
end

the simplest querying just searches across all fields; in our example this is both continent and name. search hits returning the id of the document found and a relevancy score for ranking. the full contents of a document can be looked up based on their id (and are lazily loaded unless an explicit load is given).

index.search("europe").hits.each { |hit| puts hit.inspect }
#
#
#
puts index[7].load.inspect
{:continent=>"europe", :name=>"London"}

query control will be very similiar to those that know lucene.

as seen above the simplest query allows a match against any term in any field. a particular field can be targetted though using the query form FIELD:VALUE

index.search('name:algiers').hits.each do |hit|
  puts "score=#{hit.score} doc=#{@index[hit.doc].load.inspect}"
end
score=2.0986123085022 doc={:continent=>"africa", :name=>"Algiers"}
score=1.31163263320923 doc={:continent=>"africa", :name=>"South Algiers

wildcarding is done with a asterix.

index.search('name:ba*').hits.each do |hit|
  puts "score=#{hit.score} doc=#{@index[hit.doc].load.inspect}"
end
score=1.81093037128448 doc={:continent=>"africa", :name=>"Batna"}
score=1.81093037128448 doc={:continent=>"africa", :name=>"Batni"}
score=1.81093037128448 doc={:continent=>"europe", :name=>"Batni"}

fuzzy search is denoted by tilde. an optional fuzziness factor can be supplied from 0 (very fuzzy match) to 1 (exact match only). a reasonable default is assumed if a factor is not given.

index.search('name:bitna~0.4').hits.each do |hit|
  puts "score=#{hit.score} doc=#{@index[hit.doc].load.inspect}"
end
score=1.44874429702759 doc={:continent=>"africa", :name=>"Batna"}
score=1.08655822277069 doc={:continent=>"africa", :name=>"Batni"}
score=1.08655822277069 doc={:continent=>"europe", :name=>"Batni"}

and, not surprisingly, a full set of boolean logic operators are supported.

index.search('continent:africa AND name:bitna~').hits.each do |hit|
  puts "score=#{hit.score} doc=#{@index[hit.doc].load.inspect}"
end
score=1.90322256088257 doc={:continent=>"africa", :name=>"Batna"}
score=1.6052508354187 doc={:continent=>"africa", :name=>"Batni"}

though i've no idea how it scales to a larger dataset it's doing the job pretty well for me with a modest index of approx 250,000 small text items.

loads more api doco is provided on the ferret site.

shingling and the jaccard index

on the weekend i did another experiment using shingling and the jaccard index to try to determine if two sets of data were “duplicates”

it works quite well and includes a ruby and c++ version with low level bit operations.

project page is www.matpalm.com/resemblance

code at github.com/matpalm/resemblance

brain of mat kelcey

simple text search in ruby using ferret

shingling and the jaccard index