Updated Proposal (2)

I’ve updated my proposal just a bit. again

Check it out.
Proposal: Add full text-search support to Bricolage.

KinoSearch is a search utility written in Perl.
Currently at version 0.162, it supports full text search.
Support for this utility will be added to Bricolage.

Synopsis:

The aim of this project is to include full text-search support to the content mangement system, Bricolage. Bricolage already has search capabilities. A full text-search will be integrated to the already existent search API.

To implement this four major requirements will have to be fulfilled:

An InvIndex will have to be generated for the all the data placed in each table or for the documents. An indexer function will be written which creates an InvIndex for all the documents. I intend to begin with adding full text search to objects like stories, media and templates.

Next a method will have to be added to the search API which facilitates the conversion of the search query to an Invindex using the same analyzer that was used to create the Invindex for the documents so that the complete text that is to be searched for can be recognized within any table simply by using the Invindex for the search query.

A UI widget already exists in Bricolage. The UI widget will have to be adapted
to the modified search API. Callbacks already exist for all the basic functionalities
of a search UI. A controller function will be used to utilize these functios and add
additional full text search feature. The search will required an updated set of fields
for which the search has to be executed. The name field should now also be able to take
in full text strings. When the results are found, the contexts of the corresponding
results will also need to be displayed

Since Bricolage currently boasts an array of SOAP integrated APIs, the modified
search API, or rather the full-text search API, will naturally be integrated too.
So the API should be further appended to hold functions to code the users’ search
request into XML and send it across to the server and a function that recieves the
result from the server and decodes it before displaying it the user.

The project will support not only postgresql but mysql and so on. The function that generates the InvIndex and will be designed as though it were a plug-in and if someone wanted to use a search other than KinoSearch, like Lucene or even tsearch, could use an alternate indexer function which would generate the necessary index. The search function could also be designed as a plug-in or the parameters passed to the search function could vary according to the indexer function used.

Also there The installer script will have be modified to recognize and use an already installed version of KinoSearch or install KinoSearch if its not found.

Proposed Road Map:

milestone: Start project

<May 28>.Begin Research on implementation of the full text search and the current
state of the search API in the Bricolage System

<June 2> Begin Designing the changes to the table structure and the changes to the
search API

milestone: Submission of initial design and specification.

<June 5> The design of planned additions and changes and their respective changes will
be submitted.

Coding begins

milestone: Function for incrementing Invindex

<June 12> Submission of code that generate the InvIndex for the tables holding data or documents

testing and verification of given code.

milestone: Modification of the Search API

<June 20> Submission of modifications and additions to the existing search API

testing and verification of submitted changes

milestone: Integration to the UI widget

<June 25> The Search UI widget is integrated with the modified API

<June 27> The UI widget is modified to hold the full text search results

testing and verification

milestone: SOAP Integration

<July 7> The modified search API will be integrated with SOAP

testing and verification

milestone: Final Project Submission

<July 17> The Final testing and verification process will come to an end and
the full text search feature will be fully enabled.

################################################################################################

KinoSearch vs Tsearch2 contd.

Report2:

The working principle of tsearch2 is thus:

The document is reduced into a tsvector, which is a data type which holds words in the document and also positional information of each word in the document. As mentioned in my earlier report this data type can hold only up to 256  positions  per  document.  Each  word  in  the  tsvector  is represented as a lexeme. Then a list of indexes according to the lexeme list is made.

The searches to be performed is stored in a data type called tsquery, which stores your search terms as lexemes again. Then we query for the necessary vector from the table with the tsvector column and the necessary documents are returned.

I tried building a very simple search engine from an example cited on the user’s guide to tsearch2.

=# CREATE TABLE docs ( id SERIAL, doc TEXT, vector tsvector )
=# CREATE INDEX docs_index ON docs USING gist(vector);
# CREATE FUNCTION insdoc(text) RETURNS void LANGUAGE sql AS
  'INSERT INTO docs (doc, vector) VALUES ($1, to_tsvector($1));'

=# SELECT insdoc('This is a test document I made up.')
=# SELECT insdoc('For my summer of code project for google')
=# SELECT insdoc('Which is to implement full text search for Bricolage.')
=# SELECT insdoc('I am testing tsearch2 right now')
=# SELECT insdoc('And I'm running out of sentences to put in.')
=# SELECT insdoc('So I guess I'm gonna stop here.')
=# SELECT insdoc('Okay, so maybe one more then... .')
=# CREATE TYPE finddoc_t AS (id INTEGER, headline TEXT, rank REAL)
=# CREATE FUNCTION finddoc(text) RETURNS SETOF finddoc_t LANGUAGE sql AS '
   SELECT id, headline(doc, q), rank(vector, q)
     FROM docs, to_tsquery($1) AS q
     WHERE vector @@ q ORDER BY rank(vector, q) DESC'

This was my test case:
=# SELECT * FROM finddoc('text|search')
 id |                       headline                        | rank
----+-------------------------------------------------------+------
  3 | Which is to implement full <b>text</b> <b>search</b>  | 0.19
  4 | <b>tsearch2</b> right now                             |  0.1

(2 rows)
=# SELECT doc FROM docs WHERE id = 3
                       doc
-------------------------------------------------
 Which is to implement full text search for Bricolage.
(1 row)

Something I noticed was that tsearch returned only those leximes which were exact matches. While converting search cases into leximes for tsquery, this has to be paid attention to.

I ran into some problems when I first tried to install KinoSearch .

So I checked for a list of Kinosearch users:
I started out with www.evo.com. The search was user-friendly and fast.
( I just thought I’ll mention all the steps I followed)

After I installed KinoSearch using cpan I tried out a few examples from this page.
I made no real changes to the example search engine or the test case mentioned on the page.

Check the code I used:

invindex.plx

    #!/usr/bin/perl
    use strict;
    use warnings;

    use File::Spec;
    use KinoSearch::InvIndexer;
    use KinoSearch::Analysis::PolyAnalyzer;

    my $source_dir       = '';
    my $path_to_invindex = '';
    my $base_url         = '/us_constitution';

    opendir( my $source_dh, $source_dir )
        or die "Couldn't opendir '$source_dir': $!";
    my @filenames = grep {/\.html/} readdir $source_dh;
    closedir $source_dh or die "Couldn't closedir '$source_dir': $!";

    ### Analyzer.
    my $analyzer = KinoSearch::Analysis::PolyAnalyzer->new(
        language => 'en',
    );

    ### Create a InvIndexer object.
    my $invindexer = KinoSearch::InvIndexer->new(
        analyzer => $analyzer,
        invindex => $path_to_invindex,
        create   => 1,
    );

    ### fields.
    $invindexer->spec_field( name => 'title' );
    $invindexer->spec_field(
        name       => 'bodytext',
        vectorized => 1,
    );
    $invindexer->spec_field(
        name    => 'url',
        indexed => 0,
    );

    foreach my $filename (@filenames) {
        next if $filename eq 'index.html';
        my $filepath = File::Spec->catfile( $source_dir, $filename );
        open( my $fh, '<', $filepath )
            or die "couldn't open file '$filepath': $!";
        my $content = do { local $/; <$fh> };

        ### new document.
        my $doc = $invindexer->new_doc;

        $content =~ m#<title>(.*?)</title>#s
            or die "couldn't isolate title in '$filepath'";
        my $title = $1;
        $content =~ m#<div id="bodytext">(.*?)</div><!--bodytext-->#s
            or die "couldn't isolate bodytext in '$filepath'";
        my $bodytext = $1;
        $bodytext =~ s/<.*?>/ /gsm;    

        ### value for each field.
        $doc->set_value( url      => "$base_url/$filename" );
        $doc->set_value( title    => $title );
        $doc->set_value( bodytext => $bodytext );

        ### Add the document to the invindex.
        $invindexer->add_doc($doc);

    }

    $invindexer->finish;

SEARCH.CGI

    #!/usr/bin/perl -T
    use strict;
    use warnings;

    use CGI;
    use List::Util qw( max min );
    use POSIX qw( ceil );
    use KinoSearch::Searcher;
    use KinoSearch::Analysis::PolyAnalyzer;
    use KinoSearch::Highlight::Highlighter;

    my $cgi           = CGI->new;
    my $q             = $cgi->param('q');
    my $offset        = $cgi->param('offset');
    my $hits_per_page = 10;
    $q      = '' unless defined $q;
    $offset = 0  unless defined $offset;

    my $path_to_invindex = '';
    my $base_url         = '/us_constitution';

    ### specify Analyzer
    my $analyzer = KinoSearch::Analysis::PolyAnalyzer->new(
        language => 'en',
    );

    ### Searcher object.
    my $searcher = KinoSearch::Searcher->new(
        invindex => $path_to_invindex,
        analyzer => $analyzer,
    );

    ### query
    my $hits = $searcher->search($q);

    my $highlighter = KinoSearch::Highlight::Highlighter->new(
        excerpt_field => 'bodytext' );
    $hits->create_excerpts( highlighter => $highlighter );

    $hits->seek( $offset, $hits_per_page );

    # create result list
    my $report = '';
    while ( my $hit = $hits->fetch_hit_hashref ) {
        my $score = sprintf( "%0.3f", $hit->{score} );
        $report .= qq|
            <p>
                <a href="$hit->{url}"><strong>$hit->{title}</strong></a>
                <em>$score</em>
                <br>
                $hit->{excerpt}
                <br>
                <span class="excerptURL">$hit->{url}</span>
            </p>
            |;
    }

    $q = CGI::escapeHTML($q);

    my $total_hits = $hits->total_hits;
    my $num_hits_info;
    if ( !length $q ) {
        # no query, no display
        $num_hits_info = '';
    }
    elsif ( $total_hits == 0 ) {
     $num_hits_info = qq|<p>No matches for <strong>$q</strong></p>|;
    }
    else {
           my $last_result = min( ( $offset + $hits_per_page ), $total_hits );
        my $first_result = min( ( $offset + 1 ), $last_result );

        $num_hits_info = qq|
            <p>
                Results <strong>$first_result-$last_result</strong>
                of <strong>$total_hits</strong> for <strong>$q</strong>.
            </p>
            <p>
                Results Page:
            |;

        my $current_page = int( $first_result / $hits_per_page ) + 1;
        my $last_page    = ceil( $total_hits / $hits_per_page );
        my $first_page   = max( 1, ( $current_page - 9 ) );
        $last_page = min( $last_page, ( $current_page + 10 ) );

        my $href = $cgi->url( -relative => 1 ) . "?" . $cgi->query_string;
        $href .= ";offset=0" unless $href =~ /offset=/;

        for my $page_num ( $first_page .. $last_page ) {
            if ( $page_num == $current_page ) {
                $num_hits_info .= qq|$page_num \n|;
            }
            else {
                my $new_offset = ( $page_num - 1 ) * $hits_per_page;
                $href =~ s/(?<=offset=)\d+/$new_offset/;
                $num_hits_info .= qq|<a href="$href">$page_num</a>\n|;
            }
        }

        $num_hits_info .= "</p>\n";
    }

    print "Content-type: text/html\n\n";
    print <<END_HTML;
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/html4/loose.dtd">
    <html>
    <head>
        <meta http-equiv="Content-type"
            content="text/html;charset=ISO-8859-1">
        <link rel="stylesheet" type="text/css" href="$base_url/uscon.css">
        <title>KinoSearch: $q</title>
    </head>

    <body>

        <div id="navigation">
            <form id="usconSearch" action="">
                <strong>
                Search the <a href="$base_url/index.html">US Constitution</a>:
                </strong>
                <input type="text" name="q" id="q" value="$q">
                <input type="submit" value="=&gt;">
                <input type="hidden" name="offset" value="0">
            </form>
        </div><!--navigation-->

        <div id="bodytext">

        $report

        $num_hits_info

        <p style="font-size: smaller; color: #666">
            <em>Powered by
                <a href="http://www.rectangular.com/kinosearch/">
                    KinoSearch
                </a>
            </em>
        </p>
        </div><!--bodytext-->

    </body>

    </html>
    END_HTML

Details of this code are available here

Once the testing was over, I compared the both of them.

The first criteria checked was speed; the tsearch engine I built performed much faster performed much faster.
I wrote a script to populate the tsvector tables with the US constitution as the document, which was my test case while using KinoSearch. I checked both the searches for the same phrases for both the searches and KinoSearch returned detailed results faster. Though for elementary phrases tsearch performed really well. And a more sophisticated search engine might do a better job.

Kino Search uses an invindex for the document body and the search cases are checked for using the invindex.

There are a list of additional features of KinoSearch mentioned on that page. I cite them here anyway,

• Incremental indexing (addition/deletion of documents to/from an existing index).
• Full support for 12 Indo-European languages.
• Support for boolean operators AND, OR, and AND NOT; parenthetical groupings, and prepended +plus and -minus
• Algorithmic selection of relevant excerpts and highlighting of search terms within excerpts
• Highly customizable query and indexing APIs

Building the invindex for the document in question was much easier when I used the Kino Engine.
The code too was more understandable and in Perl.

Please check this report and let me know. The question to be decided now is whether or not we intend to support mysql. If not implementing tsearch2 would be much easier as detailed in the original application.
Implementing KinoSearch would mean more changes to the installer script and bringing in unwanted dependencies.

KinoSearch vs TSearch2

Report 1:

KinoSearch vs Tsearch2

The original application submitted to Google was to incorporate tsearch2 to the existing search API of Bricolage. KinoSearch seems to be another option. There has been a discussion on which of these searches would be better suited for the purpose of integrating full text search into Bricolage.
I read up on both KinoSearch and Tsearch2.

KinoSearch is currently at version 0.162
and the file format may undergo changes in the future. This might lead to a crash of the application if these changes are not taken into consideration when an upgrade happens.

Tsearch2 is compatible only with postgresql and only for versions greater than or equal 8.3.

KinoSearch is admittedly faster and has support for incremental indexing.
It can handle a large number of documents. Tsearch has limitations when it comes to handling large documents.

Tsearch uses a datatype called tsvector which represents a document, which holds a set of unique words and their positional information. But it can only hold up to 256 positions in a single document.

On the other hand KinoSearch is an adaptation of a completely Java based search engine (Lucene). I don’t know if that is really a problem, but it would be better to avoid dependencies that are uncalled for.

If implemented tsearch can provide stability. But the only hint of instability on using Kinosearch would be if the file format undergoes a change and the alpha version (which is the current KinoSearch version) is upgraded. Though this is probable, it is not something that cannot be handled; The new file format can be checked and handled when we upgrade to a higher version of Kinosearch.

Both KinoSearch and Tsearch have communities that can provide enough support. Tsearch has complete online updates and support. The online support for Kinosearch also serves our purpose. The documentation for both these project are extensive, though I found the Kinosearch to be a bit better, especially on the understandability.

KinoSearch is after all Perl based, and after an initail encounter with both tsearch and kinoseach, I am of the opinion KinoSearch might lend itself to easier integration. Also Tsearch does not provide support for mysql and depends heavily on postgresql, while Kinosearch has a core based on Java.

Kinosearch can provide more support than tsearch and requires less compromises from the users, like a mandatory upgrade of their postgresql versions. The KinoSearch API is much more tuned to customization and is more user-friendly.

Conclusion: I think we should go with KinoSearch

Full Text Search implementation for Bricolage

This is my Google Summer of Code project.

My original Application to Google.
Proposal: Add full text-search support to Bricolage.

Postgresql versions 8.3 and above have built-in full text search support.
Support for this utility will be added to Bricolage.

Synopsis:

The aim of this project is to include full text-search support to the content mangement system, Bricolage.
Bricolage already has search capabilities. A full text-search will be integrated to the already existent
search API.

To implement this four major requirements will have to be fulfilled:

A tsearch index will have to be generated for the data placed in each table.
An additional field will have to be added holding the tsearch index corresponding
to the data it holds. A row-level trigger function to generate the tsearch index and add it to
each table will have to be written. I intend to begin with adding full text search to objects like
stories, media and templates.

Next a method will have to be added to the search API which facilitates recognition of
the tsearch index column that has been added to each table so that the complete
text that is to be searched for can be recognized within any table simply using
the tsearch index.

A UI widget already exists in Bricolage. The UI widget will have to be adapted
to the modified search API. Callbacks already exist for all the basic functionalities
of a search UI. A controller function will be used to utilize these functios and add
additional full text search feature. The search will required an updated set of fields
for which the search has to be executed. The name field should now also be able to take
in full text strings. When the results are found, the contexts of the corresponding
results will also need to be displayed

Since Bricolage currently boasts an array of SOAP integrated APIs, the modified
search API, or rather the full-text search API, will naturally be integrated too.
So the API should be further appended to hold functions to code the users’ search
request into XML and send it across to the server and a function that recieves the
result from the server and decodes it before displaying it the user.

The project will support only postgresql 8.3 upwards, as the full text search is
natively supported only by postgresql 8.3 upwards. So there will have to be a provision
to disable the full text search when lower versions are used. The installer code will
have to be modified to recognize the RDBMS and the version, so that we can disable the
full text search option for lower versions at install time. The users can modify this
setting if they upgrade. A migration script will be written which can be run to generate
the tsearch index and populate the relevant tables so that the users can enable full
text search once they upgrade. The script will function in a similar manner to the trigger
function

Proposed Road Map:

milestone: Start project

<Day 1>.Begin Research on implementation of the full text search and the current
state of the search API in the Bricolage System

<Day 4> Begin Designing the changes to the table structure and the changes to the
search API

milestone: Submission of initial design and specification.

<Day 7> The design of planned additions and changes and their respective changes will
be submitted.

Coding begins

milestone: Function for incrementing tsearch index

<Day 15> Submission of code that modifies the tables holding data to append a generated
tsearch index for corresponding data.

testing and verification of given code.

milestone: Modification of the Search API

<Day 23> Submission of modifications and additions to the existing search API

testing and verification of submitted changes

milestone: Integration to the UI widget

<Day 28> The Search UI widget is integrated with the modified API

<Day 30> The UI widget is modified to hold the full text search results

testing and verification

milestone: SOAP Integration

<Day 40> The modified search API will be integrated with SOAP

testing and verification

milestone: Disabling FT Search for versions < postgresql 8.3

<Day 45> Provision to disable the feature if a version of postgresql less than 8.3 is used

milestone: Final Project Submission

<Day 50> The Final testing and verification process will come to an end and
the full text search feature will be fully enabled.

About Me:

I, Krishnanunni P.N., am a sixth semester student of Computer Science and Engineering from
Kerala, India. The last two years have seen me as an active promoter and member of the
Free and Open Source Software Community.
I am an avid Perl, Python and C programmer. I am well versed in Lisp,Java and x86 assembly
languages.
Latest Projects :
- A PC suite for mobile phones using the Symbian 60 OS.
the connecting medium being bluetooth.

- The development of an IDE that facilitates easy program development
and debugging for various microcontrollers.

################################################################################################

Howdy world !…….

Perl makes easy Jobs easy, without making the hard jobs impossible

Well writing small hacks in Perl is a breeze. Though to be frank I must admit; I find the larger programs a bit of a hazzle.

I applied for the Google Summer of Code this year.

And my application to implement full text search for Bricolage got accepted.

Hence this blog, which shall initially contain all my Summer of Code work and whatever I find fit to put in (related to Perl, of course )