Search Engine

At DrupalCon there was a Search BoF with about 20 people present. We primarily discussed integration with the Lucene/Solr search engine, but then concluded that creating a Search API that allowed for other search engines to plug into it would likely be the best approach. This type of approach allows small site owners to keep using integrated drupal search and larger site owners the option to integrate a more robust search engine with more features.

Another important part of what happened was David Lesiuer showed all of us his own custom drupal driven search engine module called Faceted Search (http://drupal.org/project/faceted_search). He has written a module which gives a views-esque type admin interface to creating any number of different search pages that site owners can create and allow their users to use. After seeing what an incredible job he has done with his search interface many at the conference agreed that it going down the path of creating a pluggable API with more features is a must have for Drupal.

The initial release of the fuzzy search engine model is finally here. I have implemented it here on my blog for anyone that would like to give it a quick try. To download the module visit the official release page. I'd really like to get some test users so that I can get some feedback on the results and performance.

Thus far, I have been able to test it on my blog which has a limited amount of content. However, preliminary results are showing well on speed performance over search.module. I think this mostly has to do with not using temporary tables and not using a call to node_load on each of the results, I chose to join the teaser and title from {node_revisions} and {node} respectively.

Over the past week and a half or so I have made much progress on my project. The following are main accomplishments:

Completed Items

Scoring Hooks

• Scoring hooks to allow other module developers the opportunity to insert a score multiplier at the time of indexing. Scoring Hook API.
• Scoring hooks modify the score of a node during indexing, and affect the score on each word being indexed per node. This is the most effective way to do such an operation because it requires no extra time during the search. It is also effective with this new search module because of the ability to tag nodes for re-indexing, whereas search.module only re-indexes if a node's updated timestamp has been changed.
• Site administrators have control over how much of an affect each score modifier has through the control panel (Screen Shot below).

I've posted up a live version of my current development on a fuzzy search engine. You can now search through the content on my blog using my current algorithm. This is only for demonstration purposes and trial of the algorithm as I make progress on the module. I encourage you all to try a few searches based on the content of my website, including some searches with misspellings.

I'm contemplating how to implement a scoring factor hook into my new fuzzy search engine module. I believe what may work out to be the best way of doing this is allowing modules to tap into this scoring hook at the time of indexing.

Each hook should return a value between 0 and 10 as a score to add to the node being indexed. Then in the administration screen an administrator would be able to set the importance of the score given by that hook. This would allow the administrator of a drupal site to manage the scoring from different contributed modules.

Part 1 of my project (implementing synonym matching in the search index) is nearly completed, I am waiting for the patches to be accepted into core for drupal 6. In addition to synonym matching I also submitted a patch to index usernames with the nodes as requested in the Search group on drupal.org. The patches can be reviewed here, all comments welcome.

http://drupal.org/node/155262 - Taxonomy synonym search indexing
http://drupal.org/node/155254 - Username search indexing

For part 2 of my project I am to implement a fuzzy search engine in drupal.