Corey Roth [MVP] : Enterprise Search, Wildcard Search, MOSS

Disabling Full Text Search with Wildcard Search Web Part Release 5

CoreyRoth — Wed, 08 Jul 2009 18:50:47 GMT

I’ve seen quite a few posts in the forums about people wanting to not search the contents of documents. These people for whatever reason were interested in just the title and path being searched. Out of the box, this of course is not possible, but I got to thinking about how this functionality could be quite easily added to Wildcard Search. With Release 5, I added a new property called DisableFullTextSearch. Selecting this box, will cause the web part to only search the Title and Url fields. It seems to work great. I also fixed a few minor issues with default values in the web part in a few places. If you don’t need this functionality, you can stay on your existing version.

I also took a look at the download numbers for all of the releases. Here is where it stands as of today.

Version	Downloads
Release 4	439
Release 3	1586
Release 2	397
Release 1	664

Not huge numbers, but I would say that is pretty decent for a SharePoint web part. Of course I have no way of knowing how many people are truly using it, but I would like to this that this web part is really helping people out. Although it doesn’t have all of the functionality that a commercial product does, it gets the job done and doesn’t cost the community anything to use. This functionality should have been included to begin with which is why I am happy to give it to the community for their use. Hope this release is useful to you and thanks for the continuing feedback.

Wildcard Search Release 5

Announcing custom sort order support for WildcardSearchWebPart

CoreyRoth — Wed, 20 May 2009 13:56:32 GMT

I was reading the forums the other day and saw another post in the MOSS Search forum about someone wanting to be able to specify a custom sort order with the CoreResultsWebPart. WildcardSearchWebPart will allow you to specify a static full text sql query or it builds one of its own. The problem is, the CoreResultsWebPart that it inherits from has logic in it to go and add an ORDER BY clause based on data modified or relevance. This causes an issue because if you specify your own order by clause, it causes a syntax error. This post had a rather simple solution to solving the issue. The answer is to simply add your own ORDER BY clause followed by a -- (SQL comment). This causes the syntax to be valid and everything works.

After seeing this, I immediately knew I wanted to add this to the WildcardSearchWebPart. I added a new property called SortOrder which would allow you pass any valid ORDER BY clause. For example:

Title
Author DESC
ContentSource, Author DESC, Title

You should be able to sort on any manager property that you have returned in the Select Columns XML field. Give it a try and see how it works. My next step to research is to see if I can implement this with a keyword query. Try it out and post any issues you run into on CodePlex.

Wildcard Search Version 1.0 Release 4

Follow me on twitter.

Introducing Document Link Handler for MOSS 2007 Enterprise Search

CoreyRoth — Fri, 31 Oct 2008 18:30:17 GMT

Since, I have started working with Enterprise Search, I have received many requests looking for ways to improve what we can do with documents. The search results screen in MOSS, as flexible as it is, has always been lacking in features compared to search results of other ECM systems. Common feature requests I have seen from clients include: editing the document, viewing properties, linking to the folder or document library, linking to the site the document is in, zip and download, view version history, and more. Today I am pleased to announce a simple solution which can handle many (but not all) of these common requests.

So how do we accomplish this? I decided to build a simple ASP.NET Handler (.ashx) file to parse the URL from a document on the search results page and redirect a user to the page that he or she wanted. Links to the handler can easily be added by modifying the XSL of the CoreResultsWebPart.

The code of the handler is pretty simple. I take the URL passed via query string, split it to get the filename and the path and then I open an SPWeb object that references that site. Once I have access to an SPFolder and SPFile object for the document, I can determine URLs for the document library and folder. Take a look at the code if you are interested. It might need some optimization, but it is a good starting point.

Installation is simple. Install the included .wsp file contained in the package folder. The syntax is listed in the readme.txt file if you need it. This will copy the DocumentLink.ashx file into your layouts folder. To add the links to your search results, go to the results page of your search center and edit your CoreResultsWebPart. In the Data View Properties, click on the XSL Editor button. Replace the XSL in the window with the contents of the included SampleCoreResults.xslt file. If you have already made customizations to this file or you only want certain links, you may copy just the lines you need from the readme.txt file.

After you have changed the XSLT, apply the changes and execute a search query. If you receive an error, then your XSL is probably malformed. Open the XSL with Visual Studio if necessary to help find the error. If all goes well, you should have search results that look like the one below. Note, that these links will only show up for search results that are documents (IsDocument:1). This works together in conjunction with my post earlier this week on adding an edit document link to search results.

The binaries and source code are available at CodePlex. If you have any issues, please log them there on the Issue Tracker. If you have any idea of other kinds of links we can provide off of a document library, please leave a comment. This can also be used with the Wildcard Search web part. Hopefully this will help many others out there deliver even richer search solutions to their customers. Thanks.

Document Link Handler for MOSS 2007 Enterprise Search

Wildcard Search Version 1.0 Release 3

CoreyRoth — Tue, 14 Oct 2008 21:46:09 GMT

Release 3 of the Wildcard Search Web Part is now available. This minor update fixes a user reported issue where the scope property was not being used. If you are new to the blog, the Wildcard Search web part allows a user to do partial word queries with Enterprise Search (i.e.: searching for red, returns results matching redmond, redding, and red). Please continue to report bug on the CodePlex site. Thanks again for everyone's input.

Also, don't forget the SharePint meetup, this Thursday at Crawpappy's at 6pm.

What you give up with Full Text SQL Queries using Wildcard Search

CoreyRoth — Thu, 18 Sep 2008 16:21:20 GMT

I am very glad to see that the Wildcard Search Web Part has been so well received. It's nice to know so many people are getting good use out of it. I have had a number of comments about it regarding what else works with it and what doesn't so I thought I would address them today in a post. We'll start with how the Search Center works out of the box. It uses the Enterprise Search Keyword Query syntax. As you probably know by now, this syntax does not support wildcard searching. To implement Wildcard Search, I had to use Full Text SQL Query syntax which does support wildcard searching. Unfortunately, a lot of the functionality on the search results page only works with keyword syntax. Here is a list of things that will not work or will work differently.

SearchSummaryWebPart - Provides the "Did you mean..." functionality.
Best Bets - Keyword based so it will not work.
Keyword Highlighting - Obviously keyword based as well.
RSS - This is keyword based as well.
Federated Search - Works but treats your search term as a keyword ignoring the wildcard.
Faceted Search - Keyword based, but I am confident we could write code on both our parts to make this work.

Ok, so this may sound like a lot, but in my experience a lot of people don't use some of this functionality. It is just a trade off between having wildcard search or not. This really isn't a result of the Wildcard Search web part alone. It's the result of using a Full Text SQL Query and what Enterprise Search's limitations are. We can confirm this by using the Advanced Search results web part. Depending on the type of query you submit, it will do a Full Text SQL Query and the same functionality is missing. It's been my experience that customer's really do like having the wildcard functionality and rarely miss the other features. Thanks again for all the comments.

Using Wildcard Search Web Part to search for recently modified files

CoreyRoth — Wed, 17 Sep 2008 13:39:53 GMT

You may have a case where you want to search for files that were recently modified. Sure, you can use a CAML query to do this, but what if you want to look for recently changed files across multiple content sources? Sure, you can write your own Full Text SQL Query and bind it in a web part, but there is an easier way. The new FixedFullTextSqlQuery property in Release 2 of the Wildcard Search Web Part makes this really simple. You just need the right query.

Date functions in Enterprise Search SQL are similar but the syntax is a little different. First you'll need to get the current date. The GETGMTDATE function is what you need to get the current date. Next you just need to use DATEADD to subtract the number of days, months, etc, that you need for your fixed query. In this case, I'll subtract 7 days by specifying DAY and -7. You can also specify MONTH, YEAR, HOUR, etc as well. The last modified date is stored in a managed property called Write (took me a while to figure that one out). That will make your condition look something like this.

WHERE Write > DATEADD(DAY, -7, GETGMTDATE())

Then, all you need to do is form your SELECT statement. You can't use SELECT *, so you need to specify all of your columns individually. The easiest way to get this column list is to look at the Select Columns property on your control. My example below lists most of them. When you are all done, this is what your query would look like.

SELECT Rank, Title, Path, Author, Write, WorkId, Size, Description, SiteName, CollapsingStatus, ContentClass, IsDocument, HitHighlightedSummary, HitHighlightedProperties, FROM Scope() WHERE Write > DATEADD(DAY, -7, GETGMTDATE())

This will query the entire index for documents modified in the last 7 days. You may want to filter it down to a specific scope by using an additional condition in your WHERE clause.

NEW! Web Part for Wildcard Search in Enterprise Search

CoreyRoth — Mon, 09 Jun 2008 13:28:15 GMT

I have had countless people tell me that they want to be able to do wildcard search in MOSS Enterprise Search using the existing Search Center site templates. The Search Center site template uses keyword query syntax which does not support wildcards. To get wildcards you need to use a full text SQL query. Until now the only solutions to getting wildcard search was either use Ontolica's Wildcard Search or write your own search page. Neither option has ever been appealing to me. Ontolica replaces the entire search center and provides an extra layer of abstraction to your managed properties. I didn't want to write my own search page because the search center already produces great looking results and it would be a lot of effort to reinvent everything on the search page. The reason why you would have had to write your own search page is that you could not get access via conventional means to the objects to change the query.

This is why I finally decided to take matters into my own hand. I really just wanted to just inherit from CoreResultsWebPart, change out the keyword query with a FullTextSqlQuery and call it good. Anyone who may have looked at this before knows it is not that simple. The class that does all the work SearchResultsHiddenObject is marked internal. Right now this seems an impossible task unless you bend the rules a little. That's right. I decided I am going to break OO rules and use reflection to get to the properties I needed. Some people say you should never do this and that its a hack. I agree to some extent, but when you are programming against an API and the provider of said API doesn't give you the tools you need to do your job, sometimes you have to bend the rules. Let's face it. Microsoft should have given us this support out of the box and at the minimum should have allowed us to change the query that the CoreResultsWebPart executes through the API. They did neither so here we are.

So how does the code work? Well, CoreResultsWebPart happens to be the one class in all of the Enterprise Search controls that isn't marked sealed. That is good news. Through the use of Reflector, I discovered the method I need to inherit from is SetPropertiesOnHiddenObject. Microsoft was even nice enough to mark this method as virtual for me. I then got access to the type and then used the base type to get a FieldInfo object for the private field srho (which is the SearchResultsHiddenObject).

// get the type of the current object
Type coreResultsWebPartType = this.GetType();
 
// get the private field containing the searchResultsHiddenObject
FieldInfo searchResultsHiddenObjectField = coreResultsWebPartType.BaseType.GetField("srho", BindingFlags.NonPublic | BindingFlags.Instance);

Once, I got access to the hidden object, I read the value of the KeywordQuery property to get what the user searched for. I then had to set this value to null, because I was replacing the keyword query with a FullTextSqlQuery.

// get the actual internal srho object attached to CoreResultsWebPart
object searchResultsHiddenObject = searchResultsHiddenObjectField.GetValue(this);
 
// get the type of the srho
Type searchResultsHiddenObjecType = searchResultsHiddenObject.GetType();
 
// get the keyword query property
PropertyInfo keywordQueryProperty = searchResultsHiddenObjecType.GetProperty("KeywordQuery", BindingFlags.Instance | BindingFlags.Public);
 
// read what the user searched for
string keywordQuery = (string)keywordQueryProperty.GetValue(searchResultsHiddenObject, null);
 
// set the keywordProperty to null so we can change it to a fullTextQuery
keywordQueryProperty.SetValue(searchResultsHiddenObject, null, null);

It was then just a matter of forming a new SQL query string (check the code on how I did that), and setting some additional fields _IsFullTextQuerySetFromForm and m_bIsKeywordQuery.

 
// get the fullTextQuery field
PropertyInfo fullTextQueryProperty = searchResultsHiddenObjecType.GetProperty("FullTextQuery", BindingFlags.Instance | BindingFlags.Public);
 
// create a new query and set it
string fullTextQueryString = GetFullTextQuery(keywordQuery, keywordsAsQuery);
fullTextQueryProperty.SetValue(searchResultsHiddenObject, fullTextQueryString, null);
 
// this field needs to be set to true to use a full text query
FieldInfo fullTextQuerySetField = searchResultsHiddenObjecType.GetField("_IsFullTextQuerySetFromForm", BindingFlags.NonPublic | BindingFlags.Instance);
fullTextQuerySetField.SetValue(searchResultsHiddenObject, true);
 
// tell the srho that it is not a keyword query any more
FieldInfo isKeywordQueryField = searchResultsHiddenObjecType.GetField("m_bIsKeywordQuery", BindingFlags.NonPublic | BindingFlags.Instance);
isKeywordQueryField.SetValue(searchResultsHiddenObject, false);

The code for this is really pretty simple (aside from the reflection). Had the SearchResultsHiddenObject been marked public, we could have had this functionality over a year ago, but oh well.

Installation

Installation is relatively simple and instructions are included in a readme file in the document. A solution package has been provided for ease of installation. Once the package has been installed activate the Wildcard Search Web Part feature on your site collection. I went with a site collection feature because the search center site does not have a web part gallery in it. Now that the feature is activated, go to your results page in your Search Center, edit the page, and add the Wildcard Search Core Results Web Part to the Bottom Zone. You can then remove the old CoreResultsWebPart. Also note that this web part requires .NET Framework 3.5 because I used LINQ to XML to parse through the SelectColumns property.

Usage

Once you have the web part installed, you can perform a wildcard search by just adding an asterisk to whatever you type in the search box. For example app* would return mataches on app, apple, and application. You can also set the Always Use Wildcard property in the Miscellaneous property settings to always perform a wildcard search.

One thing to note. Wildcard searches reek havoc on your search relevance. Where you might be used to having nice clean looking results with your keyword searches, your wildcard searches will look different. It might not always be obvious why a particular item was returned in the search results. The best thing to do is try and see if it works for your particular situation.

This type of web part probably could have been sold, but I thought it was more important to give it to the community. This feature gets asked for by MOSS customers all the time. Finally there is an easy solution to implementing it. Since this web part is new, I am sure there are going to be issues with it. Please, log any issues you run into in installation or in us on the issue tracker of the CodePlex site. Currently, it only supports simple keyword queries. I still need to implement support for passing scopes and managed properties, so look for that in an update soon.

You can find the release files at CodePlex.

Corey Roth is a MOSS consultant for Stonebridge.