Get to know the Refinement Web Part in SharePoint 2010 Enterprise Search

CoreyRoth — Mon, 15 Mar 2010 19:22:57 GMT

In MOSS 2007, people often extending the search results experience by using the Faceted Search Web Parts. People really liked these so it looks like Microsoft decided to implement their own version called the RefinementWebPart (also known as the Refinement Panel). If you are familiar with the Faceted Search Web Parts at all, you will notice there are a lot of similarities in the way things are implemented. If you’re not familiar with what I’m talking about, it’s this web part that allows users to drill down into a set of search results based upon managed properties and other criteria. Let’s take a look at a quick example.

On the left there, you will see a set of refinements that we get out of the box. I didn’t have to do anything to configure these at all. There are a number of refinements built in including file type, site, author, modified date, and taxonomy. You can also easily create your own based on managed property which we will see here shortly.

Let’s take a look at some of the options on this web part. If we edit the page and then edit the web part, we will be able to see the options. If you’re on a small screen (or a small window in the case of a VM), you will have to scroll right to see the web part properties.

Since this is a search web part, you will see the familiar Cross-Web Part query ID. This should be synced up to whatever else you are using on the page. Usually it is set to User query. The next section is where we can configure the refinement. Make note of the Use Default Configuration checkbox. If you don’t uncheck this, anything you customize will not be saved. Each thing that you can filter on is called a category by the RefinementWebPart. The Filter Category Definition property is an XML field where each category is specified. We’ll also look at that later in the post. Other properties to note here. The Accuracy Index is the number of results it looks at to determine things to refine. I assume this it mainly there to keep things performing well. What this does mean is that if there is something unique to refine on but it doesn’t occur until result number 51, then it will not be included. You can also configure how many categories to display. This is set to 6 by default. The last thing I will point out is that you can configure how the web part displays its information using XSLT (not shown in the screenshot above).

Now let’s look at how we can add our own managed property to the category list. You should of course confirm the managed property works and that you can query on it first (do a full crawl if necessary). We then, just need to take a look at the XML for the Filter Category Definition.

<?xml version="1.0" encoding="utf-8"?>
<FilterCategories>
  <Category    Title="Result Type"    Description="The file extension of the item"    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="4"    MaxNumberOfFilters="0"    SortBy="Frequency"    SortDirection="Descending"    SortByForMoreFilters="Name"    SortDirectionForMoreFilters="Ascending"    ShowMoreLink="True"    MappedProperty="FileExtension"    MoreLinkText="show more"    LessLinkText="show fewer">
    <CustomFilters MappingType="ValueMapping" DataType="String" ValueReference="Absolute" ShowAllInMore="False">
      <CustomFilter CustomValue="Adobe PDF">
        <OriginalValue>pdf</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Email">
        <OriginalValue>eml</OriginalValue>
        <OriginalValue>msg</OriginalValue>
        <OriginalValue>exch</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Excel">
        <OriginalValue>odc</OriginalValue>
        <OriginalValue>ods</OriginalValue>
        <OriginalValue>xls</OriginalValue>
        <OriginalValue>xlsb</OriginalValue>
        <OriginalValue>xlsm</OriginalValue>
        <OriginalValue>xlsx</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Image">
        <OriginalValue>tif</OriginalValue>
        <OriginalValue>tiff</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Lotus Notes">
        <OriginalValue>nsf</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="One Note">
        <OriginalValue>one</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="PowerPoint">
        <OriginalValue>odp</OriginalValue>
        <OriginalValue>ppt</OriginalValue>
        <OriginalValue>pptm</OriginalValue>
        <OriginalValue>pptx</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Publisher">
        <OriginalValue>pub</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Text">
        <OriginalValue>odt</OriginalValue>
        <OriginalValue>txt</OriginalValue>
        <OriginalValue>url</OriginalValue>
        <OriginalValue>csv</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Visio">
        <OriginalValue>vdw</OriginalValue>
        <OriginalValue>vdx</OriginalValue>
        <OriginalValue>vsd</OriginalValue>
        <OriginalValue>vss</OriginalValue>
        <OriginalValue>vst</OriginalValue>
        <OriginalValue>vsx</OriginalValue>
        <OriginalValue>vtx</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Webpage">
        <OriginalValue>ascx</OriginalValue>
        <OriginalValue>asp</OriginalValue>
        <OriginalValue>aspx</OriginalValue>
        <OriginalValue>htm</OriginalValue>
        <OriginalValue>html</OriginalValue>
        <OriginalValue>jhtml</OriginalValue>
        <OriginalValue>js</OriginalValue>
        <OriginalValue>mht</OriginalValue>
        <OriginalValue>mhtml</OriginalValue>
        <OriginalValue>mspx</OriginalValue>
        <OriginalValue>php</OriginalValue>
        <OriginalValue></OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="XML">
        <OriginalValue>xml</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Word">
        <OriginalValue>doc</OriginalValue>
        <OriginalValue>docm</OriginalValue>
        <OriginalValue>docx</OriginalValue>
        <OriginalValue>dot</OriginalValue>
        <OriginalValue>nws</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Zip">
        <OriginalValue>zip</OriginalValue>
      </CustomFilter>
    </CustomFilters>
  </Category>
  <Category    Title="Site"    Description="Which site this document is from"    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="4"    MaxNumberOfFilters="20"    SortBy="Frequency"    SortByForMoreFilters="Name"    SortDirection="Descending"    SortDirectionForMoreFilters="Ascending"    ShowMoreLink="True"    MappedProperty="SiteName"    MoreLinkText="show more"    LessLinkText="show fewer" />
  <Category    Title="Author"    Description="Use this filter to restrict results authored by a specific author"    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="4"    MaxNumberOfFilters="20"    SortBy="Frequency"    SortByForMoreFilters="Name"    SortDirection="Descending"    SortDirectionForMoreFilters="Ascending"    ShowMoreLink="True"    MappedProperty="Author"    MoreLinkText="show more"    LessLinkText="show fewer"    />
  <Category    Title="Modified Date"    Description="When the item was last updated"    Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="6"    MaxNumberOfFilters="0"    SortBy="Custom"    ShowMoreLink="True"    MappedProperty="Write"    MoreLinkText="show more"    LessLinkText="show fewer" >
    <CustomFilters MappingType="RangeMapping" DataType="Date" ValueReference="Relative" ShowAllInMore="False">
      <CustomFilter CustomValue="Past 24 Hours">
        <OriginalValue>-1..</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Past Week">
        <OriginalValue>-7..</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Past Month">
        <OriginalValue>-30..</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Past Six Months">
        <OriginalValue>-183..</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Past Year">
        <OriginalValue>-365..</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Earlier">
        <OriginalValue>..-365</OriginalValue>
      </CustomFilter>
    </CustomFilters>
  </Category>
  <Category    Title="Managed Metadata Columns"    Description="Managed metadata of the documents"    Type="Microsoft.Office.Server.Search.WebControls.TaxonomyFilterGenerator"    MetadataThreshold="3"    NumberOfFiltersToDisplay="3"    MaxNumberOfFilters="20"    ShowMoreLink="True"    MappedProperty="ows_MetadataFacetInfo"    MoreLinkText="show more"    LessLinkText="show fewer" />
  <Category    Title="Tags"    Description="All managed metadata of the documents and social tags"    Type="Microsoft.Office.Server.Search.WebControls.TaxonomyFilterGenerator"    MetadataThreshold="3"    NumberOfFiltersToDisplay="3"    MaxNumberOfFilters="20"    ShowMoreLink="True"    MappedProperty="ows_MetadataFacetInfo,popularsocialtags"    MoreLinkText="show more"    LessLinkText="show fewer" />
</FilterCategories>

I went ahead and posted the entire XML because it’s worth seeing. For the most part the schema is pretty easy to follow. The Category element defines each thing to refine and then it has some basic configuration items such as the number of filters to display. You specify the name of the managed property to use in the MappedProperty attribute. The MetadataThreshold property is the number of results that have to be returned with that property in order to do refinement. If you are familiar with the schema used by Faceted Search, you will see a similar concept with the CustomFilter elements. These allow you map a value into something more readable. For example instead of display xlsx, it displays Excel.

One more thing I will point out is that the Category element has a Type attribute. So far I have seen ManagedPropertyFilterGenerator and TaxonomyFilterGenerator. These both inherit from RefinementFilterGenerator. None of its sealed surprisingly which means you could actually write your own custom filter for the refinement web part. That’s pretty cool. I’m not sure why I would need to yet, but you never know what you might want to be able to customize.

I want to add my own managed property, so I just add a category to the end like this.

<Category    Title="Color"    Description="Use this filter to restrict results by color" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"    MetadataThreshold="5"    NumberOfFiltersToDisplay="4"    MaxNumberOfFilters="20"    SortBy="Frequency"    SortByForMoreFilters="Name"    SortDirection="Descending"    SortDirectionForMoreFilters="Ascending"    ShowMoreLink="True"    MappedProperty="Color"    MoreLinkText="show more"    LessLinkText="show fewer"    />

All I did was copy the category that was used for author and it works great. You can tweak the individual settings if you like. Here is what my custom managed property looks like.

How does this all work though. Well it’s pretty simple. Just copy any link that the web part displays and we’ll see that it makes use of the new r query string property. It will be URL encoded but you can easily decode it and see the magic. In this case if I want to view Red products, the query string has a r parameter of the following.

r=color="Red"

If I wanted to view files modified in the last 24 hours, it uses the new >= operator.

r=write>="3/14/2010”

Don’t you just love that you can easily query against dates now using the keyword query syntax? I think the r keyword is very interesting and I think it opens the door for some very interesting customizations of the search results page in the future.

Enterprise Search Web Parts are still Sealed! Unseal them! (SP2010)

CoreyRoth — Wed, 09 Dec 2009 21:44:54 GMT

I am pretty passionate about Enterprise Search and with MOSS 2007 I saw customers really struggle because wildcard search was not offered out of the box and most of the web parts in Microsoft.Office.Server.Search.WebParts are sealed. Luckily, the most important one CoreResultsWebPart was in fact not sealed and I was able to hack together a wildcard search solution by inheriting it and using some hacky reflection. Since SPC09, we’ve been hearing “the search web parts are no longer sealed”, this unfortunately is not true at all. What they mean to say is that they have added a couple of override to CoreResultsWebPart that allow you to affect the query going in and the results going out. This is a nice win, but there is still more desired.

It took me asking a lot of people at SPC09 to finally get a straight answer. It was in the session, Customizing Search in SharePoint: Building Great Sites with Search, where I asked Harneet Sidhana at the microphone to get the story on why most web parts are still sealed. The answer? The remaining web parts are currently still sealed by design. The audience “booed”. Seriously, they did. I then asked to have the web parts unsealed and the audience applauded.

Let’s take a look at the web parts between MOSS 2007 and SP2010 Beta 2 and see what is sealed.

Web Part	MOSS 2007	SP2010 Beta 2
AdvancedSearchBox	Sealed	Sealed
CoreResultsWebPart	Unsealed	Unsealed
FederatedResultsWebPart	Unsealed	Unsealed
HighConfidenceWebPart	Sealed	Sealed
PeopleCoreResultsWebPart	Sealed	Sealed
PeopleRefinementWebPart		Unsealed
QuerySuggestionsWebPart		Unsealed
RefinementWebPart		Unsealed
SearchPagingWebPart	Sealed	Sealed
SearchStatsWebPart	Sealed	Sealed
SearchSummaryWebPart	Sealed	Sealed
TopFederatedResultsWebPart	Unsealed	Unsealed

Looking at this list, I don’t really see anything that has changed from sealed to unsealed. Do you? I will point out that CoreResultsWebPart has always been unsealed, but this is because PeopleCoreResultsWebPart inherits from it. Did you notice that anything built since the Infrastructure Update of MOSS 2007 has been unsealed?

Now I admit, the main reason we wanted to inherit from these controls was to do wildcard search. The Enterprise Search did a great job with the new keyword query improvements and this will be a non-issue now. However, can they really say we will never want to extend their web parts? Are they going to say that they have thought of every possible thing that the web part might possibly do and no one will ever extend it? As a programmer, I don’t think I would ever have the guts to make a claim like that. I know these web parts will require customization in the future. Give us the ability to do it.

I know you might be thinking, “We just wanted to inherit from these web parts because we wanted wildcard search. We have that now.” This is true and I am very thankful. This eliminates the need a lot, but more than likely we might need to make a change to something else though. Want to customize people search? What about best bets or the advanced search box? Most web parts in SharePoint are unsealed, so why are the search web parts sealed?

The PeopleCoreResultsWebPart still being sealed is a huge deal. For one, it doesn’t appear to support the new QueryManager class which means you can’t customize how it does queries or builds the results. I need to confirm this though. Take a look at the forums. They speak for themselves. Fifty results for the term wildcard in the Search forum. I am passionate about this topic because I have seen the needs of my customers. The fact that I might be stuck with another set of sealed web parts for another 3+ years is absolutely scary to me.

So what can we do? Please post a reply in the forum or leave a comment on the blog. We need to let them know our concerns while there is still time to make a change. Please, unseal the web parts. This post is cross posted to the SharePoint 2010 forums.

Corey Roth [MVP] : SharePoint 2010, SPC09, Enterprise Search, SharePoint

Get to know the Refinement Web Part in SharePoint 2010 Enterprise Search

Enterprise Search Web Parts are still Sealed! Unseal them! (SP2010)