Corey Roth [MVP] : Enterprise Search

How to: Get a Result Source Id in SharePoint 2013

CoreyRoth — Tue, 21 May 2013 15:45:00 GMT

Result Sources in SharePoint 2013 are quite powerful. However to make use of them in your queries, you have to know the Id which you pass to the SourceId parameter of a query. For out-of-the-box result sources, the Ids are static. When you start creating your own though, you need a way to retrieve the Ids programmatically. Unfortunately, the only way to retrieve the Ids is to use the managed API. This means they can’t be retrieved using JavaScript or REST. With that aside, I still wanted to show you how you could write managed code to retrieve a result source id.

For today’s example, I am going to build a console application, but you could also implement this in a web part or other code directly executing on a SharePoint server in the farm. Whenever you build a console application using SharePoint, be sure and change the platform target to x64. Then, we need to add a few references which we can find in the ISAPI folder of the 15 hive. Add the following assemblies to your project.

Microsoft.Office.Server
Microsoft.Office.Server.Search
Microsoft.SharePoint
Microsoft.SharePoint.Security

In our code, we’ll need a heap of assemblies to get started. Add the following using statements.

using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration;
using Microsoft.Office.Server.Search;
using Microsoft.Office.Server.Search.Administration;
using Microsoft.Office.Server.Search.Administration.Query;

Next, we need to get a reference to the Search Service Application Proxy so that we can retrieve the result source. There are a number of ways to do this, but I am going to get it by name.

SearchQueryAndSiteSettingsServiceProxy settingsProxy = SPFarm.Local.ServiceProxies.GetValue<SearchQueryAndSiteSettingsServiceProxy>();
SearchServiceApplicationProxy searchProxy = settingsProxy.ApplicationProxies.GetValue<SearchServiceApplicationProxy>("Search Service Application");

Once you have a proxy object, it is easy to get the result source and then the id. The proxy object has a method GetResultSourceByName method. You can pass it the name of the result source you want, but you also need to specify a SearchObjectOwner object to specify which level you want to access it at (Service Application, Site Collection, or Site). I’ll show you each one starting with Service Application. In my example, I have result resources created at each level.

This creates a SearchObjectOwner for the site collection.

SearchObjectOwner serviceApplicationOwner = new SearchObjectOwner(SearchObjectLevel.Ssa);

Next we pass it as the second parameter to GetResultSourceByName. The first parameter is simply the name of the result source.

SourceRecord serviceApplicationResultSource = searchProxy.GetResultSourceByName("Service Application Result Source", serviceApplicationOwner);

Assuming, there is a matching result source, all you need to do is use the Id parameter on the SourceRecord object.

Guid serviceApplicationResultSourceId = serviceApplicationResultSource.Id;

You can then use that Id in your own queries using the SourceId keyword.

If you want to reference a result source at the site collection or site level, you need to do a few extra steps. Start by getting a web object. You need this whether you are using a site collection or site and I’ll show you why shortly.

using (SPSite site = new SPSite("http://sp2010/sites/test"))
{
    using (SPWeb web = site.OpenWeb())
    {
 
    }
}

Inside our using block, we’ll add our code. Let’s get things at the site (web) level first. We do this by creating a SearchObjectOwner specifying a SearchObjectLevel of SPWeb and passing a second parameter containing our SPWeb object.

SearchObjectOwner siteOwner = new SearchObjectOwner(SearchObjectLevel.SPWeb, web);

Now, we pass the SearchObjectOwner to the GetResultSourceByName method like before.

SourceRecord siteResultSource = searchProxy.GetResultSourceByName("Site Result Source", siteOwner);

Like before, we can use the Id parameter to get the id of the result source.

To access result sources at a site collection, the API is a bit different than what you might be used to. We create a new SearchObjectOwner object and we specify a SearchObjectLevel of SPSite. In this case, we also pass it the SPWeb object from before even though we want the site collection. It’s a bit different but it works. Here’s what it looks like.

SearchObjectOwner siteCollectionOwner = new SearchObjectOwner(SearchObjectLevel.SPSite, web);
SourceRecord siteCollectionResultSource = searchProxy.GetResultSourceByName("Site Collection Result Source", siteCollectionOwner);

That’s all it takes to get a result source id. As I mentioned earlier, it is unfortunate that we can’t get a result source id any other way, but this will work if you are writing managed code in on-premises SharePoint 2013.

Slides and code samples from my Search talk at SharePoint Summit Toronto

CoreyRoth — Thu, 16 May 2013 14:42:53 GMT

I’ve had yet another fun week up here in Canada. I gave a brand new talk about the new ways to query search in SharePoint 2013. I had a room full of developers that asked a lot of great questions. As promised, my slide deck is below at SlideShare.

Five ways to query SharePoint 2013 Search

In the session, I mentioned that I had code samples on MSDN Code, so I am providing the link below. Try them out and download them and rate them if you find them useful.

SharePoint 2013 Search Samples

Follow me on twitter: @coreyroth

How to: Query Search with the SharePoint 2013 JavaScript Client Object Model

CoreyRoth — Fri, 19 Apr 2013 03:08:32 GMT

As a developer, if you have ever looked for information on how to query search in SharePoint, there is a good chance that you have ran into one of my articles. I’ve been writing about how to query search in a variety of different ways since 2007. They have always been some of the most popular articles on my site. SharePoint 2013 has so many new ways to query search, it has taken me time to write up all of the different ways. I am keeping the tradition alive by writing about how to query using the JavaScript Client Object Model provided by SP.Search.js. You may have already seen my post on how to query from JavaScript and REST or my post on how to use the managed CSOM to Query Search. You’ll find that querying from JavaScript looks very similar to the managed side of things. In fact even the namespaces are the same and most of the code is quite similar between C# and JavaScript. However, there are some nuances due to JavaScript so I wanted to share a complete code sample.

I am going to build my example inside a SharePoint-hosted app. If you’re not familiar with that process yet, take a look at my Client Web Part post to get started. When it comes to apps, you need to be sure and request permission to access search. Do this by editing your AppManifest.xml and clicking on the permissions tab. Select Search and then select QueryAsUserIgnoreAppPrincipal. If you forget this step, you won’t get an error, your queries will simply just return zero results.

I’m going to put all of my code in the welcome page today. My default.aspx page will have the same HTML as my other JavaScript post. I simply add a textbox, a button, and a div to hold the results. The user will type in his or her query, click the button, and then see search results.

<div>

<label for="searchTextBox">Search: </label>

</div>

</div>

We’re going to edit App.js next. Since we’re in an App, I am assuming you already have code to get the SharePoint context since Visual Studio adds it for you. If not, you can get it like this.

var context = SP.ClientContext.get_current();

I then add a click event handler to my document ready function.

$("#searchButton").click(function () {
});

In here, we’ll put our code to get a KeywordQuery and SearchExecutor object to execute our query in a similar manner to the Managed CSOM approach. Now, we need to get our KeywordQuery object using the context object we already have.

var keywordQuery = new Microsoft.SharePoint.Client.Search.Query.KeywordQuery(context);

Then, we set the query text using the value the user entered in the textbox with set_queryText().

keywordQuery.set_queryText($("#searchTextBox").val());

Now, we just need to create a SearchExecutor object and execute the query. I am assigning the results of the query back to a global variable called results. Since apps by default have ‘use strict’ enabled, you need to be sure and declare this first.

var searchExecutor = new Microsoft.SharePoint.Client.Search.Query.SearchExecutor(context);
results = searchExecutor.executeQuery(keywordQuery);

Then like any CSOM code, we have to use executeQueryAsync to make the actual call to the server. I pass it methods for success and failure.

context.executeQueryAsync(onQuerySuccess, onQueryError);

I generally prefer using REST to do my search queries. However, when it comes to processing results, the data we get back from CSOM is typed better and much easier to access. This results in less code that we have to deal with. You can get quite a bit of data back such as the number of results returned just by exmaining results.m_value. Each individual result can be found in results.m_value.ResultTables[0].ResultRows. The managed properties of each row are typed directly on the object so that it means you can access them directly if you know the name (i.e.: Author, Write, etc). Iterating the values is simple using $.each. Take a look at my example where I am writing the values into an HTML table.

function onQuerySuccess() {
    $("#resultsDiv").append('<table>');
 
    $.each(results.m_value.ResultTables[0].ResultRows, function () {
        $("#resultsDiv").append('<tr>');
        $("#resultsDiv").append('<td>' + this.Title + '</td>');
        $("#resultsDiv").append('<td>' + this.Path + '</td>');
        $("#resultsDiv").append('<td>' + this.Author + '</td>');
        $("#resultsDiv").append('<td>' + this.Write + '</td>');
        $("#resultsDiv").append('</tr>');
    });
 
    $("#resultsDiv").append('</table>');
}

The last thing to implement is the code to handle errors. It just sends the error to an alert dialog.

function onQueryFail(sender, args) {
    alert('Query failed. Error:' + args.get_message());
}

When we run the app, it will look something like this.

Executing a query will show us the four columns I specified in my $.each statement.

Pretty easy, right? Here is the entire code snippet of my App.js.

'use strict';
 
var results;
 
var context = SP.ClientContext.get_current();
var user = context.get_web().get_currentUser();
 
// This code runs when the DOM is ready and creates a context object which is needed to use the SharePoint object model
$(document).ready(function () {
 
    $("#searchButton").click(function () {
        var keywordQuery = new Microsoft.SharePoint.Client.Search.Query.KeywordQuery(context);
        keywordQuery.set_queryText($("#searchTextBox").val());
 
        var searchExecutor = new Microsoft.SharePoint.Client.Search.Query.SearchExecutor(context);
        results = searchExecutor.executeQuery(keywordQuery);
 
        context.executeQueryAsync(onQuerySuccess, onQueryFail)
    });
});
 
 
function onQuerySuccess() {
    $("#resultsDiv").append('<table>');
 
    $.each(results.m_value.ResultTables[0].ResultRows, function () {
        $("#resultsDiv").append('<tr>');
        $("#resultsDiv").append('<td>' + this.Title + '</td>');
        $("#resultsDiv").append('<td>' + this.Author + '</td>');
        $("#resultsDiv").append('<td>' + this.Write + '</td>');
        $("#resultsDiv").append('<td>' + this.Path + '</td>');
        $("#resultsDiv").append('</tr>');
    });
 
    $("#resultsDiv").append('</table>');
}
 
function onQueryFail(sender, args) {
    alert('Query failed. Error:' + args.get_message());
}

Although, I typically prefer the REST interface for querying search. I have to admit, I like the ease of working with the results in CSOM. Hopefully, you find this sample useful. Thanks!

I’ve also uploaded the full Visual Studio solution to MSDN Code Samples.

How to: Query SharePoint 2013 using REST and JavaScript

CoreyRoth — Tue, 09 Apr 2013 19:15:40 GMT

I have a session coming up at SharePoint Summit Toronto this year about the many different ways you can query search. Whenever I am working on a new talk, it is customary for me to write blog posts about my examples so here it is. :) I first learned how to query SharePoint 2013 search with the new REST API and JavaScript from looking at examples from @ScotHillier on MSDN. However, the last time I tried the example, I noticed an issue which I equate to most likely a change between beta and RTM. This post shows you my version of how to query search using the REST API and JavaScript.

For this post, I am using the RTM Office Developer tools (which I have a post coming out on soon). I am going to use a SharePoint-hosted app for my example but you could also use this in a web part with farm solution as well. When it comes to apps, you need to be sure and request permission to access search. Do this by editing your AppManifest.xml and clicking on the permissions tab. Select Search and then select QueryAsUserIgnoreAppPrincipal. If you forget this step, you won’t get an error, your queries will simply just return zero results.

For my example, I am just going to add my code to the default.aspx page in the app. I simply add a textbox, a button, and a div to hold the results. The user will type in his or her query, click the button, and then see search results. You could put this in a Client Web Part if you wanted.

<div>
    <label for="searchTextBox">Search: </label>
    <input id="searchTextBox" type="text" />
    <input id="searchButton" type="button" value="Search" />
</div>
 
<div id="resultsDiv">
</div>

Now, we need to add the necessary code to App.js. I start by removing the example code that retrieves the user information. I instead, add a click handler to my searchButton to execute the search query. If you remember from my previous post on REST, we assemble a URL by appending /api/search/query to a SharePoint host URL. For example.

http://server/_api/search/query

However, in an app, we have to request the URL to the App. One way to do this is by query string using the SPAppWebUrl parameter. SharePoint passes this parameter to your app start page automatically. We can request it with a line line this. Remember getQueryStringParameter() is a helper method that we have gotten from some of the SharePoint examples. I’ll include it in the full code listing at the bottom of this post.

var

spAppWebUrl = decodeURIComponent(getQueryStringParameter('SPAppWebUrl'));

To pass the user’s query to SharePoint, we need to include the querytext parameter on the REST URL. Be sure to enclose the value in single quotes. Again more details in my previous post.

http://server/_api/search/query?querytext=’SharePoint’

Now, we need to create a click handler for the search button, build the URL, and then execute the query.

"#searchButton").click(function () {

});

Inside the click handler, assemble the URL, using the concatenation of the spAppWebUrl, /api/search/query, and the querytext parameter. The value of querytext will be retrieved from the textbox we added earlier.

var queryUrl = spAppWebUrl + "/_api/search/query?querytext='" + $("#searchTextBox").val() + "'";

Now, we just execute the query with $.ajax(). Pass the queryUrl in the url parameter. This example uses the GET method but if you have a lot of parameters, you may consider using POST instead. Lastly, this part is key to get this example to work. The accept header must have value of "application/json; odata=verbose". The odata=verbose part is not in the MSDN example. If you leave it out, you will receive an error. The last parameters are the methods that will handle the success and failure of the AJAX call. Here’s what the whole method looks like.

$("#searchButton").click(function () {
    var queryUrl = spAppWebUrl + "/_api/search/query?querytext='" + $("#searchTextBox").val() + "'";
 
    $.ajax({ url: queryUrl, method: "GET", headers: { "Accept": "application/json; odata=verbose" }, success: onQuerySuccess, error: onQueryError });
});

Now, you need to write code to handle the success. The results come back in JSON format, but unfortunately, they are buried in a hugely nested structure. Each individual result can be found in data.d.query.PrimaryQueryResult.RelevantResults.Table.Rows.results. Your best bet is to assign this a variable and then use a template or manually parse it. In my example, I am just going to use $.each to iterate through the results using brute force. The individual columns of each search result can be found in this.Cells.results.Value. Ok, that’s confusing I am sure, so let’s look at the code and then step through it. I’m just writing out a simple table by appending HTML tags to a div.

function onQuerySuccess(data) {
    var results = data.d.query.PrimaryQueryResult.RelevantResults.Table.Rows.results;
 
    $("#resultsDiv").append('<table>');
 
    $.each(results, function () {
        $("#resultsDiv").append('<tr>');
        $.each(this.Cells.results, function () {
            $("#resultsDiv").append('<td>' + this.Value + '</td>');
        });
        $("#resultsDiv").append('</tr>');
    });
 
    $("#resultsDiv").append('</table>');
}

Effectively, I have two nested loops. One to iterate through each result and one for each managed property (column) in the results. When you get inside the Cell, this.Value will show the value of the result while this.Key will contain the name of the managed property.

The last thing to implement is the code to handle errors. It’s relatively simple and just writes error.StatusText to the div.

function onQueryError(error) {
    $("#resultsDiv").append(error.statusText)
}

When I run my app, here is what it looks like.

Executing a query in the app, gives us results but they aren’t pretty. You can pretty them up yourself. :) It also doesn’t include anything to print out the managed property names on the table. You could get the names by looking at each cell of the first result and using this.Value.

You can see, it’s really pretty easy to get started. You can further refine your REST query to request specific fields, sort orders, and more. I’ll probably write a follow-up post in the future to include some of those details. Here’s the whole source code for my example that you can work with.

var context = SP.ClientContext.get_current();
 
// This code runs when the DOM is ready and creates a context object which is needed to use the SharePoint object model
$(document).ready(function () {
 
    var spAppWebUrl = decodeURIComponent(getQueryStringParameter('SPAppWebUrl'));
 
    $("#searchButton").click(function () {
        var queryUrl = spAppWebUrl + "/_api/search/query?querytext='" + $("#searchTextBox").val() + "'";
 
        $.ajax({ url: queryUrl, method: "GET", headers: { "Accept": "application/json; odata=verbose" }, success: onQuerySuccess, error: onQueryError });
    });
 
});
 
function onQuerySuccess(data) {
    var results = data.d.query.PrimaryQueryResult.RelevantResults.Table.Rows.results;
 
    $("#resultsDiv").append('<table>');
 
    $.each(results, function () {
        $("#resultsDiv").append('<tr>');
        $.each(this.Cells.results, function () {
            $("#resultsDiv").append('<td>' + this.Value + '</td>');
        });
        $("#resultsDiv").append('</tr>');
    });
 
    $("#resultsDiv").append('</table>');
}
 
function onQueryError(error) {
    $("#resultsDiv").append(error.statusText)
}
 
//function to get a parameter value by a specific key
function getQueryStringParameter(urlParameterKey) {
    var params = document.URL.split('?')[1].split('&');
    var strParams = '';
    for (var i = 0; i < params.length; i = i + 1) {
        var singleParam = params[i].split('=');
        if (singleParam[0] == urlParameterKey)
            return decodeURIComponent(singleParam[1]);
    }
}

That’s all there is to it. Hopefully, you find this example useful. Thanks.

The complete Visual Studio for this code snipped has also been uploaded to MSDN Code Samples.

Importing Search Configurations with SharePoint Apps

CoreyRoth — Wed, 03 Apr 2013 16:21:44 GMT

The Office Developer Tools team snuck a new feature into the RTM version of the tools for Visual Studio 2012. This new feature allows you to deploy apps and actually alter the search schema on the host web. That’s right. You can deploy an app and it will directly change the search configuration on the host. They just released documentation on it a while back, but as usual, I wanted to share my experiences. That and I know you all like screenshots.

What does this feature actually do? Well let’s back up a bit. If you remember back from my post, Search is Everywhere, I mentioned the we now had the ability to export and import search settings. This works at the SSA, site collection, and site level and allows you to move everything from result sources to managed properties from one environment to another. This is big as it lets you finally promote search settings between environments and maintain a true SDLC when it comes to search. Why do we care about search configuration with apps? Well this allows the developer to package up search settings in Visual Studio 2012 and then move them to production without having to do manual steps or use PowerShell. This also means you could include search settings in an app that you would put in the Office Store. It certainly opens up possibilities.

To test this out, go to your source site collection and customize your search settings. In my example, I created a custom result source and some managed properties on our source site. In my example, I actually did this on an on-premises installation of SharePoint 2013.

This particular result source does nothing exciting. It simply limits the search to documents, but it serves as a good example. I’ve also created a managed property mapped to the Author crawled property. You may already know about this part, but I am showing it for a reason.

Now, I am going to export my search settings of my site collection, by going to Site Settings –> Search –> Configuration Export.

At this point, I could manually import the search settings using Configuration Import on another site collection. However, we want to do this from an app. Let’s get started in Visual Studio 2012. Start by creating a new SharePoint-hosted app. Once it is created, add an item to the project and choose Search Configuration.

The next step will ask for the path to your configuration XML file that you exported.

At this point the process is done. It will show you an XML editor with the contents of your search configuration. According to the MSDN documentation, you then need to edit it and set the DeployToParent element to true.

We then need to grant permissions to access the Site Collection. To do this, open AppManifest.xml and then click on Permissions. On this tab, add a scope of Site Collection and set the value to Full Control.

At this point, we are are ready to deploy. In my example, I am taking my search configuration and deploying it to an Office 365 SharePoint Online tenant. When the app deployment completes, you’ll be prompted if you want to trust the app. Trust it and then you should see your app start page.

At this point, you are just going to see you default app start page. There is nothing visible in the application. Go to the Developer Site (or the site collection you deployed to) and go to the Site Settings. Then look at the Result Sources. If everything worked correctly, you should now see your new result source there.

It was successfully deployed. Now what about the managed property? Unfortunately, it is no where to be found. If you go back to Visual Studio and look at your XML, you’ll notice that your managed property definition is no where to be found there either. If you check the source file before you imported it though, you’ll see the definition. After I noticed this particular behavior, I reached out on Twitter and @chakkaradeep reached out to me and told me that managed properties aren’t supported in this deployment model. That made me kind of sad because that’s what I want to deploy the most. I’m sure there is a technical reason though that he’ll explain to me sometime though. You can still deploy managed properties via Configuration Import though which is still a great added feature of SharePoint 2013.

You might be curious if the result source is removed when you uninstall the app. It turns out that the changes are indeed removed when you uninstall.

Aside from the managed properties not being available, this is still a pretty cool feature and it has me thinking about some new things I can do that I didn’t think were possible before. I’m pretty excited to work with it more.

Determining the most popular items in a document library in SharePoint 2013

CoreyRoth — Thu, 28 Mar 2013 19:39:39 GMT

With the combination of analytics and search in SharePoint 2013, there is a heap of new features to help determine which content is actually being used. No longer is this information buried in reports for administrators, but it in fact is available right from the ribbon in a document library. Maybe you have even noticed the Most Popular Items button in the ribbon already. You can find it on the library tab in the middle.

Click on it and you will be taken to a custom search results screen that shows you recent views and total views.

What’s nice is that you can even search and refine within the report if you are looking for something specific. You can also click on the Popularity Trends link to get a nice graph of the usage of the file over time.

Behind the scenes, this report is powered by the Popular result source. You can also use this source to look for popular items across your entire index. As a developer, the possibilities are quite interesting.

My particular tenant doesn’t have a ton of usage, but if you’re already running SharePoint 2013 you might be able to extract some valuable insights from this report. Try it out today.

How to: Sort search results by query string with the ResultScriptWebPart in SharePoint 2013

CoreyRoth — Wed, 09 Jan 2013 20:21:09 GMT

By now you may have noticed that Search has changed quite a bit in SharePoint 2013. The ResultScriptWebPart powers most of the magic behind the Search Center. It replaces the CoreResultsWebPart that we have been using since SharePoint 2007. I expected many things in it to operate the same, but this web part really has been rewritten from the ground up and that’s a good thing. One new feature (of many) it brings us is the ability to sort search results. You can specify a sort order in the query or by enabling it in the web part properties. This allows the user to pick from a list of properties. I’ll cover it in the future because it deserves a quick post. However, for you developers out there, I know that some of you will want to write some code that passes a sort order. This post explains how to do it.

Before we begin, you have to know what you can sort of. There are a lot of options out-of-the-box such as Author and Write (Modified Date) or you can use your own property. Whatever you choose, you need to check the managed properties page and make sure it has the Sortable option set to Yes - active. You do this from the Search Schema page on your site collection or Search Service Application.

Since SharePoint 2007, we have been able to pass a query using KQL using the “k” parameter to the results.aspx of your search center. We can still do this in SharePoint 2013. To add a sort order though, we do things a little differently. Using a new query string parameter called Default, we can pass in a number of properties including sort order. The format is quite a bit different though as it uses a JSON notation. To start, I am going to issue a query with out k parameter using this new JSON notation. Let’s do a search for our sales documents. To do this, I include my search center URL (/search/pages/results.aspx) and append #Default= to it followed by the JSON object with a name value pair {"k":"Sales"}. When I assemble to entire string, it looks something like this.

http://server/search/Pages/results.aspx#Default={"k":"Sales"}

Now we want to add sorting. To do that, we use the “o” parameter. It takes a collection of JSON objects. The parameter p represents the managed property to sort by and the value d is the sort direction specified as a 0 or 1. A value of 0 represents ascending and a value of 1 represents descending. For example to sort by modified date descending, we use the Write managed property. The string would look like this:

"o":[{"d":1,"p":"Write"}]

When we add it to our query string from above it looks like this.

http://server/search/Pages/results.aspx#Default={"k":"Sales","o":[{"d":1,"p":"Write"}]}

This shows us the newest documents. In SharePoint 2013, they moved the modified date to the InfoCard so you actually have to hover over a search result to see the date. To show the oldest documents, you would just specify d:'”0” instead.

As another example, we could also sort by Author.

http://server/search/Pages/results.aspx#Default={"k":"Sales","o":[{"d":0,"p":"Author"}]}

I’ve included a bigger screenshot where you can see the name of the first author in the list on the InfoCard.

We can also combine them to have multiple sort orders, but I have had mixed results with that so far. Here’s what the query string would an example look like though.

http://server/search/Pages/results.aspx#Default={"k":"Sales","o":[{"d":0,"p":"Write"},{"d":1,"p":"Author"}]}

I’m pretty excited about sorting functionality in the search center. Users have been wanting it for quite a long time and only a select few ever had it (those with FAST Search for SharePoint). If you are writing search solutions, be sure and check my post about sorting with the REST API. This technique works with both SharePoint 2013 on-premises as well as SharePoint Online Preview. All of the screenshots came from Office 365.

No metadata? No problem! Custom entity extraction in SharePoint 2013

CoreyRoth — Tue, 08 Jan 2013 16:00:00 GMT

Every organization wants better search results. However, few have actually spent the time to create file plans, content types, and managed properties to make it happen. SharePoint 2013 has a feature called entity extraction though that can actually infer values by comparing what it finds in the body of the document to a dictionary. This feature is Entity Extraction. If you are familiar with FAST Search for SharePoint or FAST ESP, you know that entity extraction is nothing new. What it does is gives you the power to infer the values of managed properties in search without having metadata in your site columns. FS4SP and SharePoint 2013 come with an extraction dictionaries for company names out-of-the-box. This means when it crawls it will search the contents of the body of your documents and if it recognizes any companies it knows, you can then search and refine on it. However, the true value comes when you create your own. SharePoint 2013 makes it really easy to create these custom dictionaries.

Let me give you a few more examples to help you really understand the significance. Most companies have a concept of a department or business unit. We can create a dictionary with all of the company’s departments such as Accounting, Human Resources, IT, Operations, etc. With entity extraction, if it finds a document and somewhere the text “Human Resources Organization Chart” is found, it then sets the value of the managed property to Human Resources. Now, when the user does an advanced search and says “show me all Human Resources documents”, that result comes back with more confidence. By using the extraction dictionary, we get better results that just using a regular keyword search. Now you may be wondering that’s nothing special, but we can also add other terms, for example, I could add “HR”, and “Benefits” to that dictionary as well and map them to the Human Resources department. Once I have a property defined like this, I can combine it with others as well. This will let us issue queries like “show me all documents from Human Resources with a document type of Policy”.

As another example, you could pre-load SharePoint with a dictionary of product names. For those of you in the energy industry, you could pre-populate it with a list of wells or names of leases. Are you starting to see the significance? In previous versions of SharePoint, we had to set site columns in our document libraries to capture these values either from the user or programmatically. Now with the right dictionary, we can assume the values with a reasonable degree of confidence.

Alright, enough explanation, let’s see this in action and hopefully it will make sense. MSDN has a great post on this but I found and error in it so that is why I am writing this post (plus I know you all like screenshots). For my example, I have a bunch of PowerPoint documents about SharePoint 2013 on my site. I’ve created a custom entity extraction dictionary that will help me refine these documents by feature in SharePoint (i.e.: Apps, Business Intelligence, Search, Social, etc).

Before you build your dictionary file though, you need to decide on what type of extraction to use. You can choose whether you want it to match a word or just part of it. You can also specify whether it needs to match exactly. The MSDN article summarizes this well, but here’s a quick recap of your choices.

Word Extraction – case-insentive, word has to match, limited to 5 dictionaries
Word Part Extraction – case-insensitive, only part of a word has to match, limited to 5 dictionaries
Word Exact Extraction – case-sensitive, word has to match, limited to 1 dictionary
Word Part Exact Extraction – case-sensitive, only part of a word has to match, limited to 1 dictionary

For today’s example, I am going to use Word Part Extraction as it gives you some of the most flexibility. You might want to use Word Exact Extraction for extracting IDs from a document. For example, you could use it to extract part numbers or invoice numbers. Those aren’t something you would typically want to refine on but you may want to query on them. FAST Search for SharePoint only provided case-sensitive extraction. This made the feature less useful as you had to accommodate all possible varieties of case combinations in your dictionary file.

To get started, we need to create a dictionary file. I started with notepad and I also edited with Excel some. The format is simple “Key,Display form”. The key is what the crawler matches in the body of the document and the Display form is what gets displayed in the refiner. Although the Display form is optional, you want to include it as it allows you to control the exact way the text looks in the refiner (including case sensitivity). Here’s the dictionary file I created.

Key,Display form
bcs,Business Connectivity Services
Business Connectivity Services,Business Connectivity Services
wcm,Web Content Management
Web Content Management,Web Content Management
bi,Business Intelligence
Business Intelligence,Business Intelligence
ecm,Electronic Content Management
Electronic Content Management,Electronic Content Management
Apps,Apps
Analytics,Analytics
Development,Development
Service Application,Service Application
Excel Services,Excel Services
Office Web Apps,Office Web Apps
owa,Office Web Apps
wac,Office Web Apps
PerformancePoint,Business Intelligence
Search,Search
Social,Social
My Sites,My Sites
Communities,Social
Visio Services,Visio Services
Workflow,Workflow

The next step is to use the PowerShell command, Import-SPEnterpriseSearchCustomExtractionDictionary. The documentation on this page is correct which is how I found the solution to my problem. The first step is to get a reference to the Search Service Application using Get-SPEnterpriseSearchServiceApplication. Then for our import command, we pass the service application, the Filename, and the DictionaryName. Here’s where the complexity comes in. The Filename requires a UNC path. That means something like \\servername\path\file.csv. It won’t take a relative path. The next tricky part is where I found an issue in the documentation. I wanted to use the Word Part extraction, it has the case of the word “WordPart” incorrect in the dictionary name. If you do specify the DictionaryName wrong, it will not work. Here are the list of valid values which correspond to the extraction types we talked about above.

Microsoft.UserDictionaries.EntityExtraction.Custom.Word.n [where n = 1,2,3,4 or 5]
Microsoft.UserDictionaries.EntityExtraction.Custom.ExactWord.1
Microsoft.UserDictionaries.EntityExtraction.Custom.WordPart.n [where n = 1,2,3,4 or 5]
Microsoft.UserDictionaries.EntityExtraction.Custom.ExactWordPart.1

For the non-exact entries, you need to specify a value at the end (1 – 5) which specifies which extraction dictionary it uses. In the case below, I did number 2 since I already messed up number 1 by specifying the dictionary name incorrectly. :)

$searchApp = Get-SPEnterpriseSearchServiceApplication

Import-SPEnterpriseSearchCustomExtractionDictionary –SearchApplication $searchApp –Filename \\server\c$\folder\WordPartExtraction.csv –DictionaryName Microsoft.UserDictionaries.EntityExtraction.Custom.WordPart.2

When you run the commands, it will look like this. I put the above in a script file for convenience.

The next part I believe is a little different than the way you did it with FAST Search for SharePoint. What you need to do is tell search which managed properties to perform entity extraction. Effectively, what we are doing here is telling search to look at the body of the document and see if anything in there matches items in the dictionary. To do this, go to your Search Service Application –> Search Schema and edit the property named body.

Scroll to the bottom and you will see the Custom entity extraction section. Now, we just need to check the box next to the dictionary we want to use. In my case, it is Word Part Extraction – Custom2. If you have multiple dictionaries you may select them here. Save the managed property after you make your selection.

After you save, you need to start a full crawl on your content source, typically Local SharePoint sites.

When this finishes, we can now begin to use our new extracted entities in our search center. The first thing we want to do is add this new extraction to the Refinement web part so that users can refine results by SharePoint feature. Go to your Search Center and issue a query that is going to get you some results that you know you can refine on. In my case, I typed the word SharePoint since I knew all of my documents were related to that. Once you get search results, edit the page. Then on the left side, edit the Refinement web part. Now, click the Choose Refiners button.

This page will list all managed properties marked for refinement. What’s really nice is that when you click on a managed property, it gives you a preview of the available refiners.

In the list of available refiners, I select WordPartCustomRefiner2. Below, you see a list of Sample values and the number of times it found a match. To configure the property, select it and click the Add button. I also added this new refiner up to the top since I want users to see it first.

Now, I gave it a Display name and you have the option to choose a different Display template as well as how it is sorted. You can actually customize how the refiner is rendered on the page with the display template too but that’s outside of the scope of today’s post. At the bottom of the page there is also a Preview button. Click on it and you will see what your search results look like before you even save the page.

If you’re happy with the way the refiners look, save the page, check it in, and publish it. Now issue a query. I am going to use the same one from before.

As you can see, we now see a list of the SharePoint features that we specified in our dictionary. Let’s try a few refiners. Let’s start by selecting Electronic Content Management. I do know that should it read Enterprise, but it was after midnight when I started working on this. :)

Notice, that in the results, that Word Document 3 comes back and you see “ECM” in it. That’s the only text in the document and it matched it to that feature since we defined it in the dictionary. Now let’s try Business Connectivity Services.

It returns several slide decks this time because it mentions it in the Intro deck, the BCS deck, UPS, etc. I’m not sue how Visio Services made a hit to it, but maybe there is something new I need to learn. :)

You can also select multiple refiners. In this case, I selected Apps and Web Content Management.

By now, I am hoping you are seeing the power of entity extraction. It gives you a new level of classification on documents and the users never had to tag anything manually at all.

You can issue queries with these dictionaries as well too. For example, if I want to search for anything tagged with Apps, I would issue the following query.

WordPartCustomRefiner2:"Apps"

I hope after reading this post, it has got you started thinking. Most organizations have next to no metadata on their documents. This isn’t a replacement for taking the time to classify your documents properly with site columns and content types. Inferring metadata will never be as good as the users taking the time to classify the documents themselves. However, it is a great stop-gap in dealing with all of that untagged content you have.

I think this is one of the most significant feature in SharePoint 2013 that most people will probably never use. :( Take the time to set up a few dictionaries and you are going to get immediate value out of search and your users are going to notice.

List of common SharePoint 2013 Result Source IDs

CoreyRoth — Fri, 04 Jan 2013 16:08:06 GMT

Although the new Result Source feature in SharePoint 2013 is really cool, writing queries against them is not. With scopes you could specify them by name (i.e.: All Sites or People). However with result sources, you must specify the id not the name. This cane make it tricky to develop custom coded solutions. Luckily, it turns out that the IDs of the built-in result sources are all the same (at least they appear to be) with the exception of the Conversations result source. In comparing an on-premises installation of SharePoint 2013 with a SharePoint Online Preview tenant, here is the list I came up with. Hopefully, it will save you some time when you need to write a query.

Result Source	ID
Documents	e7ec8cee-ded8-43c9-beb5-436b54b31e84
Items matching a content type	5dc9f503-801e-4ced-8a2c-5d1237132419
Items matching a tag	e1327b9c-2b8c-4b23-99c9-3730cb29c3f7
Items related to current user	48fec42e-4a92-48ce-8363-c2703a40e67d
Items with same keyword as this item	5c069288-1d17-454a-8ac6-9c642a065f48
Local People Results	b09a7990-05ea-4af9-81ef-edfab16c4e31
Local Reports And Data Results	203fba36-2763-4060-9931-911ac8c0583b
Local SharePoint Results	8413cd39-2156-4e00-b54d-11efd9abdb89
Local Video Results	78b793ce-7956-4669-aa3b-451fc5defebf
Pages	5e34578e-4d08-4edc-8bf3-002acf3cdbcc
Pictures	38403c8c-3975-41a8-826e-717f2d41568a
Popular	97c71db1-58ce-4891-8b64-585bc2326c12
Recently changed items	ba63bbae-fa9c-42c0-b027-9a878f16557c
Recommended Items	ec675252-14fa-4fbe-84dd-8d098ed74181
Wiki	9479bf85-e257-4318-b5a8-81a180f5faa1

Of course this doesn’t help you much for your own custom result sources, but at least you should be able to work with this list for anything built-in. This is especially useful if you want to write queries to do people search or get popular items.

How to: Set up People Search in SharePoint 2013

CoreyRoth — Thu, 03 Jan 2013 22:30:33 GMT

Looking back at my analytics, I found that my People Search in SharePoint 2010 was quite popular. I thought I would update that post for SharePoint 2013. The process is similar but there are a few things to be aware of. People Search requires you to have the User Profile Service Application up and running as well as My Sites. If you don’t have either of those configured, People Search will not work and you will likely see an error like the following in your crawl log.

Error in the Microsoft SharePoint People Protocol Handler. (Invalid URI: The URI is empty.)

User Profile Synchronization Application

Let’s start with configuring the User Profile Service Application. We need this to retrieve user profiles from Active Directory so that search can use them. If you already have your profiles imported, you can skip this section and proceed to My Sites. The nice thing about SharePoint 2013 is that you can set it up the easy way now with the new (well old from 2007) synchronization option called Use SharePoint Active Directory Import. You can still using the full sync if you want, but for those of you that don’t want to mess with the pain you had with UPS in the past, give this a try. You can always change it later. If you already have profile synchronization working, there is no reason to mess with this setting. To get to this setting, go to your User Profile Service Application –> Synchronization Settings. Choose the option Use SharePoint Active Directory Import and click ok. You’ll get a warning mentioning it’s limitations after that.

Once you are done here, go to Configure Synchronization Connections and Create New Connection. Again, you don’t need to do this if you already have synchronization up and running. Enter the name of your domain, provide an account that has credentials. This account does not need to be a domain admin (nor should it be). Click Populate Containers and select the OUs that you want to import and then save the connection. Back on the UPSA page, go to Start Profile Synchronization and select Start Full Synchronization and click OK.

My Sites

We now need to configure My Sites. Typically this is hosted on a new web application but it doesn’t have to be. Now, you need to create a My Site Host. To do this, create a new site collection on the root path (/). Choose the My Site Host template and create the site collection.

Once you have created this web application and set up it’s site collection, you will need to create some managed paths for it. To do this, select the web application that host My Sites and click Managed Paths in the ribbon.

On this screen, remove the existing Sites managed path. Then add a wildcard inclusion for personal and a explicit inclusion for my.

Now, we need to enable Self Service Site Creation. Return to the Web Applications page and click the Self-Service Site Creation. From here, select On and click Ok.

Search

At this point, you may be good to go. However, there are a few settings that you should verify. First, go to the Service Applications page. Select your User Profile Service Application but do not click on the link. Instead click on the Administrators button. Typically, what you will find here is that when the Search Service Application was created, it adds the Search service account to this list. However, if you change the default content access account (as you should), it will not get updated here automatically. Therefore, you need to add your crawl account to this list and choose the Retrieve People Data for Search Crawlers permission.

Now, we are probably ready to crawl, but it’s worth checking the content source to be sure. Go to your Search Application –> Content Sources. Edit your default content source (typically Local SharePoint sites). Ensure that there is an entry with the sps3 protocol handler (i.e.: sps3://myserver) in the start addresses. You only need one sps3 entry and typically it is just the URL of the first web application you happened to create. It does not have to be the URL of the My Site Host.

Once you have confirmed these URLs are correct, you can start an incremental crawl. When it finishes, you can go to you will be able to go to your search center and search for people in your organization. If it doesn’t work, return to your Search Service Application and view your crawl log. There usually is something there to help indicate what the problem is.

How to: Sort search queries in SharePoint 2013

CoreyRoth — Thu, 03 Jan 2013 20:58:54 GMT

Sorting search results has always been an interesting subject in SharePoint. It was actually harder to do than you would think. In SharePoint 2007, all you had was relevance and modified date (write). The KeywordQuery class did not have anything to allow you to sort. You could get around this though by using SQL Syntax though but it was not ideal. As a reminder, you should stop using SQL Syntax now. In SharePoint 2010, Microsoft added a SortList property to the KeywordQuery class however, if you tried to use it, you would likely get the error: Exception from HRESULT: 0x80040E60. This is because, code was added to prevent SharePoint 2010 to sort on anything but relevance and modified date. The only way to get sortable search results was to add FAST Search for SharePoint (FS4SP). That was unfortunate as many people wanted to sort but did not want to make that kind of investment.

Fast forward to today, and now we can take advantage of the SortList property and it really works. Let’s build off of my previous KeywordQuery post. All we need to do is add items to the read-only SortList collection. For example, to sort by author, we would add the following line before executing the query with SearchExecutor.

keywordQuery.SortList.Add("Author", SortDirection.Ascending);

Now the results are sorted by author.

We could also sort in descending order as well. For example to show files largest to smallest, use this snippet.

keywordQuery.SortList.Add("Size", SortDirection.Descending);

Here are the results:

These can be combined as well.

keywordQuery.SortList.Add("Author", SortDirection.Ascending);
keywordQuery.SortList.Add("Size", SortDirection.Descending);

The results then look like:

We can invoke this functionality when using the Search REST API as well. To do this, we use the querytext parameter to issue our query just like before, but we add the sortlist parameter. The syntax of the parameter is “manageproperty:sortorder”. For sort order you must specify ascending or descending. Be sure to include the values in single quotes. We delimit multiple sort properties using a comma (,). Using the above sort example, here is what the query string would look like.

http://server/_api/search/query?querytext='SharePoint'&sortlist='author:ascending,size:descending'&selectproperties='Title,Author,Size,Path,Write'

From the screenshot, you can see the PowerPoint Presentation is also the top result. I’ve also included a few selectproperties to make the results more legible.

I’m really excited that all users of SharePoint search now have this capability. It really adds a lot of value to the solutions that we can deliver. The last thing I will mention is that you must have the managed property you are searching on marked as sortable. I’ll talk about new features of managed properties in a post pretty soon.

How to: Use the SharePoint 2013 Search KeywordQuery Class

CoreyRoth — Thu, 03 Jan 2013 17:42:00 GMT

Over the past few versions of SharePoint, I have provided a number of blog posts on how to query search. Of those, my posts on how to use the KeywordQuery class using KQL are some of the most popular (2010 and 2007). In continuing that tradition, I am proud to present the “How to” post on using the KeywordQuery class in SharePoint 2013. The good news: your old KeywordQuery code should still work. The bad news: the way you did it is marked completely obsolete. I actually had to dig through quite a bit of documentation to find the right API that wasn’t obsolete. One thing that always provided confusion was that there were similar classes named KeywordQuery in both Microsoft.Search.Query and Microsoft.Office.Server.Search.Query. To reduce some of that confusion, the KeywordQuery class in Microsoft.Search.Query is now marked as obsolete. The new updated class is in Microsoft.Office.Search.Query. Microsoft may have killed the word Office from the name of the product in 2010, but it’s alive and well in the API.

Now let’s talk about what we need to execute some queries with our code. The code here today will work well from farm solutions in SharePoint as well as other .NET applications hosted on the same SharePoint server. We’ll be building a console application today. If you want to query remotely using the object model, then you will need to use the Search Client Object Model which I covered previously. Create a new console application, and then you will need to add a few references. These can be found in the 15 hive in the ISAPI folder. Add the following:

Microsoft.Office.Server
Microsoft.Office.Server.Search
Microsoft.SharePoint
Microsoft.SharePoint.Security

The classes we need are in the search assembly but you’ll get lots of errors about dependencies missing if you don’t include the others. We then need the following references. This gives us what we need for search, SharePoint, and for the underlying datatable object that is available after we query.

using System.Data;
using Microsoft.SharePoint;
using Microsoft.Office.Server.Search.Query;

Once we have this, we’re ready to get started. There are many ways to instantiate a KeywordQuery object. For simplicity, I am just going to pass a SPSite object for one of my site collections. You can also pass a SPWeb or you can use the proxy technique like I used in my SharePoint 2010 post. We’ll first, get our SPSite object.

using (SPSite siteCollection = new SPSite("http://server/sitecollection"))

We then create a KeywordQuery object using that site collection.

KeywordQuery keywordQuery = new KeywordQuery(siteCollection);

Now, we specify our query using the QueryText field. This is where you can use your custom KQL queries. The documentation team updated the post on this so there are a lot of good tips here. I am just going to search for the word SharePoint.

keywordQuery.QueryText = "SharePoint";

In the past, we would then set the ResultsProvider and ResultTypes that we want, but now we don’t have to. In fact, this whole process takes considerably less code. Instead we use the new SearchExecutor class that I first mentioned in my Client OM post.

SearchExecutor searchExecutor = new SearchExecutor();

We then use it’s ExecuteQuery method passing it our KeywordQuery class.

ResultTableCollection resultTableCollection = searchExecutor.ExecuteQuery(keywordQuery);

Assuming it executes correctly, we can then get our search results. Before, we used to use an indexer using ResultTypes.ReleventResults to get the table of results we need. The indexer has been made obsolete, so now we must use the new Filter method. I figured out what it required from the obsolete descriptor of the indexer. For the first value, it needs a string with a value of TableType and since the previous enum is also obsolete, we now use KnownTableTypes.RelevantResults.

var resultTables = resultTableCollection.Filter("TableType", KnownTableTypes.RelevantResults);

It returns an IEnumerable<ResultType>, so we need to filter it. I just FirstOrDefault() to get the table we need.

var resultTable = resultTables.FirstOrDefault();

Previously, we then had to load the data into a DataTable which took a few more lines of code. However, now that is not required due to the new Table property.

DataTable dataTable = resultTable.Table;

Now that you have a DataTable, you can bind it or query it as you see. You can also look at the results using the visualizer in Visual Studio.

Here is the entire code snippet.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
 
using System.Data;
using Microsoft.SharePoint;
using Microsoft.Office.Server.Search.Query;
 
namespace SearchConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            using (SPSite siteCollection = new SPSite("http://server/sitecollection"))
            {
                KeywordQuery keywordQuery = new KeywordQuery(siteCollection);
                keywordQuery.QueryText = "SharePoint";
 
                SearchExecutor searchExecutor = new SearchExecutor();
                ResultTableCollection resultTableCollection = searchExecutor.ExecuteQuery(keywordQuery);
                var resultTables = resultTableCollection.Filter("TableType", KnownTableTypes.RelevantResults);
 
                var resultTable = resultTables.FirstOrDefault();
 
                DataTable dataTable = resultTable.Table;
            }
        }
    }
}

So if you have a lot of search based code, you may need to do some updates at some point. However, I think these are good changes and simplify the process a little bit. In an upcoming post, I’ll cover some of the new things you can do with the KeywordQuery class.

How to: Seed Search Suggestions in SharePoint 2013 Preview

CoreyRoth — Tue, 18 Sep 2012 20:17:53 GMT

I was digging through some new PowerShell cmdlets for one of my talks coming up at SharePoint Conference 2012 (SPC12), and I stumbled upon the new cmdlet Import-SPEnterpriseSearchPopularQueries. I found this cmdlet to be interesting since it can be used to easily seed what shows up in the search suggestions on a given site. I took the example there and did some tweaking and thought I would share my results.

We first, need to get a reference to the Search Service Application Proxy using Get-SPEnterpriseSearchServiceApplicationProxy. Sorry, the cmdlets for search a bit wordy so that is what the Tab key is for. :)

$ssap = Get-SPEnterpriseSearchServiceApplicationProxy

Search suggestions are actually specific to the result source and site that hosts the search box. This means you could have different suggestions depending on which part of the site you are on. To achieve that, we need to get a reference to the web hosting the search center and ultimately a result source object. Let’s get the web first. Change the URL to match your server’s search center.

$searchWeb = Get-SPWeb -Identity http://myserver/search

Now, we will use the Federation Manager to get a result source. However, we need a Search Object Owner first. The example on TechNet requires a lot more typing. I luckily found the cmdlet Get-SPSearchObjectOwner. It takes a scope (Level) and an SPWeb object.

$owner = Get-SPEnterpriseSearchOwner -Level SPWeb -SPWeb $searchWeb

Unfortunately, I couldn’t find a command to directly get a result source so that means we actually have to go through the API to get a Federation Manager first.

$mgr = new-object Microsoft.Office.Server.Search.Administration.Query.FederationManager -ArgumentList $ssap

Once we have the manager object, we use the GetSourceByName method passing it the name of the result source and $owner. In this case, we use “Local SharePoint Results" as the result source name.

$source = $mgr.GetSourceByName("Local SharePoint Results", $owner)

As far as the query suggestions themselves, we will need to create a CSV file with the data. The format is simple: Query Text,Query Count,Click Count,LCID. For example:

SharePoint Conference,500,300,1033

Setting these values allows you to control the order in which items appear in the query suggestions. For suggestions to appear in the search box, the click count must be at least 5. 1033 is the LCID value for English. Here is what my file looks like:

Now, we run the Import-SPEnterpriseSearchPopularQueries cmdlet. It needs the SearchApplicationProxy, Filename, and ResultSource parameters which we retrieved with the previous lines of script.

Import-SPEnterpriseSearchPopularQueries -SearchApplicationProxy $ssap -Filename .\suggestions.csv -ResultSource $source -Web $searchWeb

If all commands execute successfully, you should not see anything returned at the command prompt. Now, you can begin using your suggestions right away. You do not need to recrawl. Go to the search center that you specified in your $searchWeb object. Start typing some of the values from your suggestions and you will see them beneath the search box.

This new PowerShell cmdlet makes it so easy to seed your search suggestions. I think I will start recommending this on new deployments to really improve the immediate value of search. Try it out today.

How to: Query Search with the SharePoint 2013 Client Object Model

CoreyRoth — Tue, 11 Sep 2012 04:41:09 GMT

When I first gave the Preview of Search in SharePoint 2013, I mentioned that we could now query search using the Client Object Model. After enough digging through the documentation, I finally pieced together the steps required to get search results back using this new API. Today, I will demonstrate how to query from a console application. To get started, we first need to add referenced to the appropriate assemblies. These assemblies can be found in the ISAPI folder of the 15 hive.

Microsoft.SharePoint.Client.dll
Microsoft.SharePoint.Client.Runtime.dll
Microsoft.SharePoint.Client.Search.dll

Now in our console application, we need to add the necessary references.

using Microsoft.SharePoint.Client;
using Microsoft.SharePoint.Client.Search;
using Microsoft.SharePoint.Client.Search.Query;

This next step should be familiar if you have used the Client Object Model before in SharePoint 2010. Start by creating a ClientContext object and pass in the URL to a site. Put this in a using block.

using (ClientContext clientContext = new ClientContext("http://servername"))

We then need to create a KeywordQuery class to describe the query. This class is similar to the server side KeywordQuery class but there are some differences. We pass the ClientContext into the constructor.

KeywordQuery keywordQuery = new KeywordQuery(clientContext);

To set the query use the QueryText property. In this case, I am doing a search for the keyword “SharePoint”.

keywordQuery.QueryText = "SharePoint";

Unlike the server object model, with the Client OM we have to use another class, SearchExecutor, to send the queries to the search engine. We pass a ClientContext to it as well:

SearchExecutor searchExecutor = new SearchExecutor(clientContext);

To execute the query, we use the ExecuteQuery method. It returns a type of ClientResult<ResultTableCollection>.

ClientResult<ResultTableCollection> results = searchExecutor.ExecuteQuery(keywordQuery);

However, the query doesn’t actually execute until you call ExecuteQuery() on the ClientContext object. If you have done a lot of Client OM work before, you might think you need to call Load() first but it is not required.

clientContext.ExecuteQuery();

Assuming no exception occurs, you can now iterate through the results. The ClientResult<ResultTableCollection> will have a property called Value. You will want to check the zero index of it to get the search results. From there, the ResultRows collection has the data of each row. This object is simply a dictionary object that you can use an indexer with and pass the name of the managed property. In the example below, I write out the Title, Path, and Modified Date (Write).

foreach (var resultRow in results.Value[0].ResultRows)
{
    Console.WriteLine("{0}: {1} ({2})", resultRow["Title"], resultRow["Path"], resultRow["Write"]);
}

That’s it. Here is what the entire class looks like together.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
 
using Microsoft.SharePoint.Client;
using Microsoft.SharePoint.Client.Search;
using Microsoft.SharePoint.Client.Search.Query;
 
namespace SearchClientObjectModelTest
{
    class Program
    {
        static void Main(string[] args)
        {
            using (ClientContext clientContext = new ClientContext("http://sp2010"))
            {
                KeywordQuery keywordQuery = new KeywordQuery(clientContext);
                keywordQuery.QueryText = "SharePoint";
 
                SearchExecutor searchExecutor = new SearchExecutor(clientContext);
 
                ClientResult<ResultTableCollection> results = searchExecutor.ExecuteQuery(keywordQuery);
                clientContext.ExecuteQuery();
 
                foreach (var resultRow in results.Value[0].ResultRows)
                {
                    Console.WriteLine("{0}: {1} ({2})", resultRow["Title"], resultRow["Path"], resultRow["Write"]);
                }
 
                Console.ReadLine();
            }
        }
    }
}

As you can see, you can get started with Search using the Client Object Model with relatively few lines of code. This certainly beats, constructing that nasty XML file that the Search Web Service takes.

I cross-posted the Visual Studio Solution to code.msdn.microsoft.com.

Follow me on twitter: @coreyroth.

Getting Started with the Search REST API in SharePoint 2013 Preview

CoreyRoth — Fri, 07 Sep 2012 15:33:19 GMT

We have a brand new REST based API for interacting with Search in SharePoint 2013 Preview. Unfortunately, there is not a lot of information out there yet on how to interact with it. I have pieced together a couple of bits to help get you started. If you are looking to bring Search into an HTML / JavaScript based application, you’ll be able to get started right away. The first piece of the puzzle is putting together the right URL to access Search via REST. To do this, we start with the server name and append /_api/search/query. Just typing this into the browser will start giving you useful information. For example:

http://server/_api/search/query

If you look towards the bottom, you will see the Key SerializedQuery. This actually shows you the parameters passed into the request. It also provides you a hint as to what type of parameters you can pass to the REST interface. The one that might immediately stand out is QueryText. This is the actual search query. Think of it as the equivalent of what you type into the search box. There are a few other parameters that you can customize here as well such as the SummaryLength, MaxSnippetLength. and more. Let’s try adding QueryText to our URL and passing in a value.

http://server/_api/search/query?querytext=SharePoint

It turns out the value always has to be enclosed in single quotes (‘) like this:

http://server/_api/search/query?querytext=’SharePoint’

Now, we get some results:

The search results themselves are included in the d:RelevantResults element. It contains an element d:Table which has Bare with me as I am going to post the XML for one of the results so that we can inspect it further.

<d:element m:type="SP.SimpleDataRow">
  <d:Cells>
    <d:element m:type="SP.KeyValue">
      <d:Key>Rank</d:Key>
      <d:Value>20.268159866333</d:Value>
      <d:ValueType>Edm.Double</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>DocId</d:Key>
      <d:Value>31</d:Value>
      <d:ValueType>Edm.Int64</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>WorkId</d:Key>
      <d:Value>31</d:Value>
      <d:ValueType>Edm.Int64</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Title</d:Key>
      <d:Value>Getting the most out of SharePoint Search</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Author</d:Key>
      <d:Value>Corey Roth;SharePoint Setup</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Size</d:Key>
      <d:Value>2425342</d:Value>
      <d:ValueType>Edm.Int64</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Path</d:Key>
      <d:Value>http://sp2010/Shared Documents/Getting the most out of SharePoint Search - SharePoint Saturday Denver.pptx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Description</d:Key>
      <d:Value>
        Template: Sam Moore; Silver Fox Productions, Inc.&#xD;
        Formatting:&#xD;
        Event Date: October 3-6th, 2011&#xD;
        Event Location: Anaheim, CA&#xD;
        Audience Type: Internal/External
      </d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Write</d:Key>
      <d:Value>2011-11-10T00:00:00.0000000</d:Value>
      <d:ValueType>Edm.DateTime</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>CollapsingStatus</d:Key>
      <d:Value>0</d:Value>
      <d:ValueType>Edm.Int64</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>HitHighlightedSummary</d:Key>
      <d:Value xml:space="preserve">Passed all &lt;c0&gt;SharePoint&lt;/c0&gt; 2010 certification exams &lt;ddd/&gt; Introduce Search in &lt;c0&gt;SharePoint&lt;/c0&gt; Online &lt;ddd/&gt; The talk will be slides with quite a few demos of &lt;c0&gt;SharePoint&lt;/c0&gt; Online features &lt;ddd/&gt; </d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>HitHighlightedProperties</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>contentclass</d:Key>
      <d:Value>STS_ListItem_DocumentLibrary</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PictureThumbnailURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ServerRedirectedURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ServerRedirectedEmbedURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ServerRedirectedPreviewURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>FileExtension</d:Key>
      <d:Value>pptx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ContentTypeId</d:Key>
      <d:Value>0x010100B1CF83A7F86DFC468C975FFF33126DFD</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ParentLink</d:Key>
      <d:Value>http://sp2010/Shared Documents/Forms/AllItems.aspx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ViewsLifeTime</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ViewsRecent</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>SectionNames</d:Key>
      <d:Value xml:space="preserve">Getting the most out of SharePoint Search&#xD;
;Getting the most out of SharePoint Search&#xD;
;Corey Roth&#xD;
About me&#xD;
;Key Theme&#xD;
;SharePoint is not aâ€¦&#xD;
;SharePoint is aâ€¦&#xD;
;Challenges with Search&#xD;
;Agenda&#xD;
;Key Takeaways&#xD;
;You have a choiceâ€¦&#xD;
;What can search index?&#xD;
;Search in SharePoint&#xD;
;Components of Search&#xD;
;Finding things easierâ€¦&#xD;
;Improve search with queries&#xD;
;Built-in Managed Properties&#xD;
;Segmenting results with Scopes&#xD;
;Querying Search and Scopes&#xD;
;Provide more search options&#xD;
;Many options for displaying scopes&#xD;
</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>SectionIndexes</d:Key>
      <d:Value>1;2;3;4;5;6;7;8;9;10;11;12;13;14;15;16;17;18;19;20</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>SiteLogo</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>SiteDescription</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>deeplinks</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>importance</d:Key>
      <d:Value>0</d:Value>
      <d:ValueType>Edm.Int64</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>SiteName</d:Key>
      <d:Value>http://sp2010</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>IsDocument</d:Key>
      <d:Value>true</d:Value>
      <d:ValueType>Edm.Boolean</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>LastModifiedTime</d:Key>
      <d:Value>2011-11-10T00:00:00.0000000</d:Value>
      <d:ValueType>Edm.DateTime</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>FileType</d:Key>
      <d:Value>pptx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>IsContainer</d:Key>
      <d:Value>false</d:Value>
      <d:ValueType>Edm.Boolean</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>WebTemplate</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>SecondaryFileExtension</d:Key>
      <d:Value>pptx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>OriginalPath</d:Key>
      <d:Value>http://sp2010/Shared Documents/Getting the most out of SharePoint Search - SharePoint Saturday Denver.pptx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>DisplayAuthor</d:Key>
      <d:Value>Corey Roth;SharePoint Setup</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Created</d:Key>
      <d:Value>2012-07-26T23:21:29.0000000</d:Value>
      <d:ValueType>Edm.DateTime</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PeopleInMedia</d:Key>
      <d:Value></d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>MediaDuration</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PictureWidth</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PictureHeight</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>DefaultEncodingURL</d:Key>
      <d:Value>http://sp2010/Shared%20Documents/Getting%20the%20most%20out%20of%20SharePoint%20Search%20-%20SharePoint%20Saturday%20Denver.pptx</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ExternalMediaURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>UserEncodingURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Community_MembersCount</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Community_RepliesCount</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Community_TopicsCount</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ReplyCount</d:Key>
      <d:Value>0</d:Value>
      <d:ValueType>Edm.Int64</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>LikesCount</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Discussion_BestAnswerID</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Discussion_Category</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Discussion_Post</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Discussion_LikesCount</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ListItemID</d:Key>
      <d:Value>6</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>FullPostBody</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PostAuthor</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>RootPostOwnerID</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>RootPostID</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>AttachmentType</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>AttachmentURI</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>MicroBlogType</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ModifiedBy</d:Key>
      <d:Value>
        Corey Roth
 
        SharePoint Setup
      </d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>RootPostUniqueID</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>Tags</d:Key>
      <d:Value></d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ImageDateCreated</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PictureURL</d:Key>
      <d:Value m:null="true" />
      <d:ValueType>Null</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ResultTypeIdList</d:Key>
      <d:Value>2;6</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>PartitionId</d:Key>
      <d:Value>0c37852b-34d0-418e-91c6-2ac25af4be5b</d:Value>
      <d:ValueType>Edm.Guid</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>UrlZone</d:Key>
      <d:Value>0</d:Value>
      <d:ValueType>Edm.Int32</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>AAMEnabledManagedProperties</d:Key>
      <d:Value>HierarchyUrl;OrgParentUrls;OrgUrls;OriginalPath;Path;recommendedfor;SitePath</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>ResultTypeId</d:Key>
      <d:Value>2</d:Value>
      <d:ValueType>Edm.Int32</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>RenderTemplateId</d:Key>
      <d:Value>~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_PowerPoint.js</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
    <d:element m:type="SP.KeyValue">
      <d:Key>piSearchResultId</d:Key>
      <d:Value>1_1</d:Value>
      <d:ValueType>Edm.String</d:ValueType>
    </d:element>
  </d:Cells>
</d:element>

As you can see, we now get a ton of information back from Search for a given search result. Let me point out some of the more interesting values. We have the usual values such as Title, Path, Author, Write (Modified Date), and HitHighlightedSummary. However, we also get a ton of new information. For images, you will get a PictureThumbnailUrl to allow you to easily show a thumbnail. This particular result is a PowerPoint and search understands the individual sections inside it now which you will see in SectionNames. SiteName provides the site hosting the item. Tags will provide any use of the Enterprise Keywords field. You also get a heap of social information about the item as well such as how many likes. Lastly, the RenderTemplateId tells you which template the result source is using to display it on the search results page. This is just one example, but you can see we get a lot of useful information about the search.

Looking at the XML in the browser is a great way to get familiar with what the API has to offer. When you are ready to start developing, it is very easy to take advantage of the REST API using JavaScript. You can do this with the $.ajax() method in jQuery and pass in the URL you used in the browser. You want to use the GET method and pass in a header telling it you want the results back as json. They have a good example of how this works on code.msdn.microsoft.com. Specifically, take a look at App.js in the example to see the code in action. It’s pretty simple.

If you want to take advantage of the REST API from C# or another .NET language, that is another story though. I haven’t been able to add a service reference for the REST API. All of the server side code examples I have seen call the REST API using HttpWebRequest which requires no less than 60 lines of code or so. I am hoping there is a better way but I haven’t seen one yet. If you have had better luck be sure and let me know.