As we start out the new year, I thought I would start out with a post talking about one of my favorite SharePoint topics, Enterprise Search. Over the years, I’ve seen a variety of search configurations. Some companies have great implementations and other companies really struggle. For those of you that struggle with search, hopefully this post will get you thinking in the right direction to get search to meet your organization’s needs. We’re going to look at how you should configure your search environment, what types of content should you index, as well as how you can take advantage of search outside of SharePoint. This post won’t be the “silver bullet” in fixing search in your company, but maybe it will help you justify a search project or two.
Improving Search Results
When interviewing clients, the number one complaint about search is usually summed up by a statement like this: “I can’t find anything!”. Whether you are using SharePoint Foundation with Search Server Express, SharePoint Server 2010, or FAST Search for SharePoint, the quality of the results you receive from search is only as good as the input you give it. What I mean by this is that you can’t just install SharePoint, turn on search, and call it good. Out-of-the-box, SharePoint gives you a very functional search engine, but you need to do some work to really enable the full power it provides. This all starts with proper planning. If you don’t have a governance plan in place, you should probably start here. As you may know this living document specifies how people use your SharePoint environment and what they can and can not do. I’ve made it a habit to start including a topic in on search in the governance plans that I write as it is critical to the adoption and success of SharePoint at a company. Of course, I could write an entire series of blog posts talking about governance but there is already plenty of material out there for you to leverage if you are interested in learning more. If you don’t have a governance plan in place already, it really can be a big project, but it’s worth the time and effort.
Is your SharePoint environment really nothing more than a glorified file share? Unfortunately, a lot of times, companies implement SharePoint, but they don’t create an ECM plan which means no metadata gets assigned to documents. In reality, this is the primary cause of why users complain that they cannot find documents. Metadata, also known as data that describes data, provides additional information about a document. Out-of-the-box, some basic metadata already exists on your default Document content type such as title, author, and date. However, this only provides limited information for search. Search really starts to light up when you get the user to add additional information about a document. This could include things like Department, Division, Document Type (i.e.: Contract, Schematic, or Invoice), Invoice Number, Category. Why would you want to add this extra information to a document? So that your users can query and refine search results with the values from these columns. Let’s discuss some examples.
If we added a site column to our documents called Document Type, I could then execute a query to return all documents that were contracts. In this case, it could return documents that were truly contracts and not any document it happens to find with the word contract in it somewhere such as a PowerPoint presentation. Do you understand the difference? It allows us to return the exact type of documents the users desire. We could then take this a step further and say show me all contracts in the Gulf Coast division. Or maybe if we had a Contract Date column on our documents, we could say show me all contracts for the year 2009.
Let’s look at another example. Say you have a backend system with product information. It’s a complex database and each product has its own unique id. For each product you have multiple documents such as design schematics, technical documents, and marketing materials. In a system without metadata your best bet would be to search by the product name. You would probably get the results you want, but you would probably get a bunch of extra results too. However, if your users stored that Product Id on each document, you could then run queries and get all of the relevant documents for that specific document. Then after the user completed the query by Product Id, he or she could further refine those results by Document Type or Department perhaps.
Think about your SharePoint environment. Can you query search like this? If you can, that’s great! You can start looking at what else SharePoint Search can do for you. If you can’t, then you should start thinking about how you can get there. If you already have some metadata in place on your documents, but just can’t do queries like this, then you are really close! It’s just a matter of configuring search to use those site columns to create a great experience. We’ll talk about that more shortly. However, if you are like the bulk of many SharePoint environments, you have some work to do. I mentioned an ECM plan earlier. In this document, you plan out the many content types and site columns (metadata) that you will need for your documents. The user will see these additional fields after he or she uploads a new document. Depending on the size of your company, this can take quite some time to develop and implement. This usually consists of a series of meetings with the various departments across your organization to gather requirements on what each one needs. Many times, companies do this incrementally, just tackling a few departments at a time and then moving onto the next. Once you finish gathering requirements, you implement new site columns and content types in SharePoint. It may seem like a long process, but it will pay off in the end.
“But, wait! My users will complain if they have to fill out additional information every time they upload a new document.” Probably, true. However, do they complain more that they can’t find the documents they are looking for? Justify the additional effort when uploading with the amount of effort your users will save when trying to find the documents later. If your users can find the document they need with just 2 – 3 clicks instead of 20, you are already saving time. You can also make use of SharePoint 2010 features such as the Content Organizer and Default Column Values which can help prepopulate values in the form for the user. For more advanced scenarios or if you are still using SharePoint 2007, you can write event receivers which can automatically set values for users as well.
Along with your own custom metadata fields, I also recommend adding Enterprise Keywords and the Ratings field to your document libraries if you are using SharePoint 2010. The Enterprise Keywords field allows your users to “tag” documents when they upload them. The Managed Metadata service keeps track of commonly used tags and prompts the user with existing terms as he or she types it in. These tags also populate the Tag Cloud Web Part and can refine search results. The Ratings field allows users to rate documents with 0 – 5 stars. As more people rate documents, it allows other users to quickly see which documents people find more valuable.
When you have a Governance and ECM plan in place, you can move on to developing a plan for Search. Your Search plan might overlap some with your Governance plan, but will also include which site columns you want to use with Enterprise Search. When Search crawls your content for the first time, it creates something called a Crawled Property which maps to your site columns. In the case of SharePoint content, your site column will get prefixed with ows_ and the spaces in your column name will get encoded. For example, a site column called, Product Id, would become ows_Product_x0020_Id. In my article on Naming Conventions, I mention this is one reason why I am not a fan of spaces in the names of site columns. Whether you want to query by, refine by, or see your site column in search results, you will need to map it to what SharePoint calls a Managed Property. You can map one or more crawled properties to a given managed property to make it available to search. In your Search Plan, you should document all of the site columns that you plan to map to managed properties so that you can go back and reference it in the future or recreate the mappings in another environment. Typically, most people create their managed properties in the Service Application UI. However, you can also create them using PowerShell. When you finish creating your managed properties, you must crawl again to make them active.
With your managed properties in place, you then need to decide how you want to use them. Typically this involves customizing the web parts inside the Search Center. You can do many customizations easily without having to write any code. By editing the results.aspx page, you can customize how search results look as well as your refinement options. If you want to display additional metadata in your search results such as Document Type or Invoice Number, you can edit the XSLT inside the CoreResultsWebPart. If you want to let users refine search results based upon your own metadata (i.e.: department or document type), then take a look at modifying the RefinementWebPart. Lastly, many times users want to customize the query before they ever execute it. You can customize the existing Advanced Search Web Part, but it’s quite ugly and not very flexible. If you want to write a little bit of code, then you can make your own user control or Silverlight application to do the same thing. Read up on the customizations that you can do and then document what you plan to implement in your Search plan.
Indexing Content outside of SharePoint
In my experience, only a small percentage of organizations know that Enterprise Search can index content outside of SharePoint. Some know that you can index file shares. However, did you know that SharePoint can index data from custom applications using SQL or WCF services? It can also crawl your public facing web site, lotus notes, and Exchange public folders. If that’s not enough, you can even write custom code to index whatever system you want. Do you have an old mainframe or AS/400 system that has data you want to show in SharePoint? Well if you can write .NET code to get to the system, you can expose its data in SharePoint through the use of Business Connectivity Services. I’ve met some CIOs, that when learning about this feature, want to index every system in their organization. I think that’s great, but you need to plan appropriately or you will overwhelm your users with search results. The last thing your user wants to see is some file that is 10 years old on your file share when they are looking for this year’s holiday schedule.
One way to allow a user to restrict a query to a subset of the index is with the use of Scopes. Scopes simply allow you to create a set of rules that include and exclude content. Out-of-the-box, SharePoint comes with two scopes All Sites and People. The All Sites scope returns everything in the content index (minus People). When you start adding content sources to index file shares or custom applications, you will want to create a scope or two that allows the user to restrict results to just those areas. For example, this would allow the user to query just SharePoint content or just File Share content. You can also create more advanced scopes using managed properties such as creating a scope that returned only Invoices. Once you create scopes, you can display them in the Search Center as well as in a drop down list on your master page. This gives the user more flexibility on how they search.
In large organizations, finding the right person to ask a question can be challenging. In SharePoint 2007, we gained People Search which allowed us to index information from Active Directory or an ERP system (through the BDC) about the people in an organization. SharePoint 2010 extends this by adding a wealth of social features. It also gives us phonetic searching which makes it easier for users to find people with difficult names to spell. Although, People Search can infer a lot of information from Active Directory and other source systems, it is only as good as the information you provide it. People also have metadata (in their profile) and you should work to have it as complete as possible. Prepopulate what you can for the user with User Profile Synchronization then encourage the people in your organization to complete the rest of their profiles manually especially the Ask Me About section. This section allows other users to know what that user is responsible for (I.e.: Accounts Payable, Benefits, or Employee Recruiting).
I also recommend making sure you have the Manager field set in People Search. By default, People Search gets this from the Manager property in Active Directory. Setting this allows SharePoint to determine your organization chart. If your organization already populates this field, then you can take advantage of the organization chart feature in SharePoint right away.
Before moving on, I recommend taking a look at your existing web applications in your organization along with any ones currently in development. SharePoint can expose the data of Many web applications whether developed in ASP.NET or another language. Before SharePoint, creating a search experience for these applications usually involved a bit of code with the use of things such as T-SQL LIKE statements. Using Business Connectivity Services (BCS), SharePoint can actually crawl your custom application by directly accessing the database tables, through WCF services, or the new BDC Entity Model. Many times you can do this indexing without having to make any changes to your existing application. By using SharePoint Designer 2010, you can connect to your SQL database or WCF services and choose the entities you want to bring into SharePoint as External Content Types. With the External Content Type, you can create a Content Source to index that application.
At this point, you might think “How do the links from the results page go to my custom application?”. This depends on your application. In your External Content Type, you can define a default action which SharePoint uses on the results page when the user clicks on a link. SharePoint can use the data it indexes to help you build a URL (think String.Format()). For example, let’s assume we are indexing a custom ASP.NET application which has product information. The External Content Type contains fields such as Name, ProductId, Description, and Price which comes from a database table. Your custom application is hosted on a server called server and it has a page which allowed the user to edit each product named ProductEdit.aspx. That page takes a query string parameter called Id representing the ProductId in the database table. SharePoint can construct this URL given the information it has to link the user directly to that product editing page. See the picture below for an example.
Sometimes you may have too much content outside of SharePoint or maybe you have a system that already has a capable search engine (i.e.: an ERP or another document management system). This calls for federated search. As another example, you might use a federated search to display results from your public facing web site or from a public search engine like Bing. You don’t want to completely index their data, but you would still like to see results from those external systems when searching from SharePoint. The Federated Search feature allows you to display results from any search engine supporting the OpenSearch 1.1 protocol alongside your local search results. If you are not familiar with OpenSearch, the results come back as an RSS feed. Even if your external system doesn’t support OpenSearch, you can write some code to refactor the results as RSS and integrate them easily into SharePoint. Here is an example, where the federated results come from DotNetMafia.com on the right side of the screen.
Exposing SharePoint Search to other applications
We can index data in other applications, but maybe you don’t want to make the user go to SharePoint to see the search results. Luckily SharePoint has an extensive API that allows you to query SharePoint remotely using web services. Calling the Search Web Service is pretty simple. Effectively you construct an XML input document and then you can call the Query method to return results as XML or the QueryEx method to return results as an ADO.NET DataSet. For a complete example, see my post on how to use the Search Web Service. This also is your stepping stone for developing advanced search driven applications. For example, you could call this web service from Silverlight to present the results using a rich UI.
Chances are the army of ASP.NET developers that you employ don’t know that SharePoint can index their applications. Consider this as an option the next time you build a custom application. There is no point in reinventing the wheel when SharePoint can do the heavy lifting for you. It might even shave off a few hours of your next project. Plus, it allows you to search from multiple sources at the same time. Just think you could get results from SharePoint, a file share, and multiple custom applications in one results screen. With the built-in refinement features, you can drill down into the results you want.
You may be thinking “All of this is great, but I bet I need FAST Search for SharePoint to make it all happen.” That’s not actually the case. Yes, FAST can do a lot of powerful things and allows you to make some great customizations to your search experience. However, most of the things I have talked about today have been around since SharePoint 2007. In fact, with the exception of People Search, you can implement everything in this post today using SharePoint Foundation with Search Server Express. If you use SPF today but don’t use Search Server Express, I highly recommend it. It adds a lot of functionality to out-of-the-box search experience in SPF.
Don’t be afraid to ask for help. This posts covers a lot and sometimes it is nice to have a little guidance along the way when you try to implement. There are many great community resources out there talking about Search including blogs, forums (2007), twitter, and SharePoint Saturdays. I won’t do a shameless plug for my consulting company today, but don’t be afraid to reach out to a firm with some SharePoint consultants to help you develop things like a Governance, ECM, or Search plan. Getting your plan correct before you start will help keep you from having to deal with problems later.
Follow me on twitter.