How to (easily) create your own custom search engine

Processing over 900 billion API requests annually and reaching over 100k developers through free and commercial offerings, the Microsoft Bing APIs are used by businesses around the world to build innovative AI powered solutions.

In this blog post we look at the Bing Custom Search API. We explore how it lets you build ad-free web, image and video search functionality for your own personal domain or collection of domains.

We’ll also see how:

• you can quickly create a search instance of the Bing Custom Search
• you can easily test your search instance through the Custom Search Portal to return Web, Image or Video data
• look at how you can integrate the Custom Search API using C#
• examine the rich data it returns

What is Bing Custom Search?

The Bing Custom Search API forms part of the Microsoft Cognitive Services family of artificial intelligence APIs. Sitting within the Search stack, the Bing Custom Search API lets you build innovative, customisable and tailored search experiences. Using the API, you can build search solutions that return results based on your preferences, block unwanted sites, promote key sites and by using statistics that are generated by the API, you can even get insights which might help grow your business!

Source: Microsoft

Some of the other key features include, but are not limited to:

• Auto spell correction feature to accurately correct spelling and display the right results to your users
• Ability to rename, clone and delete search Instances
• Flexibility to use either an API Endpoint to retrieve JSON response or use a free Hosted UI for your search experience
• Free access keys to explore, tweak, and configure your search instances as you develop
• Trusted indexing and relevance capabilities of Bing

Having access to this type of functionality lets you easily create search experiences which are customised to the things and topics that your users care about.

Bing Search Types

Searches fall into 3 high level areas, so you need to decide which one is the best fit for you:

• Custom Search API
• Web Search API
• Vertical Search APIs

Custom Search API

Lets you fetch website results (website, image and video data) for the slice of web that you define in your Custom Search Instance (and accompanying metadata) – all in one single API call.

Web Search API

This API lets you programmatically retrieve web documents that have been indexed by Bing, and you can then process, or further filter internet data based on your own unique requirements or queries.

Vertical Search APIs

These contain more features, parameters and metadata associated with each respective vertical. For reference, the Vertical Search APIs are categorised into the following areas:

• Bing Web Search API
• Bing Image Search API
• Bing News Search API
• Bing Video Search API
• Bing Entity Search API

Each API accepts their own unique set of parameters, and we’ll explore these other verticals in subsequent blog posts, but for now we will focus on the Custom Search API.

Bing Custom Search Setup

To use the Bing Custom Search API the first thing you need to create is an Instance. An Instance defines your view or splice of the web. The Instance Configuration contains the domains, websites or web pages that you want Bing to search against. You can create and configure this using the Bing Custom Search Portal.

Note: You need a Microsoft Account (MSA) to access the Bing Custom Search Portal. You also need a Cognitive Services API Key, in this screen shot, you can see I’ve generated free/trial keys in the Azure portal:

Source: Microsoft

When you’ve created your Bing Custom Search API Keys in the Azure Portal and successfully logged into the Bing Custom Search Portal, you’re presented with a series of dialogues that guide you through configuring your first domain. For example, in this screen, I’m setting up an instance to point to my blog www.jamiemaguire.net:

Source: Microsoft

After you’ve added your domain, it gets listed in the Search Experience tab under the Configuration menu, you can see this here:

Source: Microsoft

Remember that Cognitive Services Key I mentioned earlier? That must be added to the Production Tab:

Source: Microsoft

When you supply that in the Subscription Key field and click Call, your Custom Search Instance will be invoked and return data in the API Response section of the web form. I normally use Postman for testing REST API endpoints but being able to quickly test the REST API within the Portal saves a bit of time!

For reference, here is an extract of the JSON response, granted this is a lot to look through. The key points to note however were that I supplied “Twitter” as the search term, and if you look at the webPages node, you’ll see the name attribute does indeed contain the text Twitter.

Another neat way you can test the new Instance is by clicking back into the Search Experience tab and by supplying a search query into the pane on the right-hand side, you can see here I’ve supplied the text “Twitter” and the API has returned website data you would expect:

Source: Microsoft

You can also see there are options to boost or demote the site in the search results and the Portal has also suggested other domains that have been identified as being of potential interest to users of the Instance that’s just been configured.

Being able to tweak your search Instances behaviour using a few mouse clicks is ideal for non-techies!

Image Data

If you click on the Image tab, you’ll see the Custom Search then returns any images that are related to the search term “twitter”:

I don’t have any videos on www.jamiemaguire.net so nothing will be returned there!

But what about the developers out there?

Calling the Endpoint using C#

So, we’ve configured a new Bing Custom Search Instance, tested in using the online Portal and validated that it’s bringing back data we expect. How can you invoke this new Instance using C#?

At the time of writing, there are a few ways:

• hand craft the REST API requesting in a language of your choice
• use the Custom Search SDK

The following example shows you how we can consume our new Custom Search instance using C#. You can find the source code here. I won’t explain what every line does, but at a high level the console application:

• sets the subscription and custom configuration keys
• constructs the REST URL and sets the respective parameters
• adds the subscription key to the http request headers
• makes an http request
• parses out the response into a collection of strongly typed objects

Code Sample

When run, the console application displayed the following values from each web page metadata:

• name
• url
• display url
• snippet
• dataLastCrawled

Naturally you can return more fields than this and you can find a complete list of what’s available here.

Final Thoughts

Being able to build your own custom web search functionality using an API like the Bing Custom Search API shields you from the complexities of parsing human language, natural language processing, text categorisation and website indexing, thereby freeing up your time to let you focus on other business problems. I see quite a few uses for this API, for example:

Enhanced Analytics

Maybe you’re building an analytics platform that ingests data from social media APIs and websites. You could blend Bing Custom Search results with Instagram, Facebook or Twitter API data to give you a 360-degree view of topics that are important to you.

User Experience

The Custom Autosuggest feature can enhance the users’ search experience by automatically providing up to 10 suggestions for search terms that are related to their initial search. Ideal if you want to build intelligent search tooling that makes it easier for users to find the information they’re looking for within your predefined list of domains.

Safe Browsing

Or maybe you need to build a custom web search solution that ensures web browsing is kept safe for children, all of this is possible with the Bing Custom Search API!

Summary

In this blog post we’ve introduced the Bing Custom Search API. We’ve seen how you can provision a Bing Custom Search Instance through the online portal.

We’ve also explored some of the data it returns and we’ve seen how you can easily integrate the API endpoint using C# to help you deliver tailored search solutions. In a future blog post we’ll look at the Bing Web Search API and see how you can use that to tap into the power of Bing to retrieve documents from the internet and get comprehensive results.

Full source code for the C# example mentioned earlier can be found here.

6 December 2018 | Blog

Contact Grey Matter

If you have any questions or want some extra information, complete the form below and one of the team will be in touch ASAP. If you have a specific use case, please let us know and we'll help you find the right solution faster.

By submitting this form you are agreeing to our Privacy Policy and Website Terms of Use.

Previous post
Visualising Data with Microsoft Power BI

Next post
Visual Studio 2019 Preview 1 Announced

Jamie Maguire

http://www.jamiemaguire.net

Software Architect, Consultant, Developer, and Microsoft AI MVP. 15+ years’ experience architecting and building solutions using the .NET stack. Into tech, web, code, AI, machine learning, business and start-ups.