Bring deep web search capabilities to your apps

An introduction to the Bing Web Search API

In a previous post we have covered the Bing Custom Search API and how you can use it to easily build your own custom search solutions. In this blog post, we stay with the site search theme, specifically, we look at the Bing Web Search API.

What is Bing Web Search API?

Sitting within the Cognitive Services and Bing ecosystem, the Bing Web Search API is a RESTful service that provides instant answers to user queries. The API is optimal for applications that need access to all content that is relevant to a user’s search query, and lets you build intelligent deep search functionality into your apps, driven by the global services that power Bing search, all in a single API call. Results are delivered as JSON based on search relevance, and can be indexed by result type, freshness, and more.

You simply input a query to the API and it returns results in the form of web pages, images, videos, news or even JSON (all ranked by their relevance).

Features
In addition to instant answers across web, image, video, news and more, the API ships with other features that include, but are not limited to:
• Filtering search results and query parameters by the content-type for web pages, images, videos, safe search, and more
• Hit highlighting for unicode characters
• Support for localised search results by country, region or market
• Paging
• Optional Statistics module to provide in-depth analytics
• …and much more
At the time of writing, the API has only one endpoint which makes is really easy to use and just like other APIs that sit within the Cognitive Services/Bing family, there are a number of ways that you can consume it.

Consuming the Bing Web Search API

Our Use Case:

Imagine for a minute that you need to build a custom search application that can extract data from multiple data sources that include:

• Facebook
• Instagram
• Twitter
• The Web

Facebook, Instagram and Twitter: You’d be able to extract data from Facebook and Instagram using their respective Graph API by writing custom code, Twitter has their own developer API too. All 3 give you structured data in the form of JSON that you can then process or integrate with your existing application.

You can then map the JSON to strongly typed objects which gives you a nice object model to develop against.

This is an approach I used in the past when building interfaces to help National Geographic extract data from Instagram and Twitter as part of their Photo Ark initiative.

The Web: Scraping data from the web is a little more involved, however, you could build your own custom web scraper, if the structure of the HTML ever changes though your parser breaks! Your IP address might get blacklisted too if you repeatedly bombard search sites with queries.

Why go to the trouble of building a parser and mapping the individual HTML elements that contain the data you need or trying to code around potential IP address bans when the Bing Web Search API can manage all of this for you!

Postman 

Before you can consume the Bing Web Search API you need a Cognitive Services Subscription Key, if you followed the Custom Search blog post, you should already have this, but if you haven’t, you’ll need to create a Key before trying out the following examples.

The root level API endpoint is: https://api.cognitive.microsoft.com/bing/v7.0/search but there are a number of additional parameters that you can supply which you can find out more about here.

Prior to writing any code, I like to try out new APIs using Postman. This gives you instant feedback and an overall appreciation for the main parameters you can supply and the data you’ll get back. In this screen shot, you can see I’ve set the following parameters:

q: “bengal tiger”

answerCount: 1

You can also see I’ve set the Subscription Key in the Headers section:

When I click Send, Postman invokes the Bing Web Search endpoints which returns 1 result (as defined by the answerCount parameter).

The response contains URLs, descriptions, and unique ID’s for any information the API has found that relate to our search term “bengal tiger”:

Using C#

If you need to integrate the Bing Web Search API with your custom software solution, this can be done easily using a language of your choice such as C#, Java, Node, PHP, Python, and a few others. The following code example shows you how to:

• connect to the API
• execute a request
• process the data from the API and display the results in the console

You can download a sample project that includes the above example here.

SDK / NuGet
Another option you have when it comes to integrating the Bing Web Search API is to use the dedicated SDK. The SDK is available as a NuGet package so is easy to install and shields you from the low-level REST API call that you need to make. For example, in the following code extract, you can see how little code is needed to consume the Bing Web Search API when using the SDK:

Filtering Content
The Bing Web Search API contains a useful feature that lets you filter search results based on its type. For example, consider the following search string “NYE” (New Year’s Eve):
https://api.cognitive.microsoft.com/bing/v7.0/search?q=NYE

If we wanted to only include News associated with New Year’s Eve, this can be done by supplying an additional query string parameter responseFilter and settings its value to “news”:
https://api.cognitive.microsoft.com/bing/v7.0/search?q=NYE&responseFilter=news

(Alternatively, for example, if we wanted to get only web pages and News associated with New Year’s Eve, this can be done by supplying an additional query string parameter responseFilter and settings its value to “responseFilter=Webpages,News”)

After supplying this updated search parameter to Postman and examining the results, we can see that only results from News Organisations are returned:

Fox News:

The Mirror:

It doesn’t end there though; the API lets you verify if any other key answer type is missing and filter on other values that include:

• Entities
• Images
• RelatedSearches
• SpellSuggestions
• TimeZone
• Videos
• Webpages
• News

For more information regarding what these mean, check out the API reference guide here.

Market Codes
Another handy parameter you can supply to the Bing Web Search API is the Market Code (mkt). This lets you define which country you want your search results to come from.
For example, supplying a search string “finance” and setting mkt to “en-GB” and invoking the API to return 1 record (using the count and answerCount parameters) yields the following result:

Whereas supplying the same search query but setting the mkt parameter to “fr-FR” returns a URL from France:

Being able to run searches that return results more tuned towards a specific market or location lets you build customised search experiences that are more relevant for your business. For example, maybe you need to aggregate search data from a select number of countries to give you a birds-eye view of current affairs.

Applying this parameter lets you do this. You could even take this further and integrate Microsoft Cognitive Services Text Analytics API to give you further insights.

You can view a list of currently supported Market Codes and all other available parameters here.

Finally

These are just some ways that you can leverage the Bing Web Search API. I see a few more use case scenarios for the API alongwith the use case we just explored that include, but are not limited to:

News Aggregator

Do you need to aggregate news from multiple sources on the web and surface the most popular content, stories or topics being discussed?

Blend the power of the Bing Web Search API with the Cognitive Services Text Analytics API to identify the most popular content or keyphrases online then present this information in Power BI or custom ASP.NET dashboards!

Plagiarism Detection

Another use for the Bing Web Search and Cognitive Services Text Analytics API is plagiarism detection. For example, a document being drafted could be parsed into segments which are then sent onto the Bing Web Search API to see if similar text already exists on the web.

A second layer of intelligence could then be applied to said text by invoking the Cognitive Services Text Analytics API to see how many instances of Key Phrases or even Entities are mentioned, a custom algorithm could then decide if the current document contains too many reference to Key Phrases or Entities as identified by the Text Analytics API!

Fraud Detection

On many to many marketplaces (eBay for example), scammers can try to sell products or services that don’t exist. To help produce digital assets for these fraudulent listings, scammers will often duplicate real listings from other sites! By using the Bing Web Search API and the Bing Image Search API, you could, in theory, use the Web Search API to trawl the web for URLs, then use the Image Search API to check for duplicate listings from rouge sites.

Summary
In this blog post we’ve introduced the Microsoft Bing Web Search API. We’ve seen how you can easily integrate the power of Bing into an application and use it to return results using queries of your choice.

We’ve also looked at how easy it is to filter results based on your search requirements and seen how the dedicated SDK reduces the amount of code you need to write.

You can find out more information regarding the Bing Web Search API here.

Resources

Tutorial: Create a single-page app using the Bing Web Search API

Bing Web Search SDK samples

GitHub Cognitive Services REST API Samples

Grey Matter has a team of Cognitive Services specialists who can help get you set up and discuss your project plans and use case. Email the team directly: [email protected] or call: +44 (0)1364 654100.