Microsoft promises Bing Custom Visual Search later this year

At the recent Build 2019 conference, Microsoft showed developers a sneak peek of an upcoming Bing Search API for searching through a database of their own images to look for a visual match to an image rather than a text query.

The technologies that power Microsoft’s Bing search engine are available to developers and enterprises in several ways, including the Bing Search APIs. Part of the Azure Cognitive Services, the various Bing Search APIs let you search for standard web results, images, videos or news stories and customise or prioritise the results; you can also use the Bing technology for checking whether the words in a search query are correctly spelled to check the spelling and grammar of text on your own website or in an app.

The Bing Search APIs are not contained to search using words. For the times when it is difficult to use words to accurately describe an image, users can use an image – rather than a text description of what’s in an image – to search for similar pictures through an API to get the same visual search experience that’s available on the Bing site.

“In Cognitive Services today, we have the Visual Search API which allows you to take an image and we will find images like that for you across the planet,” Brian King from the global Bing products ecosystem team explained to CodeMatters. “If you see a product on TV or in a photograph, or even at a friend’s house, and you want to track down the details, being able to search by taking a photo (or using the image you saw online) to find more images of the same thing can help you find out who makes it and where you can buy one yourself. You can crop the image on the search page if you’re interested in just a particular section of it.”

Like the other Bing Search APIs, that’s useful for more than just searches, he points out. “It’s not just about creating search engines; it’s about connecting developers to a world of information – regardless of what public documents or public images you’re looking up – and being able to find and reuse that information. We have an API of all the publicly available data on the planet and we will make sense of it for you.”

For example, schools and companies can use the Bing Search APIs to detect plagiarism or to make sure that any content used in a report has been correctly referenced. Marketing teams can look for mentions of their products in news stories to see if there’s a customer satisfaction issue building up. Businesses can manage their copyright with company images to detect unauthorised resellers or misuse of the company logo and website graphics (if they show up on another website, it might be a phishing site, for example).

The current Bing Search APIs search the public web. This summer, the new Custom Visual Search API will let you search your own image content. “You will be able to bring your own catalogue of images and search just across those images,” King explains.

Obviously, that would let developers create ways for customers to search a product database with their own photos, for example, to look for the exact style of furniture or type of house they’re interested in. But you could also use it to route customer service requests about product issues to the right team by letting owners send a photograph of the problem product rather than having to look up the name or find the serial numbers. Custom Visual Search builds on top of Bing’s existing image understanding capabilities by building a custom visual similarity search experience on your own images.

“We train deep neural networks representing images as a set of features and ultimately floats and vectors within a graph,” King explains. “And then we do the approximate nearest neighbour functions on top of that to be able to find that. You can think about thousands of features representing an image. So things like the colour, the shape, how do objects interact with each other and overlap – we represent those as a set of features. When you bring an image we haven’t seen before, we find something that’s the closest we can find.”

The Metropolitan Museum of Art has been using an early version of the Bing Custom Visual Search technology for some time to let people search their catalogue visually, so if you see a piece of artwork you like, you can look it up from the photo rather than typing in a description and hoping you hit on the right keywords. It’s also an interesting way to explore how different artists have depicted similar scenes.

Several companies have also been using Bing Custom Visual Search API during the private preview for tasks like catalogue management for online retailers, King says. “We’ll roll this out later this summer in a more public preview, which is our normal path for getting into production.”

In case it’s useful or to aid in understanding the different APIs and Services, here is a quick guide with examples:

Bing Search APIs – A set of JSON-based restful APIs as part of Cognitive Services, which allow developers to bring and connect the knowledge of the web to their own applications and services.
Example use cases: Build your own search engine. Plagiarism and fraud detection, company reputation management.

Bing Visual Search API – A JSON-based restful API as part of Bing Search APIs, which allows developers to search the web using image as the query (rather than text), where the results allow you to understand what is in a given image and find similar images on the web.
Example use cases: Finding duplicate and similar across the web, identifying objects and entities like landmarks, flowers, dogs, and more. Image Copyright management.

Custom Visual Search API – A brand new service that allows developers and retailers to build their own instance of visual search across their own image data. The service provides capabilities to ingest a developer’s image catalogue, build an index and provides a JSON-based restful API to consume the output of the service.
Example use cases: Retailers can build a visual search experience to search across their own catalogue of images for their end consumers. Developers can create line of business application to improve their internal services and applications.

The Grey Matter Services team are Azure experts and can provide consultation on the Cognitive Services suite including the Bing Search APIs.

Contact [email protected] or call +44 (0)1364 655 124.