Which Office tools are being powered by AI?
Blog|by Mary Branscombe|8 November 2018
Office has been building in AI tools for quite some time, from the Clutter feature in Outlook that uses the newly open source Infer.NET probabilistic framework to learn what kind of email you’re interested in, to the voice recognition Dictate feature in Word, PowerPoint and Outlook that started out as a Microsoft Garage plugin to add the speech recognition from Cortana to Office. With Office 365, Microsoft is adding more AI-powered tools to Office, and developers can access many of the AI techniques used to build them with Azure Cognitive Services if they’re inspired to try something similar in their own apps.
The new Ideas ‘insight service’ in Insider releases of Excel is like a personal version of Power BI; it creates visualisations of trends and patterns in the data in an Excel table, looking for outliers in the data that it can rank, steady trend patterns over time, groups that it can cluster and any outliers that call outside them, and where a single factor accounts for the majority of a total. You don’t just get a canned visualisation; Excel creates a selection of Pivot Charts in the Ideas pane and you can drag them into your spreadsheet and drill in to start exploring the data.
Ideas aren’t just for the Windows version of Excel; they’re also in the most recent Office for Mac release for Office insiders (full release October 2018).
The Anomaly Finder API that’s in preview through the Cognitive Services Labs lets developers monitor time series data with machine learning, applying a statistical model to detect anomalies, calculating the expected value and measuring how far the actual value deviates from it. That could be a change in the pattern of requests to a service, unusual revenue reports or possible fraud; anything that doesn’t fit the standard statistical pattern of your data.
The PowerPoint version of Ideas suggests designs, layouts and images (it’s similar to the automatic design options in Sway). Drag pictures into a slide deck, say from the Your Phone app, and PowerPoint creates an automatic caption using image recognition, which is great for accessibility (that’s also available in Word, and on the Mac for PowerPoint and Word users).
Other design help is coming to the online version of PowerPoint first: if you choose a suggested design that includes bullets, it uses the text for the bullets to pick icons that illustrate the bullet point. It can also turn a list of dates and bullet points into a visual timeline laid out on the slide with the dates formatted and highlighted, and point out inconsistent spelling and punctuation inside a slide deck, to help you clean up sentences that aren’t grammatically wrong but don’t match style.
Developers could use the Computer Vision cognitive service to create captions automatically, and the Bing Image Search API to find images of a particular type (including clip art, line art, photos and animated GIFs) to match selected text. The Bing Visual search API can identify what’s shown in a photo and search for related or visually similar images to show as alternatives.
The Office Lens app on Android and iOS can correct the perspective of documents, whiteboards and business cards that you take photos of. Send them to OneNote and the built-in OCR recognises the text in those images. A new version of the Excel app on Android for users signed up to the Office 365 Insiders Program will get a similar feature called Insert Data from Picture; take a photo of a printed or even a hand-written table to get a live table of data in Excel that you can edit and perform calculations with. Developers wanting to add this kind of feature to their applications can use the OCR Service in the Computer Vision cognitive service; there’s also a new preview OCR engine that adds a Recognize Text interface to the API that you can try out for English text, including handwriting.
Excel has two new linked data types, Stocks and Geography; currently rolling out in Office 365, these retrieve more information about a company stock or location – like the current price, recent price change, population or average local fuel price – and let you add it to the spreadsheet by adding a column. The Cognitive Services Knowledge APIs can extract common entities from text; the Bing Entity Search API finds entities like people, places and businesses, or the Text Analytics API can pick out organisations, people, places, dates, times and quantities (like distances and temperatures) and provide Wikipedia links for them.
Outlook is starting to use machine learning to improve search results. If you make a spelling mistake when you’re typing in a search, and there aren’t any results for that incorrect spelling, Outlook will show messages that match the correct spelling; that’s currently in the Office Insiders slow ring (now known as the monthly targeted channel) so will be in production versions of Outlook on Windows soon. Search results in your inbox will soon put the three most relevant results at the top of the list, based on how frequently you exchange email with the people in those mails, and how many of the keywords in your search term are in the mail in the right order, so messages with the specific phrase will pop up even if they’re much older.
The Computer Vision API. Source: Microsoft
The Teams application that’s part of Office 365 uses facial detection (which developers can access in the Cognitive Services Vision API) to isolate your face when you’re making a video call and automatically blur out the background, hiding confidential information on whiteboards, co-workers or family members who happen to be in the room or pets that try to photobomb the meeting.
For technical advice or to discuss Azure and Cognitive Services, contact Grey Matter: +44 (0)1364 654100.
Contact Grey Matter
If you have any questions or want some extra information, complete the form below and one of the team will be in touch ASAP. If you have a specific use case, please let us know and we'll help you find the right solution faster.
By submitting this form you are agreeing to our Privacy Policy and Website Terms of Use.
