Nuance & Microsoft Team Up on Conversational AI for Doctors’ Offices

Xconomy Boston — 

Nuance Communications’ plan to lighten the clerical burden on physicians is coming into focus, and it involves an assist from tech giant Microsoft.

Nuance (NASDAQ: NUAN), the 27-year-old speech recognition technology pioneer, in February revealed it’s developing a device that would sit in exam rooms and listen to conversations between doctors and patients. The system’s artificial intelligence-based software works in the background to capture the exchange, parse clinically significant information, and automatically save it in the correct spots in patients’ electronic medical records. Doctors could also use voice commands to quickly retrieve information, such as a person’s treatment history. The goal is to reduce the amount of time doctors spend on administrative tasks, thereby freeing them to focus more on patients and hopefully curtail physician burnout, an ongoing problem nationwide that can hamper the quality of care and increase its cost.

Burlington, MA-based Nuance is testing the new listening system—part of a new category of technology offerings it dubs ambient clinical intelligence, or ACI—with a small group of pilot customers, including the University of Nebraska Medical Center and Rush University Medical Center. Nuance plans a wider rollout in early 2020. (You can click here to see videos that demonstrate how the product works. Nuance vice president Kenneth Harper will share more details on Tuesday at an Xconomy conference near Boston, X·CON 2019: Digital Health Gets Real.)

But it’s a daunting task to develop reliable speech recognition software that also understands context, especially in an industry as complex as healthcare. That’s partly why Nuance wants to work with other companies to enhance its ACI product. Today it announced Microsoft (NASDAQ: MSFT) is one of its key partners. Nuance’s leadership team believes joining forces with Microsoft will help boost the ACI system’s technological capabilities and improve the system’s accuracy more quickly.

“It’s a super thorny problem that we think we can make yield, but we’ve always been concerned about how long it will actually take to get between point A and point B,” says Joe Petro, Nuance’s executive vice president and chief technology officer, in an interview.

Nuance has been developing ACI for about two years, Petro says. The company had an existing relationship with Microsoft, including using its cloud computing service Azure to host Nuance’s Dragon Medical One software, a speech recognition product that more than 500,000 caregivers worldwide use to document clinical information by dictating to a smartphone or computer. Meanwhile, Microsoft has been raising its healthcare ambitions, including developing a product similar to ACI called Project EmpowerMD, an “intelligent scribe service.” The two companies ultimately decided the best approach would be to work together on clinical documentation technology, Petro says.

The companies aren’t sharing the financial terms of the agreement, but the plan is for Nuance to lead the marketing effort, Petro says. The core product will include Nuance’s ambient listening device, speech recognition software, and other technologies, and it will be hosted on Microsoft’s Azure cloud service. He says Microsoft’s AI researchers will also contribute “in a substantial way” to the effort to train the machine-learning software to be able to accurately interpret and document conversations between doctors and patients.

The two companies will also do an inventory of their respective AI software products to figure out how best to combine them into one offering. Petro says there’s “a lot yet to be determined,” but he says that Microsoft’s EmpowerMD technologies will be “harmonized into the ACI platform” in some way.

“This is very much a legitimate collaboration working on a joint development schedule,” Petro says.

Nuance is also partnering with electronic medical records software vendors, including Verona, WI-based Epic Systems, to ensure its ACI software integrates seamlessly with medical records systems, Petro says. For its part, Epic unveiled early-stage plans in August for voice-controlled software that would help doctors more easily conduct patient visits. Nuance doesn’t view Epic’s planned product as competition; in fact, Nuance technology would power much of it “behind the scenes,” Harper says.

“We have a very close relationship with Epic,” says Harper, who is Nuance’s general manager of healthcare virtual assistants. “Most of what they’re doing [in speech tech] is in partnership with us.”

One of the big questions is how accurately the ACI product will transcribe conversations and document key info in the electronic medical record. Petro and Harper declined to share statistics about ACI’s early performance. For now, humans are heavily involved in the process, with a team of medical scribes reviewing the software-generated transcriptions and documentation for errors, Nuance says. The idea is the software will learn and eventually become accurate enough that it doesn’t need human oversight, or, at least, much less. (The patient’s doctor would still be required to review and sign off on the clinical notes, regardless of how accurate the software eventually becomes.)

Nuance is initially deploying ACI in a small number of specialty areas of healthcare, including orthopedics and dermatology. It’s tackling specialties first because they have “fewer chief complaints” and “tend to be more predictable in terms of what’s being discussed,” Harper says. Once the technology masters patient visits for 23 outpatient specialties, Nuance intends to deploy it in primary care and “acute care” settings, he says.

“The reason we’re going in that direction is when you see a primary care physician, what are you going to see them about? It could be anything,” Harper says. “By having all this data for the first 23 outpatient specialties, it will be a huge starting point for us.”

Protecting sensitive health data could be a concern with ACI. Harper says patients must give explicit consent for the device to record conversations with caregivers, and the audio data are encrypted both during transfer and when they are “at rest,” Petro says. The device also has a camera that uses infrared lasers to better track who is speaking, but Harper says the camera can be turned off, the video data stay on the device, and the footage only shows heat signatures and outlines of people, so they’re not identifiable. Petro notes that Nuance handles billions of health data points each week with its existing services, and he says its practices comply with patient privacy and security laws.

The commercial rollout of ACI comes as the company navigates a significant reorganization of its business. At the beginning of October, Nuance spun out its automotive division into a publicly traded firm called Cerence (NASDAQ: CRNC), which has a market value of about $580 million. Nuance retained its healthcare business and its enterprise division, which helps businesses build conversational interfaces with consumers, such as customer service chat features.

Nuance, currently valued at about $4.3 billion, has a lot riding on ACI. It views the product as a meaningful advancement of its speech tech for healthcare, and will offer it as an upgrade for its Dragon Medical One customers, Harper says. If the new product works well, it could help retain and perhaps increase Nuance’s significant healthcare customer base. (Petro says about 90 percent of the hospitals in the US and more than half of American physicians use at least one Nuance product.)

But if ACI fails, it could deal a blow to the company’s foothold in the market. In addition to getting the technology right, Nuance will have to fend off competition from companies such as Suki, Saykara, Sopris Health, and Notable. One also has to wonder whether other tech giants that are investing heavily in speech recognition and other AI technologies, such as Amazon (NASDAQ: AMZN) and Google (NASDAQ: GOOGL), might try to compete with Nuance’s ACI. Google reportedly has done some work in this area, but it doesn’t appear to have brought such a product to the market.

Harper likes Nuance’s chances regardless of whether the competition gets stiffer.

“Our customers believe we’re the ones who can do this because we are their trusted advisor,” Harper says. “We’ve been selling to them for a long time. We know what we’re doing with the data.”

Still, Petro acknowledges that healthcare is a difficult industry to change, and this is a complicated product to get right.

“This is going to be a journey,” Petro says. “We’ll make some mistakes along the way.”