Geppetto Avatars Melds Emotion of Disney, Smarts of KITT

[Corrected 5/15/14, 3:41 pm. See below.] The fact that Norrie Daroga considers Walt Disney the forefather of his company’s interactive virtual characters is telling, and not just because Geppetto Avatars was partly named after the woodcarver who created Pinocchio, the puppet who dreamed of being a real boy, in Disney’s classic 1940 animated film, “Pinocchio.”

Disney’s team understood that the secret of a successful animated movie was crafting characters that can touch audiences emotionally. Geppetto Avatars, a Mequon, WI-based startup with offices in San Francisco, Los Angeles, and Boston, is seeking that emotional bond, too. But it is also trying to do something harder, something that Disney could only imagine. It is building virtual characters, or avatars, that actually interact with people, characters that could become complementary caregivers, teachers—maybe even friends. As co-founder and chief science officer Mark Meadows says in a video on the company’s website, Geppetto is creating “smart puppets that can listen to you, read your emotions, and understand you and reply in real time.”

There’s a place for such empathetic characters in today’s health care system, Geppetto believes. Take the thorny and expensive problem of asthma patients making frequent hospital visits because they experience repeated asthma attacks. One way to help prevent those attacks is to check in with asthma patients every day—yet understaffed nursing departments often don’t have the resources to contact patients that frequently. Geppetto’s characters can. Patients could chat daily with an avatar on their tablets, smartphones, or laptops, says Daroga, the company’s CEO and other co-founder. The software would log how the patients are feeling and how often they’re using their rescue inhalers. That info would then be sent wirelessly back to the caregivers, who can intervene with patients at high risk for another attack. The avatar can also look online at the forecasts for weather, pollen, and pollution, and warn patients not to go outside when conditions are bad, while also sending email or text messages or making a phone call reminding them to take their meds.

During the interview with Xconomy, Daroga played a video of an exchange between David, a middle-aged man with asthma, and Sophie, an avatar dressed in a lab coat and holding a patient chart. She asks why he had five asthma attacks the previous week (he says he’s catching a cold), and what his condition is hindering him from doing (“walk up stairs, play with the kids, go see a movie,” he responds).

Geppetto Avatars demoGreen and red dots are superimposed over David’s image on the screen, indicating the observations by the facial recognition software. Next to it is an octagonal-shaped heat map with emotions listed on the edges, like fear, shame, indignation, and confidence. As David speaks, blue and white-colored octagonal clouds shift around the grid and hover in the sectors corresponding to his perceived emotions. For example, Sophie explains that the local weather forecast indicates a 65 percent increased risk of an asthma attack. “You got some popcorn in the house?” she adds in a lower-pitched voice. “Ah sure, I get it, not going out today,” David replies, as the heat map indicates he’s feeling fear and shame at that moment. Sophie advises David to keep his inhaler nearby, and he tells her to send him a text message reminding him when to use it.

In addition to improving the care of patients like David, Geppetto’s virtual assistants could speed diagnoses of diseases by conversing with patients, cross-referencing their responses with online scientific databases, and creating a list of possible disorders the patients might have. That would save doctors time, and maybe even improve healthcare.

The avatars also could help pharmaceutical companies that are developing drugs for cognitive diseases like Alzheimer’s or depression, Daroga says. Being observed or interviewed by Geppetto’s avatars could be a more reliable way to measure patients’ improvements or declines while taking the drug than observations by doctors or nurses. Humans just aren’t as good at objectively and scientifically charting small changes, he explains.

The avatar, on the other hand, could track a patient’s daily speech improvement over the course of a month, or which emotions the patient often shows, measured by things like the tone of voice, word choices, and facial expressions, Daroga says.

“Now after three weeks, the conversations [display] 90 percent more confidence or 90 percent more happiness, you’ve got a touch point,” Daroga says.

The potential applications for Geppetto’s avatars go beyond healthcare. The company also plans to market its platform to education companies that could use it to help children with learning disabilities improve their reading, spelling, and other skills. The two-way conversation is more engaging than a video, and the avatar can customize the instruction to each child, Daroga says, ending the lesson early if it senses the child is getting frustrated, for example.

If a soothing virtual teacher or caregiver who’s attuned to your every emotion sounds a bit like science fiction, well, that’s not surprising. Hollywood and pop culture have long been fascinated with the idea of virtual companions or computers that can “think” and hold meaningful conversations with humans, from the HAL 9000 in “2001: A Space Odyssey” to KITT in “Knight Rider,” and more recently with Tony Stark’s Jarvis and Samantha in “Her.”

Those fictions have been inching ever closer to reality in the past few years with intuitive Q&A systems like Apple’s Siri and IBM’s Watson. And over the past couple of decades, tech companies have worked toward creating avatars with the sophistication of a KITT or Jarvis. Some companies have seen commercial success building interactive virtual assistants that provide online customer service for clients in sectors like health insurance, the airline industry, and antivirus software, including WA-based Next IT and CA-based VirtuOz, which was acquired last year by Burlington, MA-based Nuance.

But Geppetto goes a crucial step beyond all the current offerings, according to Daroga, by giving its avatars the ability to detect emotion and respond differently based on the user’s mood.

“If I walk in and say good morning to my assistant and I’m gruff, her response will be different. No avatar can do that today,” Daroga says. But Geppetto’s avatars will, he adds.

How? Part of the answer lies in a combination of several software technologies—artificial intelligence, data analytics, natural language processing and voice-to-text translation that allow the avatar to understand human speech, and a facial recognition program to help the avatar read a person’s mood. Geppetto has developed its own voice-to-text module from scratch and created proprietary natural language processing and artificial intelligence programs using open source code. “I won’t be able to compete with somebody who focuses on voice-to-text or AI or just one of those things,” Daroga says. “I’m taking the best I can of all of those technologies…using them in ways that people haven’t done.”

More importantly, though, Geppetto is augmenting those technologies with art, storytelling, and psychology. The handful of anthropologists and sociologists the company has hired are just as crucial to its goals as the software developers and graphic designers, Daroga says, in the same way that Disney’s team needed expert storytellers in addition to animators. “There is a certain charm to it that you need to be able to build into it,” says Daroga. “There’s an art to this; it’s not just science, and that’s what’s great.”

But will science and art translate into sales? What Geppetto is trying to accomplish with computer programs that can interpret and respond to human emotion is a well-researched topic, but there haven’t been many examples of commercial success in this area, says Andy Palmer, a Boston serial entrepreneur who founded Koa Labs in Cambridge, MA, and has advised Affectiva, a Waltham, MA-based company that built facial recognition software to analyze people’s emotional responses to online videos.

“That’s the real challenge” for Geppetto, Palmer says. “What are the commercial applications that are going to be the most compelling over the next 10 years that are going to have the basic economic fuel to drive a significant company that can do dramatic things?”

Geppetto’s business model is to sell monthly subscriptions to the avatar platform, which will be launched this year. Daroga anticipates considerable interest, including among insurance companies and hospitals that are under increased pressure to reduce 30-day readmissions and cut costs under the Affordable Care Act, and pharmaceutical companies that could benefit from more clinical trial data and the revenue they’ll earn as a result of faster diagnoses of rare diseases. Geppetto is in talks with interested pharmaceutical companies, Daroga says, and has already formed a partnership with New York-based Evidence-Based Solutions and signed up some early customers, including Milwaukee-based TCARE Navigator (a startup Daroga also helms). [An earlier version of this paragraph mistakenly called Evidence-Based Solutions a customer, when it is in fact a business partner. We regret the error.]

Geppetto is also looking for investors. The company has raised $500,000 in seed money from family and friends, says Daroga, who has invested in the company. It’s now seeking between $5 million and $10 million for a Series A funding round.

The key question for potential investors is whether or not Geppetto’s initial applications for its platform hold promise as a viable business, Palmer says. He likes what the startup is trying to accomplish, but wonders if its aspirations are too lofty, at least for the initial product. His advice is to follow a strategy similar to Affectiva, which picked a simple and practical objective—measuring people’s “emotional connection” with advertisements, brands, and media—and is building the business around it, he says.

“Things that feel real exciting but aspirational may not be the best thing for you to start a company [and] get off the ground. Things that are very pragmatic may go a much longer way,” Palmer says. “Maybe [Geppetto has] cracked the code, but it’s a very hard thing when you’re on the cutting edge the way these guys are.”

Daroga admits the company must not bite off more than it can chew in the early stages.

“If we start chasing the asthma app and sit down and try to build 22 apps and have no client for them, it’ll kill us,” Daroga says. “We’ve tried to figure out how do we stay on course. The diversification of opportunities is both an opportunity to reduce risk, but is also one to lose focus.”

Geppetto has 10 staff, along with about 20 scientific advisors and contract workers around the globe, Daroga says. They include 3D animators in Argentina, voiceover actors in Los Angeles, and an expert in Germany working on technology that helps the avatar perceive emotion from facial expressions. (That last project is one of the difficult but vital facets of Geppetto’s platform that still needs to be finished before the product can hit the market, Daroga says.) Another challenge for the team is designing avatars that find the sweet spot between appearing too “whimsical” to take seriously and so human-like that they’re “creepy,” Daroga says.

Geppetto has four pending patents and four provisional patents covering avatar technology developed by Meadows and his team at HeadCase Humanufacturing, a now-defunct L.A. company, Daroga says. (Meadows, who lives in the Bay Area, is an artist who has founded three companies and worked in design labs at Xerox-owned PARC, SRI International, and the Waag Society. Daroga, who moved from India to the U.S. at age 16, is a mechanical engineer and attorney who has spent more than three decades in business leadership, including as chief administrative officer for Milwaukee-based Metavante Technologies. He left Metavante in 2008, a year after the company went public and two years before it was acquired by Fidelity National Information Services.)

Even if Geppetto’s business model works, it still must compete with giants like Google, Apple, Microsoft, and Nuance that are developing products or snatching up companies with technologies like those on Geppetto’s platform. “The odds are always against you,” Palmer says. But the market for this type of technology is still a wide-open frontier, he adds. “The script hasn’t been written yet for how these kinds of [platforms] are going to make it in the market and be used effectively.”

The company’s ultimate success may turn on whether Geppetto can deliver on its promise to create virtual characters that humans can relate to. Studies show that people often open up to a virtual character, and almost develop a form of companionship. That’s especially likely to happen with a friendly-looking avatar as compared with speaking to a bodiless voice or typing sentences into a text box. Daroga points to research into the potential for virtual characters to interact with lonely elderly people and in therapy for soldiers with post-traumatic stress syndrome.

“A human being asks them, they say, ‘I’m fine,’” Daroga says. “They talk to an avatar and they talk about all their fears, all the stuff they can’t tell their commanding officer. And now you’re able to help them. The avatar is nonthreatening; it’s not judgmental.”

Trending on Xconomy