Helping users express themselves is key to technological adoption, says Snap’s principal researcher

Mid-conversation with Dr. Andrés Monroy-Hernández, he places a virtual filter of a tabby cat on his head while he keeps on answering questions. Just another day in the life of the principal research scientist at Snap Inc. “How to convince people to use your technology is the number one question,” Monroy-Hernández said in an interview with Calcalist. “Part of the answer is making sure that your technology helps users express themselves. If the tool is flexible enough and allows people to see or express themselves in new ways, they’ll be drawn to it,” he said.

Monroy-Hernández and Snap have a difficult task ahead of them. The company behind the popular app Snapchat had a bright start, but since its founding in 2011 the company has suffered from copying by the likes of Facebook Stories or others and is looking for ways to maintain its position as a leading power in augmented reality (AR). To do that the company has encouraged over the years a philosophy of innovation.

Just this week, the world saw that philosophy in action when Snap acquired Israeli startup Voca.ai, which has developed virtual sales agents powered by artificial intelligence. “In the 1970s it was very difficult for people to express themselves with computers, but today it’s simple. With software like Adobe Photoshop you can really express yourself. We’re seeing people adopt technology time and time again. Technology often starts as a useful tool for professionals or used in a sophisticated professional setting, but branches out to simpler use. Take printers for example. Today anyone can use their home printer to print party invitations,” he said. “This is already happening with AR as well, and people are installing AR apps onto their personal devices. First, they’ll start with something simple, like virtual lens filters, putting a cat on their head. Next, they might do something silly to distort their face. Maybe later they’ll be able do something even more complex. These are baby steps.”

As part of an effort to find out what exactly qualifies as self-expression, Snap collaborated on a project with Microsoft Labs and BBC R&D to host a virtual creative challenge where 11 teams from universities in eight different countries participated, among them Virginia Polytechnic Institute and State University, Carnegie Mellon University, Interdisciplinary Center Herzliya (IDC), and others. “One of the things that we are trying to do, based on interactions with academic professionals, is to cultivate a community of people who are interested in the topics that we are curious about. The community tends to be diverse, the more people you have the more ideas you’ll have as well. We wanted to see whether they could innovate. Augmented reality is a relatively new field, and that’s why having a diverse community helped the development of our product.”

You’re working with groups all over the world. How can you develop a technology that will appeal to multiple cultures?

“We think about augmented reality as a clean slate and are trying to understand how people interpret the technology and bring it into their own lives. Our idea is simply to give people tools that fit their personalities, cultures, and orientations. One of the challenges of our test was to speak to people from multiple backgrounds. There are many elements in augmented reality that I felt were missing. That’s when it hit me that AR isn’t simply the transition from 2D to 3D, we also have to think about how our physical world is part of the virtual world, and how virtual objects respond to things, such as changes to the weather or our heart rate. I think that the more this field develops, the more we’ll see more of this type of influence.”

Pokemon Don’t-Go

The highlight of the challenge was meant to take place this past summer, during the annual Association for Computing Machinery (ACM) conference on media interactions in Barcelona. But like almost every other scheduled event this year, the coronavirus (Covid-19) pandemic threw those plans into disarray, and the conference took place virtually. The distance didn't change much of Snap’s work with the teams, but it did make them re-evaluate their work.

“We came up with an idea of sharing location experiences even earlier. Our tools are programmed for remote interactions, and if you look at your phone many of the things we use have distance in mind, like messages, apps, etc. We thought of ways we could support close face-to-face interactions. When the pandemic broke out, we wondered whether we should even continue to test this, since most people were already social-distancing. But we realized that even if people aren’t interacting at the office or at school, we still have face-to-face interactions at home with our family members or roommates. We realized that people were looking for additional tools for interactions, such as board games. This helped us approach the issue in a new way.”

Over the years, Monroy-Hernández has built platforms that millions of people use, whether it was at MIT or at Microsoft. He started off developing Scratch, an award-winning online community that teaches children how to program for free, then continued to Calendar.help, Microsoft’s first human computing device; then Sana, a health app that is designed to help nurses in Zambia. He was crowned as one of the Top 35 Innovators Under 35 by the MIT Technology Review, was rated as one of the top influential Latinos in Tech by CNET, won the prestigious MacArthur Digital Media Award, served as a fellow Harvard researcher, and was a faculty member at George Washington University. Next summer, he is planning on joining the computer science department at Princeton University.

“I’ve had several projects in healthcare and education that were designed to help children express themselves. I’ve worked with projects that dealt with the ways that people and political groups use social media during a crisis, and also funny things like what I’ve done with Snap. I find they all share a common denominator, even if it doesn’t appear that way from the outside.”

Monroy-Hernández calls that common denominator social computing, or building technology that helps, supports, and encourages people to collaborate. “Wikipedia is a classic example,” he said, “before it got started, the idea that strangers would collaborate to write the largest encyclopedia in the world was unimaginable. Part of the reason it’s happening is because technology has lowered the level of friction. Before technology made it possible, we had to lock ourselves in our rooms and work for a year. Now, it’s possible to constantly make small edits and be part of a great community.”

Since 2017, he’s been managing Snap’s science research center. “I see AR as a new tool for self-expression that allows us to do things we could never do before, whether that involves creativity, political expression, or any other form of self-expression. It’s all on the same spectrum.”

What’s in store for the future of AR?

“I think that it is as diverse as human self-expression. I’ve seen demos from different companies that are developing tools for AR at workplaces. For example, using AR to operate devices. When it comes to self-expression, we’ve seen people design their own party invitations or short videos. Now we’re looking into how we can enable everyone to use it.”

When asked whether Snapchat users will adopt a more political tone like other social media have, Monroy-Hernández recalls Eitan Zuckerman’s Cute Cat Theory. “According to Zuckerman, any technology that could be used to upload pictures of a cute cat can also be used for political expression. Only when you feel comfortable enough using technology to do something casual like uploading a picture of a cat, will you feel comfortable enough to use it for other things in your life. Part of the thinking behind this is what researchers do, we think about the spectrum - going from serious to fun, from fun to creative.”

According to Monroy-Hernández, one of the most important considerations when developing technology is how well people can understand and use it. “Part of our job is to function as educators, to help people think more critically. The first thing that comes to mind for me is Adobe Photoshop. In the 1980s, people began editing pictures and we were all required to become aware that it was possible. We should all be more skeptical or analytical about the way information is relayed to us,” he said. “There is always a new frontier when it comes to what can be edited. Once things received a literal stamp of approval, which served to authenticate the validity of a letter. The race to develop technologies is neverending. People should constantly be aware that the information they are receiving isn't always the same information that was sent.”

The Israeli team’s project

The Israeli team that participated in the challenge was led by Dr. Jonathan Giron and Ido Wald, along with Prof. Doron Friedman from the Advanced Reality lab and Dr. Oren Zuckerman from the Media Innovation Lab at IDC. Six students from different faculties worked alongside them. The team then presented a virtual reality character, named Cottage. “We were interested in the interaction between physical objects, virtual objects, and users,” Wald told Calcaslist in an interview. “We searched for a way to communicate a complex system through the use of a story, some type of character that could facilitate control of it. Smart house systems, for example, often create a feeling of detachment because things that happen are disconnected from the user. But when a system tells you about itself, and its connections are clear, it strengthens the sense of control and the general user experience.”

“The moment when an object responds and creates a kind of continuous communication with a user, it becomes a two-way relationship. The field is still in its infancy, but once a technology comes out that is ready for widespread use in terms of user experience, a new layer of reality will be formed,” Giron added.

People have been saying it for a decade

“Even with AR, the challenge is great,” Wald added. “It’s not so simple to get people to put on AR goggles, even if they work flawlessly. The change is not around the corner, but challenges like Snap’s makes it feel like we’re getting close.”