Emotionally compelling AR Views

Despite being an author and promoter of “augmented reality” games for more than a decade now, a dedicated enthusiast if you will, I find that what gets shown off most often as augmented reality in the end seems like a boring gimmick. This is disappointing in some ways (not all—experimenting in a medium has its own value). But even though this is what I see peddled around, I don’t think it has to be that way. What most tech enthusiasts and bystanders understand AR to be could become something more alive. And I have some ideas about how to get there.

Update – I added a section at the end clarifying where inspiring work in AR can be found. There’s lots out there but you need to be looking for it.

To begin, let’s look at AR as typically advertised. Here’s an example that is about par for the course:

AR is defined as a smartphone-based mechanic whereby a user points their phone’s camera at an image target. When the image is recognized, the AR software layers new media atop it in the video feed of the phone.

Technically it is a neat trick, but don’t you feel empty seeing things like this? I mean if you wanted to see a bit of web-slinging from Spiderman, why would you want to see it this way instead of full screen or in a more complete game? One can imagine using a bit of this as an aha! moment, but imagined as anything more than a brief moment, it just feels janky and pointless. At least to me.

And it’s not just that the commercial market is especially vapid. There has been tons of educational interest in AR where the standard demo reel is similar: you point your phone at a textbook and some poorly-rendered 3D content appears to jump up from the page. Why you would want to see that as a 3D model instead of a simple photo, and what possible use it might have in deepening a conversation never seems to come up. It’s just about the spectacle, which to be honest, never seems that spectacular once you get past the bit of magic that it works.

Other than complaining that I find others’ interests a bit empty—and I’m sorry for that, I really do encourage the learning involving in simply trying out new things to see where they go without knowing that they are what someone else calls awesome or worthwhile—I’d like to take this occasion to bring up two more positive ideas:

  1. AR has been, is, and should be about more than this little trick of through the lens visualization
  2. In creating new media, the point is not to show off a technical gimmick but to use these tools to create emotionally engaging experiences, something that occupies continued interest not just immediate attention.

A Better Definition of Augmented Reality

So what is a bigger picture of what AR is and can be? I’m not the best at distilling ideas into soundbites, but here goes:

Augmented reality is what you get whenever you build a world form both the “real” world (aspects of physical presence, perception, social realities, etc.) and “virtual” bits (goal structures, player characters with specific features and abilities, fictional non-player characters, etc.). There are many ways to do this, it is not always new or solely the province of smartphones and other advanced technologies, and it is not limited to the mechanics for combining real and virtual elements into the creation of new worlds that we have at our hands today.

This is a general definition. It is not quite as general as “interaction through mobile technology” nor is it similarly aligned; ultimately, AR doesn’t have to be built with a smartphone; what its built from should be a secondary concern. The primary focus is the hybrid character of the experience, combining perceptual reality with other layers of meaning. Naming AR as a general thing serves two purposes: 1) To get our mind off the mechanics for a moment to be able to see our work as enacting a broader purpose, and 2) to open our minds to the already existing ways in which we augment reality to inform our new creations. The latter should be of great help when it comes to looking for places to borrow ideas from in constructing new experiences with AR.

What is this broader purpose to which AR might point? One possibility is to unhide the world. Creating AR worlds can act as a bridge for players, bringing them into new worlds that they live alongside but cannot see. We can also see AR as a tool for bringing people together in a place, both by playing and by making together. Too often we think of the outside, “real” world as singular and obvious when, in fact, so much of it is hidden so often for so many. Our direct perception of the world does not inform unless we have a framework in our minds through which we can interpret what we see, hear, and feel as having meaning.

Take any branch of science or profession and it is easy to imagine how a person with that training sees a different world in front of them than those outside that perspective. A geologist looks at the rocks around me differently than an artist who sees something in another way entirely than my five-year-old. The history of a place is almost always hidden from plain view, though if you knew some of what to look for, you could see the roots of the now in the past.

At a very minimum, we can think of augmented reality as the potential to unhide places and ways of seeing. By combining what any person can see with what a specialized professional knows (or some other unique perspective), through various techniques of situating media in place, we can begin to teach people how to see and think in new ways. By doing more than presenting media, by creating worlds where these new abilities have uses and consequences, by involving those learning to see in stories of their own, we can do better than tell. We maybe could give them a reason to care.

Now there is more to this potential than providing glasses that allow someone to see and act through the lens of a trained professional. There are lots of lenses, and in a given setting, the combination of perspectives might be the most important thing. Another that is easy to understand is time. The world we live in has changed over time, but all we can see is what is here now. At least without help. The phone can both situate historical media in its original location, helping viewers to feel what a place used to be like, and also be a conduit to other representations of a place: maps, diagrams, and other records. Creating something more akin to a world, not just a media presentation, we can create purpose for the viewer, make them a player in this world, and have their role and actions embody something valuable about what we want them to see and feel.

When it comes to constructing an augmented reality, although a smartphone is by no means needed, it sure has a lot going for it. These devices are with us wherever we are, and their abilities serve to connect us to various contexts and extend our perception. They know where we are, what they are in the presence of, and what they are looking at. They can present media, collect it, and contain and connect to vast virtual worlds. We can see them as both portals and mirrors. AR should not be seen as a subset of mobile media, but rather as a form of design that often embraces what the phone has to offer.

Mechanics of AR

Looking a bit more deeply into the mechanics, we can go back to the “AR view”. The mechanic I maligned above (the AR most people seem familiar with) makes key use of the smartphone, a person’s presence and recognition of a target image, the ability of the smartphone to be pointed at and recognize this image and then, in the presentation it shows to the user on their screen, to augment that sight with media content from another world. But the “AR view” is only one such tool. Other obvious and useful mechanics that can be used to stitch together worlds:

  • GPS location
  • QR Codes
  • Bluetooth Beacons
  • Contextually relevant information (e.g. put together every third letter on the plaque you find at the statue)

All of these are currently possible in ARIS by the way. SCVNGR had a few others once upon a time too. And alas, I do miss Bump! Imagine too how this list might grow longer in the future as the capabilities of these personal devices expand and become more personal. What about a game that required various heart rates at specific times? And we can also see how a mechanic that allows a person access to a world would also permit collaborative or competitive action with other players, participants in that same hybrid world.

There are a couple mechanics that appear to have extensive usefulness, and which could be had in ARIS with only minimal funding. Here’s two:

Custom maps – like Pokemon Go, instead of having your game contents float above a regular ole phone map, actually be able to customize this representation of physical space.

Passphrase – the dictation technology already present in our phones means it is only a small jump to be able to connect game contents to what you say. Of course language learning folks would jump at this, but even as a general game mechanic it would be really cool to have a passphrase that knocks next steps.

AR is everywhere already

It may sound abstract to put it this way, but AR is everyday. AR is already a large part of the phenomenon of mobile technology, how these devices have come come to redefine much of our lives in the last few years, even if it is not named AR too often.  Regardless of whether this changes, and we decide to start calling more things AR, it is useful to look at bits of life reframed through the use of a mobile app as examples of AR, instances where a hybrid world has been created and the extra layers emphasize certain aspects of the existing world to create a new understanding, ability to see, or ecosystem even. The role of these devices and their software has already begun to become invisible, but we can peel back the curtain and see the thought in design that makes them tick, see how our experiences are redefined through augmentation.

A great example of unnamed augmented realityis the app WAZE. It is for drivers of cars, and helps them to get through traffic and other obstacles. It features way finding (maps and signs are the original AR and have obviously been supercharged by smartphones’ ability to know their surroundings and access network knowledge about the state of the world), collective reporting of traffic and incidents, and represents its understanding of the state of the world by varying how it shows this world to a driver (e.g. green roads for light traffic, red for slow). Users of WAZE even have custom avatars and compete for points all of which only exist in the app though they are influenced by what players due in reference to the outside world as it is a part of the combined, augmented reality. These pieces of WAZE are hallmarks of virtual worlds we usually call games. And they bring with them all of the attendant guidance of experience we have come to associate with that medium.

Towards an Emotionally Engaging AR View

So now that we’re all agreed: AR is about creating hybrid worlds, not popping up some 3D model, we are compelled to use its mechanics for good not boredom. So how might this ability be used for something compelling? Could the narrow mechanic of the “AR view” be used for good? Where might having one piece of media triggered by the recognition of another get interesting? I don’t have all the answers obviously, but here’s a hunch.

We need to reveal the stories that lie behind objects. AR should help us see into them, and the person doing the seeing has to have a reason to look deeper.

Here’s an example that occurred to me today. I set up the experience in ARIS in a few minutes. It goes like this:

I’m walking through my kitchen and notice a bit of kid art on my fridge.

walking through kitchen

I have an ARIS game open on my phone and have it in a mode where it is looking for AR image targets (this feature uses a library from Vuforia). When the app running on my phone recognizes the kid art, an image it has in its database, and I tap on the screen, I see this:

(You didn’t know ARIS could do this? Well, surprise! I guess this is a strange way to announce a new feature, but yes, ARIS now—in beta—does this AR image recognition trick)

The kid art posted on the fridge is an artifact from a fun evening, a moment in the life of our family that is now gone forever but which we remember through memorabilia, as is the video. Though both refer to the same memory, normally the piece of paper and home video go on to lead separate lives. The AR moment in this case connects the physical reminder on the fridge to the video. This example indicates some of the potential of reconfiguring the access and replay of memories.

I think this is a very good moment of AR. It felt exciting to come up with, and my wife genuinely smiled when she tried it out herself. Likely, this particular moment is only really relevant to my immediate family. Even though I hope you understand and appreciate the meaning of it as an example, I wouldn’t expect you to want to come to my house and scan the image to uncover this video. It’s just someone else’s kid being silly. But it does make me imagine what it would be like if many things could easily share their stories.

Context is important, and we should not forget this as we move ahead. A 3D model of some ancient Greek urn just doesn’t have much connection to a random kid, no matter the quality of rendering, just as watching my kid be silly isn’t interesting to most people. There has to both be purpose in uncovering something hidden in plain sight and a reason why the person might actually bother doing so.

Once more too, this isn’t just about AR image recognition, AR, or game design. When we go about our business of wanting to improve teaching and learning, are we simply extending a content fetish? Are we passing along knowledge and procedures that fit into a generally accepted sense of what schools and studying should be about, or are we engaging with our deepest sense of what it means to be alive and a real sense of wonder and discovery?

And it may be in the end that our representations in the genre of AR do feel a bit staid and limiting. To be sure, we need not only look for meaning in product. The journey is not only important for what’s at its end. Trying to figure out these new techniques, applying them in new situations, and putting all the bits together to get something working, there is joy and meaning there: learning, camaraderie, etc. You hear this all the time from those who work in film and TV. They find the end result to be a bit trashy and not worth their time to watch. But the work that goes into making this trash is quite rewarding and varied.

So my first bit of advice to those looking to get into AR by way of education is to make sure that your process illuminates at least as much as the product you hope to produce. Even if the media you produce doesn’t wow the world (or me) hopefully making it gave you a chance to try something new and see how it works.

It’s not just me

Although I opened up this diatribe against the way I see AR portrayed generally in the world, I don’t want to give the impression that most of the work near me in AR falls into this pit. What has made this work so worthwhile over the last decade is the fascinating and inspiring work of others in this space. Whether it is those making AR platforms like ARIS, Taleblazer, or FreshAir, or the teachers, researchers, students, and community members using these mechanics as a way to rethink and recreate their worlds, I find myself continually renewed by their work. To me in my everyday life, this is AR. It is the realization that most people do not get to see this good work, that most presentations of AR out there in the world are so much more limiting than what I see around me, that caused me to write this piece. The articles about how AR is the future, the VC pitches, the TED talks, and the majority of what is uncovered by a brief Google search—this is what I’d like to push back against, hopefully with the effect that more might realize there could be some substance to AR, not just a niche gimmick.

My example above indicates where we might take the mechanic of the “AR view” as ARIS incorporates it. But ARIS and these other tools have been around a while, and there are tons of examples of moving work there. I can’t list them all of course, and I’d really like to hear about more, but here are a couple to get started if you’re looking for real work that has been done with the sort of zeal I hint at:

In the book, Mobile Media Learning: Innovation and Inspiration, there are many stories, told plainly by inventors in many fields, that fit this mold. I was very lucky to to have a chance to help bring them to light. It’s ETC press so PDF’s are downloadable free and paper copies are very reasonable.

More recent efforts include:

Prof. Terri Nelson’s Paris Occupé is an intricate design with a strong emotional core. I wrote about this game briefly before and linked to a recorded presentation.

School teacher and artist based in Vancouver, Craig Brumwell has created a couple games now to help both students and the public at large to connect more deeply to the history in their communities, particularly around the legacies of the world wars. Here’s his description of the design of his first game, Dilemma 1944.

If you want to search more broadly, I’d recommend throwing in the name of one of the platforms above (like arisgames), avoiding the too terse cheerleading sites and top ten lists you often see in tech and ed tech journalism. And if you’d like to share your story, get it out there, and maybe consider sharing it with me.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s