I hear dead people

I’m preparing for the upcoming Advanced Institute on Digital Archaeology Method and Practice at MSU. I’m really excited about it; I’m one of the instructors! I’ll be talking and teaching augmented reality for archaeology:

Low cost and low friction methods, tools, and best practices to capture and present archaeological materials in 3D for research, publication, teaching, or public engagement. Special emphasis on physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, digital imagery or geospatial data and information.

So what does that mean in practice? I was thinking on this the other day, and I’ve been building a demo application to put some of my ideas into practice. I’m tinkering with it during quiet moments today, but this ‘diary in the attic’ will give you some idea of the flavour of what I’m after:

Shawn dusted off the old diary. ‘Smells of mould’, he thought, as he flipped through the pages.

Hmmph. Somebody was pretty careless with their coffee.

I think it’s coffee.


Doesn’t smell like coffee. 

What the hell…. damn, this isn’t coffee.

Shawn cast about him, looking for the android digital spectralscope he kept handy for such occasions. Getting out his phone, he loaded the spectralscope up and, taking a safe position two or three feet away, gazed through it at the pages of the diary.

My god… it’s full of….


The thing about hand-held AR is that you have to account for *why*. Why this device? Why are you looking through it at a page, or a bill board, or a magazine, or what-have-you? It’s not at all natural. The various Cardboard-like viewers out there are a step up, in that they free the hands (and with the see-through camera, feel more Geordi LaForge).  I’m trying to make that hand-held AR experience feel more obvious, part of a story. That is, of course you reach for the spectralscope – the diary is clearly eldritch, something not right, and you need the device that helps you see beyond the confines of this world.

Without the story, it’s just gee-whiz look at what I can do. It’s somehow not authentic. That’s one of the reasons various museum apps that employ AR tricks haven’t really taken off I think. The corollary of this (and I’m just thinking out loud here) is that AR can’t be divorced from the tools and techniques of game based storytelling (narrative/ludology, whatever).

In the experience I put together above, I was trying out a couple of things. One – the framing with a story fragment, so that the story that emerges from the experience for you (gestures off to the left) is different from the story that emerges from the experience for you (gestures off to the right). (More on this here). I was also thinking about the kinds of things that could be augmented. I wondered if I could use a page of handwritten text. If I could, maybe a more self-consciously ‘scholarly’ use of AR could annotate the passages. Turns out, a page of text does not make a good tracking image. I used a macro in Gimp (comes prepackaged with Gimp) that adds a random waterstain/coffeestain to an image. The stained diary actually made the best tracking images I’ve ever generated! So maybe an AR annotated diary page could have such things discretely in the margins (but that takes us full circle to QR codes).

Augmented reality works best when it fits seamlessly into our interactions with the world. If you tried out the experience above (you needed to download and print out the pdf, and then install the app on your Android phone; point your phone at each page from about two to three feet away) you’ll see that I chose to augment via sound rather than whizbangy visual effects (although one such effect is really rather neat). We often think of our smartphones and tablets in terms of visuals (retina displays and so on), but the visual is only one sense; hearing the past via AR makes for a much more visceral experience. Visuals can cause ‘breaks-in-presence’ because when things are ‘wrong’ somehow, it is immediately apparent. We get kicked out of the experience easily when the 3d model doesn’t sit on the page correctly, or it flickers because the phone isn’t powerful enough. Audio on the other hand requires us to pay attention in a way that is more subtle. Generations of movies have taught us how to respond emotionally to certain sounds. So I want to experiment with an aural augmented reality for archaeology. That’s my agenda here.

Another element in the calculus of AR is framing the approach. One of the things I tell my history students who are interested in video games, the mechanic of the game should be illustrative of the kind of historical truth they are trying to tell. William Urrichio pointed out in 2005 that game mechanics map well onto various historiographies. What kind of truth then does an augmented reality application tell? In the very specific case of what I’ve been doing here, augmenting an actual diary (a trip up the Nile, from New York, starting in 1874), I’m put in mind of the diaries of William Lyon MacKenzie King, who was Prime Minister of Canada during the Second World War. King was a spiritualist, very much into seances and communing with the dead (his mom, mostly). I can imagine augmenting his ‘professional life’ (meeting minutes, journals, newspaper accounts), with his diaries such that his private life swirls and swoops through the public persona, much like the ghosts and spirits that he and his friends invoked on a regular basis. King was also something of a landscape architect; his private retreat in the Gatineau Hills(now a national historic site) are adorned with architectural follies (see this photo set) culled from gothic buildings torn down in the city of Ottawa. MacKenzie King might well be a subject whose personal history, or whose historic estate, might be very well suited indeed for an exploration via augmented reality.

After all, the man lived an augmented reality daily.


If you’re interested in finding out more about how to use AR or how to build your own, watch my research blog. I’ll be putting up some tutorials in a couple weeks. I’d love to connect with folks interested in trying out similar things.

[parts of this post are nicked from my research blog, Electric Archaeology; you can also find the credits for the augmentations there].