Art and augmented reality: Q&A with Cuseum founder Brendan Ciecko

On March 18, 1990, 13 highly valuable paintings were stolen from the frames of the Isabella Stewart Gardner Museum in Boston in what remains the largest unsolved art theft in history. While the paintings still remain at large, Boston museum tech company Cuseum has offered an interesting fix using augmented reality technology.

In a project called “Hacking the Heist,” Cuseum has experimented with augmented reality in an effort to restore the paintings, using the technology to put the artwork back in the frames using a mobile device. The Cuseum team has successfully recreated two paintings with this technology and may experiment more with the project in the future. The company is also working on other projects using augmented reality technology to make the museum experience as interactive as possible.

The Daily had the opportunity to talk to Brendan Ciecko, founder and CEO of Cuseum, about "Hacking the Heist" and Cuseum's other projects, as well as the future of augmented reality technology and tips for young Tufts students interested in the world of technology. 

Tufts Daily (TD): What exactly took place during the heist 28 years ago, and how did that serve as an inspiration for “Hacking the Heist”?

Brendan Ciecko (BC): As you might know, 28 years ago the Isabella Stewart Gardner Museum had 13 significant works stolen from its collection. This is the largest unsolved art theft in history of its kind in the world. For those who lived in Boston or grew up in the area, many are familiar with the story. It’s something that my team, visit after visit to the museum, would talk about and think about, but it wasn’t until recently after completing some preliminary augmented reality-based experiments at the office that a member of my team said, “You know what’d be really interesting? To put the stolen art back into the frames of the Isabella Stewart Gardner Museum.”

TD: How exactly does augmented reality work with this project?

BC: We’ve been experimenting with Apple’s ARKit, which is a developer framework that promotes an advanced augmented reality toolkit, so to speak. In their latest release, they made available some new capabilities. Now, your phone has the ability to detect not only horizontal surfaces but vertical surfaces, like walls that have paintings on them for instance, as well as images. We’re able to leverage those capabilities to detect the space around and layer in the paintings that would have otherwise been on display at the museum. All the user would need to do is lift up their phone and the magic happens.

TD: To my knowledge, you’ve started out by recreating two of the paintings; is the ultimate goal restoring all 13 for the museum?

BC: Our first tests were basically going through and seeing if this even works and to push the technology to its limit in a space that’s dimly lit and has all sorts of different optical challenges. Since then, we've been talking amongst ourselves to see if it makes sense to approach the remaining 11 works or just focus on one of the galleries, but that’s all to be determined.

TD: So for now, what has the response been like with the two pieces you’ve done so far?

BC: Even in the basic instances of going to the museum and testing it out, people would surround us and ask us, “Are we seeing what we think we’re seeing?”, “How are you doing this?” and “What’s going on?” People would come up and say things like, “Woah, I didn’t realize that’s what the stolen paintings looked like," so if they weren’t amazed by the technical capabilities of seeing the stolen paintings back in its frame, they were mesmerized by things like, “Wow, that’s what the painting looked like; I had no idea." So, on both sides, it’s been interesting to see people respond to being able to see something that doesn’t typically exist anymore in that space.

TD: Do you envision being able to get to the point where anyone would be able to come in and use the technology to see the paintings?

BC: That remains to be determined. We haven’t really laid any concrete plans around making this publicly available in the App Store, so it’s something that may or may not surface beyond the experimental stage. There has been a huge swirl of interest around the project both locally and internationally. It’s been interesting to see the conversations that have been started, whether or not it becomes something everyone can use on their visits.

TD: For the immediate future, what do you and your team want to work on with the augmented reality technology or just in general?

BC: We’re in the process of launching a new way to experience a museum, so as part of that we have five or six institutions around the country that are piloting this new technology that potentially annotates the gallery around you, so if you walk into the museum, every object and painting has a little pop-up that you can interact with to to access more audio and text. We’ll be rolling that out in the upcoming weeks, so we’re pretty excited about it in particular.

TD: Looking ahead to the next 10 to 20 years, what do you foresee with augmented reality technology and its capabilities down the road?

BC: In terms of 10 or 20 years, the way that we have historically accessed information has changed very quickly. Today, one could argue that your phone is your new interface to accessing information, and some of the things you’re hearing people talk about is how voice is the next interface with which we interact with the world around us. Something that’s interesting from a technology perspective is down the road, and there are some companies right now experimenting with it, is being able to control computing experiences using your brain. There is a company right here in Boston called Neurable, and they’re basically making a way for your brain to control what you see on the screen, so examples like that will likely become more common down the road.

TD: For any young computer science/engineering majors or anyone interested in working in technology here at Tufts, what should they take away from projects like "Hacking the Heist"?

BC: I think it’s about experimenting as much as humanly possible and putting out whatever you’re working on. A lot of people, especially in startup communities, there’s a phrase: “stealth mode." Breaking through the idea of stealth mode and showing people what you’re working on and talking about it and reaching out to people is crucial. The beautiful thing here is that we actually had some people with Tufts affiliation working on this project with us. Someone who was involved with some of the testing went to the School of the Museum of Fine Arts at Tufts, and we have a Tufts computer science major whose been helping with some of the testing as well. From a general perspective, it’s about experimentation. There’s so much innovation happening in Boston that if I were a Tufts student, I would be reaching out out to all of these cool companies and showing them all of the interesting things I’ve been working on and starting a conversation that way.

Editor's note: This interview has been edited for clarity and length.