AR Graffiti

Background

In the spring of 2013, Andrew Bueno and I were third-year students in the Bachelor of Computer Science and Arts program at Carnegie Mellon University. We were taking a class called Interactive Art and Computational Design under professor Golan Levin, and volumetric filmmaker James George was an artist in residence in Golan's studio.

James was developing a technology at the time called the RGBD Toolkit (now depthkit), whereby an artist could laser cut a special mount to rigidly join a Microsoft Kinect to a DSLR camera. Using James' toolkit, we could align the Kinect depth data with the DSLR color data to create these beautiful sort of 2.5-dimensional volumetric videos.

A sample output frame of the RGBD volumetric pipeline

The Tech

In February 2013, I approached James with the idea that volumetric video might be really cool to render in AR on a mobile phone. (In 2013, this was a fairly novel idea!) I built a prototype in the next two weeks, and although I could only render the mesh stream a few times per second, I felt like I was onto something.

In March, things picked up. James had shared the RGBD toolkit with Aaron Koblin (who would later found Within), and Aaron wrote a WebGL shader that could render the mesh stream much faster than my Processing-based prototype. I teamed up with Andrew Bueno, and we ported Aaron's shader to GLSL, allowing us to render that much higher quality mesh stream in Unity and export a mobile app. We were the first developers to render RGBD images on mobile! With the tech done, it was time to make art.

The Art

Andrew and I created five augmented reality graffiti installations around Pittsburgh, and our resulting video got some traction online. Several journalists picked up the story, and we were reblogged maybe a hundred times, which was just surreal. A local music and arts festival approached us about some follow-up work, and we gladly said yes.

We built five more AR installations in 2014, this time for the VIA 2014 Festival in collaboration with artist Vince McKelvie. Vice picked up the story, and the festival experience was amazing for both of us.

In the end, Andrew and I moved to different cities after college graduation, so we closed the AR Graffiti chapter of our lives.