top of page
live sketch header.jpg

Live Sketch

Time Line

5 months

Project Type



Team of Two
(John Ma, Ian Sterling)

Machine Learning
Augmented Reality

User Scenario

UX / UI Design

Lens Prototyping

Snap Inc. Sponsored. The Future of Storytelling with Augmented Reality is a transdisciplinary studio that will explore the next phase of narrative design using cameras, machine vision, machine learning, augmented reality, and game development software.

What is Live Sketch?

Explores how the machine vision models interpret a simple human input and context to author unexpected AR content. 


Users can share their sketch with friends for them to add to and view in the network. 


Generates unique 3D objects based on the sketch inference, as well as, the objects, spaces, and people recognized in the scene.

Screen Shot 2020-08-25 at 10.58.05


An account of imaginary and events told for entertainment.

An account of past events in someone's life or in the evolution of something.



Evolution of the English Alphabet

Simple letterforms evolve over time to have a deeper meaning for humans

How would something like this work for the future of machines? what meaning would they derive from simple drawings?

Screen Shot 2020-08-25 at 10.38.02

"What if the machine could create deeper meaning out of simple gestural illustrations which humans created? How could this evolve over time?"

Screen Shot 2020-08-25 at 11.07.50


Machine Learning:

Generating 3D Assets from pre-existing 2D Data Sets

Screen Shot 2020-08-25 at 11.10.47
Screen Shot 2020-08-25 at 11.11.00

We were able to signify the technology in which machine learning can identify a 2D image into a whole new 3D Assets.

System Diagram

Screen Shot 2020-08-25 at 11.18.16

Click to Enlarge the image

Interaction Mockups

Screen Shot 2020-08-25 at 11.22.54

Asset changes based on context and environment: Sketch becomes a surfboard on the sand and when users moves the camera to the ocean it transforms into a shark.

Animation exploration of how assets can morph between one another in real time.

Screen Shot 2020-08-25 at 11.25.41

Making it Social: Sketches can be shared but will always be different based on what’s in the environment

Live Sketch Video

bottom of page