live sketch header.jpg
Source: ACCD IMMERSION LAB

Live Sketch

Time Line

Jan - May 2020

Project Type

Scope

About

Team of Two
(John Ma, Ian Sterling)

Machine Learning
Augmented Reality

User Scenario

UX / UI Design

Lens Prototyping

Snap Inc. Sponsored. The Future of Storytelling with Augmented Reality is a transdisciplinary studio that will explore the next phase of narrative design using cameras, machine vision, machine learning, augmented reality, and game development software.

What is Live Sketch?

Explores how the machine vision models interpret a simple human input and context to author unexpected AR content. 

 

Users can share their sketch with friends for them to add to and view in the network. 

 

Generates unique 3D objects based on the sketch inference, as well as, the objects, spaces, and people recognized in the scene.

Screen Shot 2020-08-25 at 10.58.05 PM.pn

Stories

An account of imaginary and events told for entertainment.

An account of past events in someone's life or in the evolution of something.

giphy.gif

Inspiration

Evolution of the English Alphabet

Simple letterforms evolve over time to have a deeper meaning for humans

How would something like this work for the future of machines? what meaning would they derive from simple drawings?

Screen Shot 2020-08-25 at 10.38.02 PM.pn

"What if the machine could create deeper meaning out of simple gestural illustrations which humans created? How could this evolve over time?"

Screen Shot 2020-08-25 at 11.07.50 PM.pn

Research

Machine Learning:

Generating 3D Assets from pre-existing 2D Data Sets

Screen Shot 2020-08-25 at 11.10.47 PM.pn
Screen Shot 2020-08-25 at 11.11.00 PM.pn

We were able to signify the technology in which machine learning can identify a 2D image into a whole new 3D Assets.

System Diagram

Screen Shot 2020-08-25 at 11.18.16 PM.pn

Click to Enlarge the image

Interaction Mockups

Screen Shot 2020-08-25 at 11.22.54 PM.pn

Asset changes based on context and environment: Sketch becomes a surfboard on the sand and when users moves the camera to the ocean it transforms into a shark.

Animation exploration of how assets can morph between one another in real time.

Screen Shot 2020-08-25 at 11.25.41 PM.pn

Making it Social: Sketches can be shared but will always be different based on what’s in the environment

Live Sketch Video