publication

press

context

The history of hand shadow play is nearly untraceable which was prevalently practiced long before the existence of Greek shadow show Karagiozis or Chinese shadow puppetry Pi Ying Xi. It is a prelinguistic and transcultural form of storytelling that entertains, educates, and instills moral values in the younger generation; it is also a stimuli of creative production, by mimicking the things we see, and by telling the stories we relate. Narratron, in that sense, has deeply embedded AI into this intelligent collective effort of hands, eyes, and brains as a true “fairytale copilot”. Its nature of multimodality that combines visual, auditory, tactile, and textual I/O, supported by the collaboration of  LLM, image classifier, speech synthesizer and diffusion models, demonstrates how seamlessly we are able to make bodily interactions with AI. Through bridging the digital and the physical, we are now bridging the ancient and the future.

‍

user experience

The user's experience with Narratron begins with the startup screen, which creates a serene and focused ambiance, setting the stage for an immersive journey. As the user turns on Narratron, they are greeted by the instructions, preparing them for the captivating experience that lies ahead.

Once the startup screen fades away, the user is free to explore and play with their hand shadows in any way they wish. They can experiment with different shapes, sizes, and movements, allowing their creativity to flow unrestricted. Narratron's camera captures the intricate hand shadow shapes created by the user.

The captured hand shadow shapes are then analyzed by AI algorithms integrated into Narratron. These algorithms interpret the hand shadows and translate them into animal keywords. The keywords serve as the foundation for the next step of the process: generating a complete story.

To generate the story, Narratron employs the powerful GPT-3 language model. The animal keyword identified from the user's hand shadow is processed by GPT-3, which harnesses its natural language processing capabilities to create a cohesive and imaginative narrative. The AI-generated story seamlessly combines plotlines, dialogues, and descriptive elements to deliver a captivating storytelling experience.

While the story is being generated, Narratron simultaneously generates a corresponding image using Stable Diffusion that represents the animal associated with the user's hand shadow. This image is then projected onto the surface, adding a visually immersive component to the experience. The vivid and dynamic images enhance the user's connection to the narrative, further immersing them in the story world.

As the story is projected onto the surface, the user is enveloped in an audio-visual storytelling experience. The projector synchronizes the visuals with an audio narration, bringing the story to life through voice acting, sound effects, and music. The combined audio-visual elements create a multi-sensory experience that engages and captivates the user's attention.

To initiate the storytelling experience and progress to the next chapter, the user interacts with Narratron by spinning the knob, reminiscent of vintage movie projectors. This physical interaction adds a sense of nostalgia and tactile engagement to the overall user experience.

The spinning of the knob serves as a natural and intuitive way for users to navigate through the story. Each rotation of the knob signifies a progression, unlocking a new chapter and revealing fresh elements of the narrative. The user's active participation in this process creates a sense of agency, allowing them to dictate the pacing and flow of the storytelling experience.

software stack

Teachable Machine: To enable hand gesture recognition, Narratron utilizes Teachable Machine, an online platform that allows for the training of custom machine learning models. The hand gesture recognition model is trained using Teachable Machine's interface, which captures and analyzes various hand shadow shapes to recognize and interpret user movements accurately.

‍GPT-3.5-turbo: Narratron employs the powerful GPT-3.5-turbo model developed by OpenAI for generating dynamic and engaging stories. The model is trained on a vast corpus of text, including literary works, articles, and other narrative sources. To specifically train the story generation model for Narratron, The Ugly Duckling story might have been used as a training dataset, providing a foundation for generating narratives based on user inputs and animal keywords.

‍Stable Diffusion: For image generation, Narratron utilizes the Stable Diffusion algorithm. This algorithm leverages deep generative models to produce visually appealing and contextually relevant images. The image generation model has been trained using a custom dataset created specifically for Narratron. The training data likely includes various animal images and visual elements related to storytelling.

Custom Model Training

‍Replicate: In order to train the image generation model, Narratron employs the Replicate framework. This framework allows for the development and training of custom deep learning models. Using Replicate, the image generation model is trained on the custom dataset, incorporating techniques such as conditional image generation to ensure that the generated images correspond to the user's hand shadow animal keywords.

Backend
Data Processing and Integration: The backend of Narratron handles data processing and integration tasks. It receives and processes user input from the hand gesture recognition model and passes it to the story generation and image generation components. It also manages the integration between the different components of the system to deliver a seamless and synchronized user experience.

context