Read the World Around You

A vision experiment with Shiseido and Google

Introduction

Project Theia

Project Theia is an ongoing collaboration between Google and Shiseido to explore innovative ways technology and cosmetics can enhance the lives of the visually impaired. Braille Nails is the first of these explorations.

Braille Nails

Inspired by Japan’s fascination with nail art, Braille Nails combines the latest in material design and computer vision technologies to make this decorative artform more expressive and useful for the visually impaired.

Concept Film

Throughout the Braille Nails project we held interviews and focus groups with visually impaired people and experts in accessibility to better understand our users, their daily pain points and ultimately their needs. It was through this research and their feedback that we began to develop Braille Nails, leveraging technology to solve real-life problems.

How Does it Work?

The Braille Nails kit consists of two parts - a set of ten distinctive nails and a companion camera-badge.
The nails double as a marker, allowing the camera to track the user’s hands movements even in visually complex environments.

Self-expression for all

We’ve combined 3D printing, textural design and distinctive colours to create a range of nail designs that evoke different moods through both sight and touch.

Technology as a fashion accessory

The camera-badge is made from the same materials as the nails themselves, to compliment the wearers outfit, not distract from it. This also means that the badge can be customized alongside the users nails at a nail-salon, allowing infinite possibilities for self-expression.

Image of nail and camera device

Reach out and touch the world

Different hand gestures trigger different functions, extending the users understanding of their environment beyond the cane - for example:

Hold your thumb up and Braille Nails will describe the scene to you.
“There is a straight path in front of you with two benches to the left”

Point at the sky and it will give you a weather report.
“It’s cloudy and might rain soon. You may want a coat”

Hold your hand up to get a situation report.
“You are standing on Komazawa Dori opposite Ebisu Station in Ebisu, Tokyo”

You can even mark your favourite locations with a thumbs up gesture
“I’ve added this location to your list of favourite places.”

Current Status

Prototype

Braille Nails is a working prototype, but there’s still a lot to do to get it ready for mass production.

Coming Soon

The next stage of Braille Nails will involve implementing a wide range of additional features.

A Wider Range of Nail Designs
Self expression is core to our design process. We want to expand the range of available designs through collaborations with famous nail artists, customization, and user input.
Improved Camera Device
The resolution of the camera device will be improved to increase accuracy. We will also embed a mobile data connection so the device is truly independent and handsfree.
Choose Your Companion
Currently, the device is using a standard text-to-speech engine. We will customise this to provide the user with different voices and personalities to choose from.
Four tactile nail designs
We have produced a range of four different tactile nail designs for field testing. Each was selected by a panel of visually impaired users.
Japanese Language Responses
By training it for the local environment, we will also re-label the training data to ensure it is more useful across English and Japanese. We are also exploring passing the results through the Google Translate API to allow it to respond in any language.
Gesture Function
Gesture recognition expands the functionality of Braille Nails beyond image recognition allowing it to provide additional utility, such as text recognition, weather, location information and much more.
Open Source
Braille Nails has been built upon multiple open-source technologies, including Google Tensorflow. We will make the project available for other developers to build upon.

In Collaboration with

Braille Nails is an experimental collaboration between many different companies.

Project Members

  • Shiseido
  • Masato Kosukegawa
  • Nodoka Kagaya
  • Junko Koike
  • Satoko Tomizawa
  • Tomoaki Yamura
  • [Shiseido Global Innovation Center]
  • Yuko Nakanishi
  • [Shiseido Beauty Creation Center]
  • Miyako Okamoto
  • Chizuru Fukuike
  • Ryoji Okazaki
  • [CSR · Communication Department]
  • Tomoko Enji
  • Misa Okuzawa
  • Google ZOO
  • Gene Brutty
  • Tim Seddon
  • Rosa Uchima
  • Yosuke Suzuki
  • Joe Fry
  • Bryan Tanaka
  • HAKUHODO i-studio
  • Shunsuke Nishihara
  • Yuto Kumon
  • Yukiko Sano
  • Saki Kato
  • Yuta Ikeno
  • Takuya Asahida
  • Shoko Miyake
  • Yota Odaka
  • Jyutaro Mochizuki
  • HILLTOP
  • Yosuke Morishita
  • Bontemps Justin
  • Yohei Kamei
  • maxilla
  • Takahito Matsuno
  • Seiya Suzuki
  • DIGITAL ARTISAN
  • Mayumi Ide
  • National Institute of Special Needs Education
  • Susumu Oouchi
  • workcam
  • Kenko Nakamura
  • coconoe
  • Yoshihiro So
  • Takashi Okada