PiX-C - Pictures: Express & Communicate
Parameswaran Raman
Georgia Institute of Technology
Aravind Ramanathan
Georgia Institute of Technology
Abhinesh Rao
Georgia Institute of Technology
Ellen Yi-Luen Do
Georgia Institute of Technology
Hiral Modi
Georgia Institute of Technology
ABSTRACT
In this paper, we describe the motivation behind, design and implementation of PiX-C: a system designed to help children in the autism spectrum express themselves and communicate. This is a visual input mechanism in which the user is presented a set of images based on the environment and context that he/she wishes to communicate in. The user then selects appropriate images and PiX-C helps generate a valid grammatical sentence out of it.
General Terms
Autism, Natural Language Generation, children with autism
Keywords
Keywords are your own designated keywords.
1. INTRODUCTION
PiX-C is a project aimed at individuals suffering from Autism so that their learning could be individualized and occur across all developmental domains. Autism Spectrum Disorders (ASD) are developmental disabilities characterized by difficulties with social interactions, impairment in verbal and/or nonverbal communication, and the development of repetitive, unusual, or highly-specialized interests. Individuals on the autism spectrum exhibit a bias towards using visual instead of verbal mental representations for various tasks. We believe that this problem could be exploited to develop a unique method of Visual Communication that children in this spectrum could make use of. PiX-C is a project which strives to achieve the same and make some positive impact on the lives of people suffering from Autism Spectrum Disorders.
2. DESIGN
PiX-C stands for Pictures, Express and Communicate and is a system that displays the child a set of images and lets him/her choose the ones which best convey what he/she wants to express. Based on these set of images, a Natural Language Generation algorithm generates the sentence which helps the child in communicating with others.
PiX-C also defines several contexts which a child could use, such as “School”, “Home”, etc. These images can be initially used to select the surrounding, or current environment that the child is in. For example: When the child chooses an image of a Home, the system would display pictures of her dad, mother, sister, other family members, dog, etc, while choosing “School” it would bring up other relevant pictures like 'teacher', 'white board', etc.
With a combination of pictures that represent people, actions and objects, the child will be able to express and communicate very effectively.
We envisage that alternate visual input devices such as the Adaptive keyboard developed by Microsoft, would be very useful for our system. Such a keyboard with its dynamically changing keys would help PiX-C display images relevant to the current content to the child. An prototype of how such a keyboard would look with our implementation is shown below. In this picture, the child would first select the context that he/she wants to communicate in – “Home”, “School” and so on. Based on that a cluster of images would be shown in the keyboard which correspond to set of persons, actions and objects that the child normally encounters in his/her daily life. The child would select a person, action and an object to convey his/her emotion as a sentence.
3. IMPLEMENTATION
Currently we have a web user interface prototype which helps us to demonstrate the idea in place and which does a reasonable amount of Natural Language processing behind the scenes on the inputs (images) it currently has. The flow chart below describes a sample use case of the system. Maya, is a seven year old kid in the autism spectrum, who is unable to express herself properly. Her brother recently got her a candy and she currently wants to email her granddad about it. In the PiX-C system, Maya chooses the home context to show up images relevant to her home. She then chooses pictures of her brother and candy to form the sentence Brother got me Candy. This sentence is then emailed to her granddad. PiX-C enables parents/caregivers to add new sentences and related image sequences. This could provide a personalization experience for the child.
The implementation of the Natural Language Generation module is currently a very basic one. A model is formed with the help of a training data set of around 150 sample sentences and phrases. These are used to construct a sentence graph with assigned sequence probability to phrases. When the user picks three images, the concepts corresponding to these images are pulled in
and a phrase generation sub-system works with the sentence graph to find out the sequence of phrases with the maximum probability. This sentence is then shown to the user. It has been observed that while most of the sentences are grammatically correct since they are based on the learning from the data set provided, some of them have semantic weaknesses. However, this is not a limitation of the PiX-C system and the semantic accuracy could be improved with more number of training samples.
The figure below shows a sample screenshot of the PiX-C web system. The child here selects images of a boy, eat and food to convey the message “The boy eats dinner”.
4. FUTURE WORK
The PiX-C system can be enhanced to provide audio cues which can augment the visual input mechanism. Subsequently, the system can be evaluated with users to gather feedback and improve it further.
Future work would also include evaluation and user studies using the prototype we have developed. The data collected from these would help us determine the success of our new visual input method.
5. ACKNOWLEDGMENTS
Our thanks to Prof. Ellen Yi-Do, GVU Center, Georgia Tech for making the project and research great fun and to all the project members for the initial brainstorming sessions to come up with the project design and subsequent implementation.
6. REFERENCES
[1] Fact Sheet at the Center for Autism and Developmental Disabilities, John Hopkins School of Public health (http://www.jhsph.edu/cadde/Facts/autism.html).
[2] Visual thinking in Autism, Design and Intelligence Lab, School of Interactive Computing, Georgia Tech (http://www.dilab.gatech.edu/vita.html).