Interactive Storytelling for Children with Autism Adhish Bhobe College of Computing Georgia Institute of Technology Atlanta, GA 30332 USA adhishbhobe@gmail.com Andrew Harbor College of Computing Georgia Institute of Technology Atlanta, GA 30332 USA andrew.harbor@gmail.com Sanika Mokashi College of Science Georgia Institute of Technology Atlanta, GA 30332 USA sanikamokashi@gmail.com Amol Shintre College of Computing Georgia Institute of Technology Atlanta, GA 30332 USA amol.fhc@gmail.com ABSTRACT “Kinect the Dots” is an application designed to help therapists, teachers and parents of autistic children engage the child in an interactive storytelling experience. It ties the power of traditional storytelling with the full body interactions enabled by the Kinect. Its novel interaction model aids innumerable uses in therapy and promotes social and peer interaction. ACM Classification: H5.2 [Information interfaces and presentation]: User Interfaces. - Graphical user interfaces. General terms: Design, Human Factors What is Autism? Autism is a complex neuro-developmental disorder. It affects 1 in a 110 children in the US today. It is manifested as speech and language delay, repetitive stereotypical behavior, challenges with motor planning and difficulty with social interactions. Different people with autism can have very different symptoms. Autism is thought of as a “spectrum” disorder, a group of disorders with similar features. The various interventions and strategies used to deal with autism include Applied Behavioral Analysis (ABA), Speech Therapy, Occupational Therapy and Physical Therapy. Storytelling Storytelling plays a big part in therapy for autistic children. It is used in conjunction with songs and the children are encouraged to enact the parts of different characters from stories. Storytelling helps them master language skills, improves their listening skills, increases their attention span and develops their curiosity and creativity. Keeping a child engaged while telling them a story isn’t easy though. Children with autism especially tend to get distracted easily. The teachers from the Lionheart School (a specialized school for children on the autism spectrum) use a lot of props, songs and other activities to keep the children engaged. A teacher from the school described her experience with the story of Jack and the beanstalk. To explain the concept of climbing up they made the children pull a rope down and asked them to imagine Jack climbing up on the beanstalk. One of the teacher commented that the students can’t visually see the character Jack climb the beanstalk and thus couldn’t understand the action or connect very well with the character where they were teaching up and down movements to the children using the story. Digital Storytelling A lot of different approaches use digital interactive storytelling to engage children in stories and let them play a more active part than what traditional storytelling would allow. IPad apps such as the Fantastic Flying Books of Mr. Morris let children (and adults!) be a part of the story by flying through the magical world, tumbling through a storm and even learning piano. The TellTable system from Microsoft Research lets children author stories collaboratively using Microsoft Surface. “People in Book” from Nokia Research is a system that supports children and longdistance family members to act as characters in children's storybooks while they read stories together over a distance. Approaches like these lack in utility for children with autism on two levels: (1) Trying to keep them engaged in such technology can be a challenge sometimes (2) It’s not as useful for children with autism, since they don’t use their full body, often not speech, and they don’t interact with their peers or teachers much. The use of Kinect On further research,we realized that the Kinect has the potential to create a big impact in therapy for children with autism. A lot of children with autism seem to naturally take to the Kinect. The Lakeside center for autism uses the Kinect for multiple goals. There are some specific goals of therapy that the Kinect can address – as the Lakeside center for autism has been doing These are very encouraging results but they had to adapt their goals to fit the technology which was limiting the extent to which the technology could aid the therapy. However, we decided to go a step forward. Our solution is a customized interactive storytelling application using the Kinect that parents, teachers and therapists can use. Use Case At a very high level, our solution lets a teacher or a parent tell a story in a more engaging way to a child with autism. Unlike traditional storytelling, which is to a large extent one-way communication, our application lets the child be an active part of the process in the following ways: --through controlling the characters via his/her body movements --coloring different scenes from the story and --even select components for particular scenes. The entire experience of illustrating the story through kinect is both engaging for the child as well as intuitive. Architecture and Technologies --We are using C# for our code base --Microsoft’s Kinect SDK Architecture Diagram: https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/architect ure_diagram.png?attachauth=ANoY7cryu2UOLNen4gp7A E_KcTzirJSDeIIEsX2Za49eLu_VhsLfc42W8_IEf4TalzrCl VVJScRlGJapsaxS1B018EZBGGsFPW2WGF0e5K78rqwMvhfmwuv 5qPe3ytt4s96ngvQRAawmF1phxYeS3w2P2jzJP3cqlGyaZ vbQa4Hql8bz9iE4zNI_GiwBNXuVkot8CZ43XDzy5V6b0 W9U_k5956hIT_K7Zxtrk3zvNapeHlOY9XJr2Q%3D&attredirects=1 The entire code base is based on C#. Traditional Digital storytelling consists of sounds and pictures. Through Kinect we extend this engagement further. Through the inherent skeletal movement tracking we set trackers to track the movements concerning selection required for selection of colors and scenes/images. Trackers are set for tracking user movement. Movement pertaining to foot movement is tracked for particular scenes. On initiating the tracker, the characters on the scene are corresponding set to move based on the user’s move- ment. Trackers are set for coloring, motion detection (climbing action) and drag/drop action. Evaluation and Results We evaluated our system at the Lionheart School for children with Autism. We targeted the use of our system against the most common behavioural characteristics of autistic children. These included 1) Motor skills, Full body movement, Gross motor control 2) Speech 3) Communication,Responsiveness and Situational Awareness We found the following results (seen in our videos) regarding these characteristics: The child was able to effectively show responsiveness to the scenes and the happenings in the story as well as communicate with the teacher and respond quickly to the instructions and requests made by the teacher. In one instance, the child was required to color a black and white image of the Castle using the colors on the screen. The child was able to pick the appropriate color matching the Castle and color the Castle with minimal assistance from the teacher https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/colorfina l.png?attachauth=ANoY7cqBxHDhVcv6l6ZswpWoY8k0e 1K8gn7vrLZSWND1dRuw3tCQLVtdJ9LqTFxLU8tAXmy oksjR5JKUuLWnw4zlLBC0mCZLUv3Ofs522QDWaHVTQVrOdH1ihUr0ngRbsqVX4 GttVy2U-Py6cpRb0lsYZ84onMFP4FfmUvYx4guNdK0wN1iUjepl0wGy5nngPXTATUlpp8WMMSG5TRXr8dgCeaOgrKw%3D%3D&attr edirects=0 Certain scenes required the child to use leg and hand movements. With minimal or no instructions from the teacher/therapist, the child was able to intuitively use his leg movements to play with the characters on the screen. In one of the scenes the child was required to move his legs to initiate a corresponding movement of the character ‘Jack’ up the beanstalk on the screen. The child was able to do this with minimal assistance or prodding. https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/climbfina l.png?attachauth=ANoY7crPBKnWrMrP7c1RSZ9tvGdcpMa4PfhjGBulKrT1jeTRkBG04APB6BPmJfN KvxWUhOTny6aBcUFVrjYNoUhVcvJhEtxtzrlBGg_HEN lLTwnj6sqmszah45ROF5jyATpxqk3RdACDt_UGXTz8X PxNcp1YuHla6maJ08Ogp5JjSJRlnPyUs3cNVroR3JvevvozE_Tz0nJfzp Z6puG-uxA13HdK1ZWQ%3D%3D&attredirects=0 There were instances where the child was required to make decisions by picking one of the scenes based on the theme of the story. He was required to drag and drop the appropriate image. This required him not only to be decisive but also use motor skills to bring the most appropriate image to the forefront. We saw that he was able to do this no intervention from the therapists/ teachers. Before: https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/dragdrop final1.png?attachauth=ANoY7cqUYtT6WmOJAcMmQJxX Zg7si8tUH0FsoMSkFrT4P1sEFN22QNhnZ_rWHR1P2Pe767mS841tykV0l0SEYbUt17GU1 UDQWmWrTjOM9qul32kNgH3mL9LCy2MiiW9dsmo3QAlMKdvwGYO5nB8H3iSPeAQ80YUybfr5NZmIMBJ5WgeFT9nJ 3XgV6WsO8xaFIzXcC4gwJBVGqb2jnAf75Q5htxu5jJA% 3D%3D&attredirects=0 After: https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/dragdrop final2.png?attachauth=ANoY7crMGxL30lY2UK0cpMzfooZ uo5wO9VXkSNT0H6ojz7a29x3QuHHgMymSv2cX0IV1_ z3M2BY3BzftQyRuZFkyti946R8be3H_LUMxRYFegmY v4M1NGugRpx8gB3ozi34XyWFGGH9Fc7AWCpjWTNk Zv_PBHGb72FcRyFIbDjffS8FYDSmh4X5zeUiP0f9Z1M Dvnq2E138eLqdKUzAKjHBiF7D7pHcszw%3D%3D&attr edirects=0 Other miscellaneous findings included the improved engagement of the child with the story as well as the teacher while the story proceeded. We will continue evaluating the system with more participants (every child with autism is different). The measures that we will use include: how often the children respond appropriately (responsive communication), how often the child asks questions or engages with the teacher on his or her own (initiation of communication), instances of use of speech, instances of full body movement. Specific measures for speech and gross and fine motor control can be used for specific children who have problems in these areas. Business case The healthcare industry is growing from $100 billion today to $150 billion in the next 2 years. Families spend anywhere from $6000 to $50000 per year on autism. Also the cases of autism have increased significantly over the years. Autism Statistics: https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/graph.pn g?attachauth=ANoY7cpO0C_rRTkCDTMpirchKa0SiKwz zkocB9FNlkXe1JLcgGWXYE6EbtxL9y2grwRAoINuciaK OhVozWe5mCNJ6oyR_In131Cve3L4ZLT9SkOBHbhKgt67lHAX1yfym6ZwT0WyOwud_tDRcpLXAoR7VKY6 7EnIz0og13rJ7VpWdJa3G330dPalLK1Qk4sFUbTO19c1_ Q1EZzTYSeHrZQGyQbmj0B7aw%3D%3D&attredirects= 0 Current scenario lacks any specific and efficient solution to tackle autism and its effects. As scientists and medical researchers keep looking for cures, it is critical that we embrace new technologies to help support the therapists today. Storytelling is one of the constituents of therapy aids in tackling autism but faces several obstacles.Kinect and storytelling together can be an effective combination to tackle autism. It provides a customisable and low cost method in combating autism. Pre- designed stories customised to various levels and kinds of autism can be useful and easy to use by therapists and parents alike and intuitive for autistic children. We plan to package the software primarily for educational, research and therapy institutions that are looking for direct and simple ways to collect results and evaluate them. Kinect the dots will do this without significantly increasing the burden on parents. Growth Strategy -- Currently our ecosystem consists of software and content developers along with our consumers which includes parents, therapists as well as teachers. Software developers are involved with developing and maintaining the system. Content developers are involved with developing the story content and its constituent parts. The consumers include therapists, parents and teachers using the software in educational and therapy sessions. https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/currentsy stem.png?attachauth=ANoY7cpHfrBO1osIoVLOalD5uZPMmz2_YymTCwIM_6bjhdXxIb_Dy3P47xtjdrX9pzhBP2 8iUkSYTaS3LksbgR5oFEFHcbs_XbhciWvWgsOwaQtK4tbesgmTtSWjNiW7v1s5v q2IoCgDfDEoWlubfuIwu2H3TMRcvwq85Bo18VS6cSRn 9aQoBxgjPLortVslS3v04WQ0CPEtFXM4qRkksfvo0bSQo JVw%3D%3D&attredirects=0 -- In the future we plan to transform this into a platform and a distribution channel which can be used by a network of developers. The content can be developed by content developers who can then contribute to and add to the existing library of content.This content can then be customized and used by our consumers based on their needs. https://3600923478673570363-a-1802744773732722657-ssites.googlegroups.com/site/kinectthedots/images/futureeco system.png?attachauth=ANoY7cpDMbUEUS1SZWq5gnexw_w1XTbHb942_HhAMYoDpwcUmn4Zw3zNgG LE6pWi2Rk2_hnbizLVwaimk62PaVmMa0tCSVxKLCqk Hsu5UdU1GMEDP9WGMr5u2vtb9GFLlisGXmbusttLZIeZxh2iEMIzqg6VGqkKhtHuksfyVGdu6N9qhiORGD0 mgf2cUDHa_IXSHMtnVtvNuCMpbfS3AQ_UBOxz7uEg %3D%3D&attredirects=0 REFERENCES 1. Interventions and therapies used in treatment of autism: http://www.autismcenter.org/treatment_in tism: http://www.autismcenter.org/treatment_inter ventions.aspx 4. Children and Autism : http://www.youtube.com/watch?v=bG0vko7GHD Q&feature=related 5. What is Autism? Do you know the Signs? : http://www.youtube.com/watch?v=lbXjWcX9kQ&feature=related 6. Lionheart School: http://www.thelionheartschool.com/ 7. Telltable from Microsoft Research: http://research.microsoft.com/enus/projects/telltable/ 8. “People in Books”: Sean Follmer, Rafael (Tico) Ballagas, Hayes Raffle, Mirjana Spasojevic, and Hiroshi Ishii. 2012. People in books: using a FlashCam to become part of an interactive book for connected reading. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12) 2. Importance of storytelling for young children: http://www.littleonesreadingresource.co m/childrens-stories.html 3. Lakeside Center for Autism (Video): http://www.youtube.com/watch?v=uuP6d42hK8k Lakeside center: http://lakesideautism.com