Background
Advances in machine learning, artificial intelligence, and embedded computing are revolutionizing human-computer communication and cooperative learning. Cooperative learning applications include computer-aided training of military personnel and heavy equipment operators, performance coaching, and tutoring American Sign Language (ASL) for individuals who are deaf or hard-of-hearing (DHH). In addition, distance learning options are becoming increasingly necessary. Thus, a need exists for accessible distance learning options for DHH students and learners of gesture-based tasks.
Invention Description
Researchers at Arizona State University have developed a gesture-enabled learning environment that includes a gesture-matching model which can monitor and provide direct feedback to a learner of gesture-related tasks by comparing gestures performed by a learner with those of a model tutor. Scalable and adaptable, the gesture-matching model accommodates a variety of gesture styles and can be instantiated within the system to provide direct tutor feedback, assessments, and automated translation services within the learning environment. Virtual reality capabilities are also incorporated to provide an accessible distance learning environment.
Instructors, students, and/or interpreters can access the learning environment online through virtual reality or conventional systems to participate in a lesson. The gesture-matching model can be instantiated in the form of an AI-enabled tutor, allowing students to interact using gestures. This aspect can be especially relevant to applications involving the teaching of gesture-based tasks including sign languages.
The gesture-matching model can be used to instruct teachers, interpreters, and/or students on gesture-based vocabulary related to the learning material, particularly for technical terms previously unknown to deaf or hard-of-hearing (DHH) individuals. In this manner, the learning environment can be made accessible to DHH students in physical or online classes by enabling a remote tutor to physically observe and more effectively translate class communication including lectures and conversations.
This innovation is covered by U.S. Pat. No. 11,307,667.
Potential Applications
• Remote learning
• Gesture-focused virtual education
• Deaf and hard-of-hearing education
• Health and collaborative rehabilitation
Benefits and Advantages
• Enables deaf or hard-of-hearing (DHH) individuals to effectively engage with a virtual learning environment, especially when highly technical material is involved
• Allows gestures to be better communicated in a distance or virtual platform
Related Publication: Personalized Technical Learning Assistance for Deaf and Hard of Hearing (PDF)
Research Homepage of Professors Sandeep Gupta and Ayan Banerjee