BioVisualSpeech: An Interactive Platform for Speech Therapy with Visual Bio-feedback

Research teams: CMU); (NOVA.ID.FCT); INESC ID)
Organizations:  CHLC); ESSA); HDE); (SCML); Voice Interaction
Main Research Area: Speech therapy; interactive environment; multimedia analysis; visual bio-feedback
CMU Portugal Program webpage:
Principal Investigators: Sofia Cavaco (FCT/UNL; NOVA.ID.FCT); Maxine Eskenazi (CMU; LTI)

This project proposes to research natural and multimodal human computer interaction mechanisms for providing bio-feedback in speech and language therapy, through the use of serious digital games. Traditional behavioural treatment approaches for speech and oral motor disorders can benefit greatly from visual biofeedback, for patient motivation purposes, for tracking orofacial motion, and for evaluation of the patient’s speech productions. These instruments are useful, in the sense that patients can watch themselves as they pronounce the sound or do an orofacial movement or activity, and this way better associate their intention to their actions. This project proposes a novel approach to include biofeedback in speech-language therapy by providing the patient with a visual self-monitoring of his/her performance combined with a reward mechanism. This approach offers significant benefits to the patient as well as for the speech and language therapist (SLT). Our approach provides the SLT with a tool to gather and evaluate the patient’s speech progress, and to measure the coordination of the patient facial muscles in terms of orofacial motion.