Gesture Research

Quick Access Links

Research

We are investigating the multimodal alignment across hand gestures, body movements, prosody, intonational phrases, morphosyntax, discourse, and pragmatics.

Part of this work involves creating a gesture labeling system that carefully identifies features that are mutually exclusive. Our system gives particular attention to the kinematics of gestures, separating the physical movement from the interpretation of that movement. As we focus on piecing apart the kinematic features, we bear in mind how this system would accommodate automatic feature tracking. As Artificial Intelligence and Machine Learning technologies advance, this system envisions what automatic gesture labeling would look like and how it would benefit researchers.

Collaborations

Working with Pilar Prieto, Patrick Rohrer, Nuria Gilbert, Ingrid Vila, Julia Florit (Universitat Pompeu Fabra in Barcelona) on M3D labeling system including a pragmatic labeling scheme

Publications

• Ren-Mitchell, A. and Shattuck-Hufnagel, S. (2022), Looking for patterns: Multimodal exploratory visualizations. Presented at the 9th meeting of the International Society for Gesture Studies, Chicago Ill., July 2022
• Click to view presentation

• Shattuck-Hufnagel, S. and Ren-Mitchell, A. (2022), Gesture rests as indices of prosodic structure in academic lecture style speech. Presented at the 9th meeting of the International Society for Gesture Studies, Chicago Ill., July 2022
• Shattuck-Hufnagel, S., Ren, A., Mathew, M., Yuen, I., Demuth, K. (2016) Non-referential gestures in adult and child speech: Are they prosodic?. Proc. Speech Prosody 2016, 836-839. (link)
• Shattuck-Hufnagel, S., Ren, A., (2012) Preliminaries to a Kinematics of Gestural Accents. Presented at International Society for Gesture Studies 6 San Diego.
• Shattuck-Hufnagel, S., Ren, A., Tauscher, E. (2010) Are torso movements during speech timed with intonational phrases? Presented at Speech Prosody 2010.

Team