This SCG gesture coding manual describes our methods for gesture labeling, labeling them for their movement, so that we can explore the relationship between the kinematics of gesture with: prosody, intonational breaks, morphosyntax, discourse coherence, and pragmatics.
Expand and collapse the toggles below to show and hide more information, labeling notes, and extended support for labeling each features.
We use ELAN, a video annotation tool created by Max Planck Institute, to label the co-speech gestures in our video samples. To download and learn more about how to use ELAN, check out http://tla.mpi.nl/tools/tla-tools/elan/. Some tips for using ELAN are provided below.
Our coding process alleviates the decision-making burden on the labeler by focusing on a single cognitive task at a time. This means that a labeler may have multiple passes even when labeling a particular dimension, such as the gesture stroke. For each pass, the labeler does only one thing. The broad task of identifying the stroke is not the same as the detailed task of defining the stroke boundaries. Some labelers may prefer to organize their work flow in other ways, and that is okay. Acknowledging and separating out the tasks by what kind of mental work the labeler is doing can have implications for future automatic labeling of gestures.
Most of the features described in this labelling manual are kinematic, and are labeled while the video is muted. This prevents the speech signal from influencing the labeler's perception of what is going on with the hands. So unless otherwise mentioned, these features are labeled with the video sound off.
Most videos are 30 frames per second. Be sure to restrict your annotation boundaries to video frames. You man see a still frame in ELAN in the milliseconds between video frames, but that is merely computed. There is an ELAN preferences setting that allows you to restrict your labeling to video frames.
Starting from an unlabeled video sample, we create a tier to chunk areas of non-gesturing. We then go back to the beginning of the sample and create a new tier for labeling more defined chunks of gesturing. These chunks are later refined into Perceived Gesture Groupings (PGGs). Next, focusing on these chunks, we roughly label for where the strokes are. Refining the boundaries of the stroke happens when labeling for phases.
Gestural strokes can appear grouped together. They may look like a set of repeated hits, beats, or something else. Novice labelers can identify perceived gesture groupings (PGGs) even when the boundaries between strokes may not be so clear.
When labeling PGGs, include the gesture strokes that appear to group together. As you do this, you will develop mental hypotheses about why and how you are grouping gestures. Save your ideas and notes for later discussion. The PGG annotation range does not have to be perfect, so long as all the gesture strokes that should be in a group are included in the range. The start and end boundaries will be later refined to include all the individual phases around the stroke.
If you feel that you need a hierarchy of PGGs, feel free to do so. We use PGG1 for the smallest groups and PGG2 for larger, higher-level groups (where a PGG1 is grouped with other PGG1s or with a single gesture). In our experience, a third level grouping, PGG3, may be found for some speakers and for longer videos (In one 30-minute sample, we found only 13 PGG3s).
Leave the annotation value blank. When you are done labeling the entire sample, use ELAN’s “Label and Number Annotations” tool to automatically label and number the PGG annotations. We often use this as an ID number to refer to the PGG when referring to the PGG. It is more convenient than referring to the time stamp.
Before looking at PGG reliability, SDG labels must be finalized.
Because perception of gesture grouping levels can vary slightly between labelers, we have two labelers annotate the same sample, with a consensus labeling round for the final annotations. These differences are usually about the level of detail, for example, PGG1 for one labeler may be a PGG2 for another labeler. Disagreements are only counted when the larger PGG does not contain all the strokes of the smaller PGG that it overlaps with. <INSERT EXAMPLE OF DISAGREEMENT, EXAMPLE OF AGREEMENT>
Following Kendon's widely accepted view, we label Gesture strokes and their related phases. Strokes are described as the peak of effort in a G-Phrase in Kendon 1980: “A phrase of gesticulation, or G-Phrase is distinguished for every phase in the excursionary movement in which the limb, or part of it, shows a distinct peaking of effort, ‘effort’ here used in the technical sense of Rudolf Laban (Dell 1970). Such an effort peak, or less technically, such a moment of accented movement, is termed the stroke of the G-Phrase.” The stroke phase is necessary for identification of the movement as a gesticulation unit. There can also be supporting phases around the stroke, including preparation, hold, and recovery from the peak. These phases are not always used.
The phases we label are:
Recovery is when the hand is in motion, moving to a rest position. Relaxed is when the hand is not moving, in a rest position. Relaxed cannot occur without recovery. This means that between two strokes, if there is a non-moving phase, the non-moving phase is labeled as post-stroke hold.
It can be difficult to label where gesture strokes start and end. The gesture may be too slow, too fast, or too blurry. For the case of the video being too blurry – our video framerate is 30 frames per second, and though that is much too low for automatic motion capture, it did allow us to use the frames where the hands clear up in the frames as boundary markers for the annotation label. "Clearing up" means that the hand has slowed down
(Image here)
Each gesture phase has its own tier. The annotation value is created with “Label and Number Annotations” tool under the Tier menu option. After all phases are labeled and reviewed, they may be combined into a single tier.
If you are able to use motion tracking, use the calculated velocity and acceleration to help determine the start and end of the gesture strokes. As for what to track, you can choose the point of the index finger or the center of the palm.
Stroke-defined groupings are similar to Kendon's G-Phrase, but includes the relaxed phase. It allows for focus on the kinematics of movement, separate from the semantics and pragmatics of gesturing. Including the relaxed phase in SDGs allows us to test various hypotheses related to intonational breaks in speech and discourse structure.
SDGs include all the phases around a stroke. So the start and end boundary depends on what phases are associated with a stroke.
The annotation value is created with "Label and Number Annotations" tool under the Tier menu option.
Handedness refers to which hand or hands are actively gesturing during the stroke. Both hands may be moving in synchrony or asynchrony. One hand may be moving while the other hand is inactive or aimless.
Handedness is labeled for what is happening during the stroke but the annotation range is the same as the SDG. Make a copy of the SDG tier and use ELAN's "Remove Annotations or Values" tool under the Tier menu option to remove only the values. Use this as a blank template for labeling for handedness.
Palm orientation does not always have to be labeled. It is important to know it exists when labeling for hand shape. Having palm orientation as a separate kinematic feature allows hand shape labeling to have a smaller list of options to choose from.
Palm orientation is labeled for palm orientation during the stroke. If the hand rotates during the stroke, palm orientation is labeled for the direction the palm is facing near the end of the stroke. It captures a moment in time, unlike trajectory shape. The annotation value for palm orientation can be a combination of the options below, in order to capture the 3-dimensional cartesian direction the palm is facing. For example, when the palms are facing the head, it would be SU for Self and Up. If the palms face each other, it would be I for Inside.
Unless you are interested in change in palm orientation during a stroke or SDG, the annotation range can be the same as the SDG. Make a copy of the SDG tier and use ELAN's "Remove Annotations or Values" tool under the Tier menu option to remove only the values. Use this as a blank template for labeling for palm orientation.
Our hand shapes are labeled independently of palm orientation. This helps keep the number of different hand shapes to a minimum.
You can add more to the library if they are distinct hand shapes. For uncertainties between two hand shapes, we used a 5-point slider. We use both hand shape abbreviations with a number from 1-5 in between. For example: C4O. The first letter indicates that it is a cupped shape that the labeler initially saw, the number indicates how close this shape is to the second handshape, Open. C4O means it more like Open than Cup. It is equivalent to O2C. So, the order does not matter.
Here's our list of hand shapes to use. Check the library of hand shapes at the end of this page. It contains images of the different hand shapes as a reference.
Hand shape usually stays the same during a stroke or SDG, so the annotation range can be the same as the SDG. Make a copy of the SDG tier and use ELAN's "Remove Annotations or Values" tool under the Tier menu option to remove only the values. Use this as a blank template for labeling for hand shape. Depending on your research question, you may want to do this differently.
The value is usually a single capital letter abbreviated from the word that would best describe the hand shape. For example, we use “F” for “fist,” “O” for “Open,” and “R” for “Relaxed.”
Two-handed hand gestures often use the same hand shape. We label them the same way as one-handed hand gestures. If they are different, label for the dominant hand.
When the two hands have different hand shapes, label for the dominant gesturing hand. If the non-dominant hand is not passive, label for the hand shape in a tier called non-dominant hand shape. Since there is a handedness tier that defines which hand is the dominant gesturing hand, you can compute which hand has which hand shape. This decision is focused on minimizing number of tiers needed for labeling, and making it easier for labelers to label for hand shape. If you prefer to have a separate hand shape tiers for left and right hands, you may do so.
Hand shapes may vary for different speakers. One speaker’s default “Cup” hand shape may be more rounded than another speaker’s “Cup” hand shape. If you are doing a inter-speaker comparison of handshapes, we recommend going through the video and take screenshots of the different handshapes used.
Location refers to where the hands are in respect to the body.
Slight movements that are visible might not be captured as having a location change because the trajectory motion does not travel far enough to qualify as a substantial change.
Use the mapping grid to for the values to use. Location is annotated as (right_hand_y),(right_hand_x);(left_hand_y),(left_hand_x). The y-values can have half values. For example, “3.5” would refer to the middle of the torso. Annotating the right hand first makes the labeling process faster.
(Insert image)
Every gesture stroke has a start location and an end location even if the locations are the same. For large gestures that cover more space, also label the extremes in between the start and end locations. Location can be labeled in a single tier, copying the stroke tier, and contain both the start and end locations as the annotation value. Or location can be labeled in two tiers, one for start location, one for end location, and, if necessary, one for location between if the trajectory path follows an extreme curvature.
Location labeling can use automatic tracking data for the hands and body. Make sure to use hand location relative to the body, rather than absolute location.
Several analyses can be done for location: distance traveled during a stroke, comparing strokes that have similar start or end locations, and more.
Trajectory shape labeling describes the shape of the gesture stroke's trajectory.
Observing the active hand movement, imagine the stroke tracing out a straight path, curved path, or, if looking at multiple gesture strokes, a looping path.
To label for trajectory shape, play the video segment of the stroke and then the entire SDG or a longer segment that includes the stroke to verify the trajectory shape. Play the segment over and over a few times. Identify the trajectory shape, making sure it refers to the stroke, not the preparation phase or the recovery phase.
Hand shape usually stays the same during a stroke or SDG, so the annotation range can be the same as the SDG.
The annotation value is either S, S-H, S-D, C, or L for each type of trajectory shape.
Make a copy of the SDG tier and then use ELAN's "Remove Annotations or Values" tool under the Tier menu option to remove only the values. Use this as a blank template for labeling for trajectory shape.
The straight trajectory shape is usually a vertical down or up motion, and can occasionally be diagonal–in most examples we’ve seen, the motion is moving upwards and out– or horizontal, usually outwards. Keep in mind these variations are usually infrequent perhaps because they take more energy to carry out, and may occur more often for a particular speaker.
When deciding between labels for curved or the variations of straight-horizontal and straight-diagonal, be sure to review the video snippet containing the entire SDG with all its phases and to evaluate the influences of the preparation and recovery phases on your perception of the stroke trajectory phase. Take them into consideration as contextual clues but remember that the trajectory shape label refers to the stroke phase, not to any of the other phases.
The looping trajectory shape must apply to multiple consecutive strokes that do not have a visual or temporal pause between them. A single loop without preceding or succeeding loops would be labeled as a curved trajectory shape. Regarding successive looping strokes, they are cut into individual “loops” at the end of the loop, or the lowest point in the loop, and each are labeled as a stroke.
The main difference between curved and looping is that there is a pause, hold phase, or break in movement path, between successive curved strokes, whereas these do not occur between successive looping strokes. Another helpful cue is the preparation phase which may occur before each curved strokes, but only before a set of successive looping strokes. Likewise with the recovery phase, which may occur after each curved stroke, but only at the end of a set of successive looping strokes.