Speech Communication Group

Gesture Research

Introductions

Paragraph 2

Resources

Research

text here

Publications

publications

Introduction Overview for Gesture Labeling at Speech Communication Group

  • Complete the CITI certification
  • Gain access to the Dropbox folder
  • Readings

Gesture Labeling

  1. Introduction to ELAN, video meeting or in-person meeting tutorial
  2. Introduction to PGGs and strokes. Introduction to phases.
    • Learn to label for PGGs and strokes. Imprecise labeling.
  3. Learn about what normal phases and strokes look like, and learn about what odd phases look like. (for example upwards strokes and downward prep.) Learn about trajectory shape to help think about this.
    • Label for phases, be sure to be aligned with video frames. Refine stroke labels as you do so. Precise labeling.
  4. Review strokes and phases labels. Consensus labeling if using a new sample.
  5. Create SDG tier based on stroke and phase labels.
  6. Learn handedness labeling. Copy SDG tier, clear out tier annotation labels, use blank annotations to label for handedness of strokes.
  7. Learn and review trajectory shape labeling. Label copied SDG tier for trajectory shape labeling.
  8. Learn about hand shapes. Read and review notes on hand shape characteristics. Learn about hand shape difference labeling. Label hand shape differences.

Discourse and ToBI (RPT) Labeling

  1. Learn to use Praat
  2. Label sound file for words. Make a copy of the words file and break it down into syllables. (There's some shortcut for this. I think it's make sure you have the cursor between where you want to break it apart, then click in the place on then waveform where you want to break it.)
  3. RPT labeling for syllables that have pitch accent.
  4. 3 and 4 breaks?

Coding Manual

New UROP Procedures - Introduction

CITI certification

  • If you have not already done so, please complete the CITI course for Human Subject Research: https://about.citiprogram.org/en/series/human-subjects-research-hsr/ (Or: https://www.citiprogram.org/index.cfm?pageID=154)
  • Click on the button that says "Buy Now". You do not have to pay for it. It will take you to a page where you can choose your institution, Massachusetts Institute of Technology, and you'll be asked to log in with your MIT Certificate.
  • Email me and Stef your certificate. We need your certificate for our records.

Dropbox access to corpora

I will share the Dropbox folder with you, using your MIT email. You should have unlimited data through MIT. It's called Exhaustive-Gesture-Labeling. It's a large folder. In the folder you'll find the two main folders, Corpora, and Readings for UROPs. The Corpora folder contains our entire Corpus! For starters, until you've gained expertise,

  • Do not delete anything.
  • Any new files you add that you work on should have your initials appended to the filename.

Pre-meeting preparation

  1. Download and install ELAN (85MB for MacOS): https://archive.mpi.nl/tla/elan/download
  2. Download Louvre video (9.8MB): https://www.dropbox.com/s/qm3ncz7alioakvp/louvre.mp4?dl=0
    • Watch and listen. Think about:
      • What do you notice about the speaker's gestures?
      • How do the gestures relate to the speech, e.g. in timing and in meaning?
      • What body parts are gesturing during speech?
  3. Download but do not open yet:

Meeting to show how to use ELAN

  • Introduction to Gesture Research - Stef

Using ELAN tips and tricks - Ada sharing screen

New UROP tasks

(This changes depending on when we share)

The first task is to learn to label Perceived Gestures Groups (PGGs), and then to label individual Stroke-Defined Gestures (SDGs)

  • Start off with labeling Perceived Gesture Groups for new samples,
    • Train on London or HistEng
    • Label Daileader samples (start with PGGs, and general area of strokes)
  • Then label Stroke-Defined Gestures
    • Just label these roughly—-you can make them more precise when you learn to label the phases of the gestures (i.e. preparation, hold, stroke, hold, recovery)
  • Assign Daileader PGG labeling (Daileader gestures look a bit choppy so it may be quite easy for new labelers to label for PGGs here)
    1. Assign sample to label on their own, maybe also pick out individual strokes. Label strokes roughly—-you'll
    2. Early- 4m30s- 7m07s (more interesting section about Charlemagne, historians, monks family)
    3. High- 0m0s - 3m12s (mostly about table manners, chivalry, stories; small occlusion around 36s-46s, topic transition at 3m12, to include it, go up to 3m30s)
    4. High - 10m6s - 14m03s (this is longer, but includes some discourse transitions)

Read papers in Readings for UROPs folder

Priority list

  1. Kendon 1980 (introduces gesture groupings) (1.5MB) https://www.dropbox.com/s/2kykl4afjdrn637/kendon1980gesticulation_speech.pdf?dl=0
  2. McNeill 1985 So you think gestures are non-verbal (1.7MB) https://www.dropbox.com/s/r7exqp23275q6v4/McNeill1985so-you-think-gestures-are-nonverbal.pdf?dl=0

Later

  • Access to Notion
  • Read documents on what to do and not do in the Dropbox folder and in Notion platform until they are more experienced

Rules for data management

Ultimately all data should be in the form of textgrid files. This is because it's easy to import into and export from ELAN.

Tiers should be completed before exporting to textgrid files. This means they need to be reviewed and vetted for by an experienced labeler for that tier.

Pre-reviewed and completed tiers also need to be saved and stored in the archive folder inside the textgrid folder.

Generally each tier should be saved as its own textgrid file. However there are a few exceptions:

  • PGGs should be saved together. This includes PGG1, PGG2, and PGG3 if any.
  • ToBI labels should be saved together. However, pitch-accented syllables and any ToBI tier that is explored and analyzed individually can be its own file.
  • Depending on how phases are labeled, whether by different labelers or one labeler, if they are created in different tiers, they can be saved in the same textgrid file. Usually a single tier is created that has combined all the phases. This is usually what we will use when looking at the phases. However, we may need the individual phase tiers for analysis.
  • Overall, any tiers that are labeled together as a set can be grouped together in one textgrid file. Of those tiers, if any of them are used individually for analysis, they should be also available in their own textgrid file. That way, someone looking to import them into an ELAN file for analysis will be able to grab just that one.

Importing textgrids into ELAN

  • When you import them into ELAN, make sure to click on the checkbox for "Skip empty intervals / annotations" so you don't get the in-between blank intervals.

Self-Training Procedure for Learning to label different dimensions

Overview: Use a pre-labeled sample. Observe how it is labeled. Hide the labels. Label it on your own. Unhide the pre-labeled section. Compare your labels. Discuss.

  1. Create a new ELAN file. (ELAN: File> New> Add Media File, Select the video file from the media folder)
  2. Locate the textgrid files you need (In folder Corpora> London> textgrids >london-sdg.textgrid, and london-handedness.textgrid)
  3. Import the textgrid files into the ELAN file (ELAN: File> Import > Praat TextGrid File…) Be sure to check the box for "Skip empty intervals / annotations"
  4. Copy the sdg tier (ELAN: Tier> Copy Tier) Select the SDG tier. Don't need to check any checkboxes. Just click on the next button until you get to the finish button.)
  5. Rename your copied tier (Right click on the tier name, "Change attributes of…") and change the tier name.
  6. Clear the annotations for your copied tier (ELAN: Tier> Remove Annotations or Values > deselect any select tiers and select ONLY the copied tier. Make sure you select "Annotation Values" and not "Annotations" and choose "All annotations")
  7. Look at the first 30 seconds of pre-labeled handedness labels. (0m30s to 1m00s) and familiarize yourself with them.
  8. Hide the pre-labeled handedness labels. (Right click on the tier name, "Hide…")
  9. Label for the handedness labels on your copied tier from 1m00s to 1m30s.
  10. Unhide the pre-labeled handedness tier (Right click on any tier name, "Visible tiers…" , select the pre-labeled handedness tier to unhide it)
  11. Compare with the original labels. How did you do? Where do you agree or disagree? Why do you think the differences arise? If possible, discuss with someone with more expertise in labeling this dimension.

Once you're confident, you can start labeling a sample that does not already have that tier labeled.

Self-Training Procedure

Overview: Use a pre-labeled sample. Observe how it is labeled. Hide the labels. Label it on your own. Unhide the pre-labeled section. Compare your labels. Discuss.

  1. If you are training using *.textgrid files, start by creating a new ELAN file. (ELAN: File> New> Add Media File, Select the video file from the media folder). If you are training with an ELAN file *.eaf, skip steps 2 and 3.
  2. Locate the *.textgrid file you want to use for training.
  3. Import the *.textgrid files into the ELAN file (ELAN: File> Import > Praat TextGrid File…) Be sure to check the box for "Skip empty intervals / annotations"
  4. If the tier or dimension had been annotated without sound, make sure your sound is muted or use the ELAN controls to mute the video sound.
  5. Observe the first 30 to 60 seconds of the annotations on the pre-labeled tier and familiarize yourself with them. Where does the annotation start and end? Why do you think that is? Does the label for the annotation make sense to you? Why is it labeled this way?
  6. Hide the tier. (Right click on the tier name, "Hide…")
    • Note: If you are learning to label a tier that has pre-determined annotation locations, such as handedness or hand shape, you can just copy the original tier, and then empty the annotations so that you have a tier with blank annotations. (ELAN: Tier> Remove Annotations or Values > deselect any select tiers and select ONLY the copied tier. Make sure you select "Annotation Values" and not "Annotations" and choose "All annotations")
  7. Create a new tier (Tier → New Tier…) for your labels.
  8. Label the same segment you have observed.
  9. Unhide the pre-labeled handedness tier (Right click on any tier name, "Visible tiers…" , select the pre-labeled tier to unhide it)
  10. Compare with the original labels. How did you do? Where do you agree or disagree? Why do you think the differences arise? If possible, discuss with someone with more expertise in labeling this dimension.

Once you are confident in labeling this dimension, you can start labeling a sample that does not already have that dimension labeled.

Occluded Regions

The occlusion tier is mainly to mark areas of occlusion so that we can ignore them when labeling for gestures. Sometimes when hands are partially occluded we can still see or surmise what is going on. So for the cases that Riley's been running into, you don't need to mark those regions where you see one hand actively gesturing. You can mark it as occluded if the hands are out of the frame for a long time. If they are just going out of the frame for a short time, you don't need to mark it as occluded. When the hands are behind the speaker's back, you also don't need to mark that as occluded.

Phases Labeling

Phases do not have to be in an SDG, do not have to be followed by strokes, holds can follow preps, etc.

SDG Tier

Create an sdg tier. SDG contains a stroke and its accompanying phases (preparation, pre-stroke hold, stroke, post-stroke hold, recovery, relaxed) Note: Relaxed is just the pause between gestures and is grouped within SDGs for data analysis.

Copying SDG for labeling

  1. Copy the sdg tier (ELAN: Tier> Copy Tier) Select the SDG tier. Don't need to check any checkboxes. Just click on the next button until you get to the finish button.)
  2. Rename your copied tier (Right click on the tier name, "Change attributes of…") and change the tier name.
  3. Clear the annotations for your copied tier (ELAN: Tier> Remove Annotations or Values > deselect any select tiers and select ONLY the copied tier. Make sure you select "Annotation Values" and not "Annotations" and choose "All annotations"

Trajectory Shape Labeling

  • (London sample) Learn trajectory shape labeling. Find a 30sec sample in london, review it for the trajectory shape labels. Use the textgrid named "london-kin-trajshapeAR-1119.TextGrid" for the official trajectory shape labels. Hide the pre-labeled tier. Create your own tier for trajectory shape. Compare. We can discuss your thoughts during our next meeting. Note: Trajectory shape is labeled using the duration of the SDG, but refers to the trajectory shape of the stroke. This can be difficult since the preparation phase lends more curvature to the stroke, and does influence the trajectory shape label. Ideally we want to focus on the trajectory shape of the stroke. (After our meeting, we will ask you to label trajectory shape for Louvre sample)
    • S (straight)
    • S-H (Straight-horizontal)
    • S-D (straight-diagonal)
    • C (curved)
    • L (looping)
    • CP (Complex Path) if all else fails, use this for illustrative gestures, often iconic gestures

Handedness Labeling

  1. Create an sdg tier. SDG contains a stroke and its accompanying phases (preparation, pre-stroke hold, stroke, post-stroke hold, recovery, relaxed) Note: Relaxed is just the pause between gestures and is grouped within SDGs for data analysis.
  2. Save the newly created SDG tier as a textgrid file. Put it in the text grid folder for the Louvre sample. (ELAN: File: Export as…> Praat TextGrid, only select one tier. When you hit OK, you'll get a dialog box for encoding format. Go with the default UTF-8)
  3. Copy the SDG tier (see steps 4-6 above)
  4. Label for handedness.
  5. (Next we'll do a reliability check of your labels compared to each other. And then save the official tier in the textgrids folder)

    Handedness labels from speaker's perspective (tip: put sticky notes on the sides of your computer screen)

    • LH (left hand)
    • RH (right hand)
    • RHD (right hand dominant)
    • LHD (left hand dominant)
    • BHES (both hands equal synchronous)
    • BHEA (both hands equal asynchronous)

Label Gesture Size

  • Use numbers 1,2,3,4,5
  • Very small can be the range of a small finger motion
  • Small can be the range of wrist motion, with little bit of elbow motion
  • Medium can be more elbow motion with some shoulder motion
  • Large would be more shoulder motion
  • very large would be extended motion

Analysis and Quantification

Should I include this as a section for each category or tier to be labeled?

Intro

Team

Name
Institution