Video-scanning system for measurement of lip and jaw motion* Martin J. McCutcheon, Samuel G. Fletcher, and Akira Hasegawa Departmentof Biocommunication,Universityof Alabama in Birmingham, UniversityStation,Birmingham, Alabama
35294
(Received 26 October 1976; revised 20 December 1976)
An instrument for measuringlabial and mandibular positionsduring speechis described.A modified
videosystemis usedto measurethe positions of smallreflectingglassbeadsattachedat selected poihtson the face and to a thin wire anchored on a mandibular tooth. The system allows continuousdata collection from up to 16 reflector points at a sampling rate of 100 Hz. Reflector movementswith instantaneous velocitiesup to 45 cm/sec can be determined with a 0.3-mm resolution. Techniquesfor data acquisition and analysisand displayare described. PACS numbers: 43.70.Ny, 43.70.Bk
camera and monitor modified for a 100-Hz framing
INTRODUCTION
Recent work in speech physiology has seen the development of a number of new research tools for the study of labial and mandibular functions. One reason for this
developmentis the necessityfor large volumesof data to determine strategies used in execution of coordinated movements of the articulators. Because of radiation, use of cinefluorographic technique may be contraindicated for this purpose. Apparatuses developed for dental studies typically are bulky and preclude simultaneous
studyof labial motions•-s andare therefore not well suited for speech physiological investigations.
Strain-gauge devices developed by Sussman and
Smith4 andby AbbsandGilbert• havebeenusedfor the study of labial and mandibular movements.
Also, Ohala
et al. e reportedsomepreliminaryresultsusinga photoelectric technique to transduce mandible movements. More recently, a low-dosage x-ray microbeam system has been developed to track small lead pellets attached
to the speecharticulators.7 It still introducesa definite risk
factor
however.
The instrumentation described in this paper is part of a custom-designed system for synchronous acquisition of data from labial, lingual, mandibular, and acoustic
sources.e As illustratedin theblockdiagramof Fig. 1, the system consistsof (1) a palatometer which detects tongue-to-palate contact with electrodes imbedded in a .very thin pseudopalatal plate that adheres to the
subject'shard palate, (2) a modified video system (gnathometer) that detects the locates the position of small light reflectors attachedto the speaker's face, (3) a
rate, (2) a video processor, (3) a set of X and Y coordinate storage registers interfaced to a P DP 11/40 computer. An internal clock provides a common time reference for all components of the total instrumentation system. ,
During data acquisition, the subject's face is illuminated with a uniform distribution of light and reflections from the small beads on the face form light spots on
the camera sensor. The camera utilizes a scanning process for converting the intensity of the spots into sharp peaks in the signals fed to the video processor. Scanning is effectively horizontal with 256 lines per field, with a framing rate of 100 Hzo
The video processor contains circuitry for detecting the peaks in the video signal, a network of adaptive gates, and two eight-bit X and Y counters synchronized to the system. The adaptive gate network is used to distinguish reflector data from spurious noise peaks in the video signals and to provide a unique set of coordinates for each reflector. During scanning, output signals from the gates cause data in the counters to be shifted to temporary
storage registers.
The registers
can
store data for up to 16 points per frame. The order of storage of coordinate values in the registers is determined by orientation of the points in the field with respect to the path of the scanning process. After scanning is completed, the coordinate values, together with data from other acquistion devices, are
parallel filter bank for spectral analysis of incoming
speechsignals, (4) a P DP 11/40 computerfor the control and acquisitionof data, and (5) data-display devices
.JGNATHO1
RA-[METER 1 ' •
that can be used in real time as monitors or as playback devices driven by the computer.
JIOEO•
The palatometer has been reported previously by
Fletcher, McCutcheon, andWolf.9 This paperwill be
-mGRAPH m m PDP mmy4ømm
concerned primarily with the system used for measuring labial and mandibular positions. I.
SYSTEM
DESCRIPTION PSUEO0PALATE
A block diagram of the video system is shown in Fig.
2. Its major componentsare: (1) a COHU model 6150 1051
J. Acoust. Soc. Am., Vol. 61, No. 4, April 1977
FIG. 1.
LED DISPLAY
Block diagram of total instrumentation system.
Copyright¸ 1977 by the AcousticalSociety of America
1051
Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 130.113.111.210 On: Fri, 19 Dec 2014 10:12:45
1052
McCutcheon, Fletcher,and Hasegawa: Videoscanning of lip andjaw motion
1052
CLOCK
AND
SYNC
TO
COMPUTER COUNTERS COORDINATE
8 BIT X
ADAPTIVE CAMERA
DETECTOR
STORAGE
FIG. 2. Block diagram of the video-scanning system.
8BITY
GATES
REGISTERS VIDEO i
PROCESSOR
FROM COMPUTER
DIGITAL COMPARATORS
transferred to a memory buffer in a PDP 11/40 computer.
The system throughout rate is sufficient to allow continuous data acquisition for up to 20 rain as data are im-
mediately transferred from the buffer memory to either magnetic tape or disk storage. To facilitate adjustments during data acquisition, signals from intermediate processing stages can be selectively mixed with the camera video and displayed as superimposed images on the monitor.
The monitor can also be operated as a display device driven by the computer with the necessary D/A conversion accomplishedby digital comparators (Fig. 2). The data then appear on the monitor screen as intensified
light spots. By controlling the rate at which data are updated by the computer, the display can be operated at a normal rate or in slow motion. Data acquisition and
display can also be carried out in a test mode, independent of the computer. II.
SYSTEM
PERFORMANCE
System performance is determined by a number of factors: lighting intensity and distribution, reflector characteristics, lens optics, camera sensor, scanning characteristics, The minimum
and the electronics. reflector
size that can be detected
de-
pends on the magnification provided by the optical system. The hardware detection scheme requires that the reflector
lines.
image subtend two or more successive scan
The reflectors are l-ram diam. glass beads
coated with glossy white enamel paint and attached to the skin with a medical
adhesive.
Their
small
size and
weight(0.05 g) afford minimummassloadingto the soft tissues of the lips and other structures.
Detection of
moving reflectors dependsprimarily on the dynamic response and scanning characteristics
of the camera sen-
sor. To evaluate the system's ability to locate a moving point, a number of, tests were run using a motor
FIG. 3. Anatomical reference points and coordinate axes in the midsaggitalplane. Points O and M are at the tips of the maxillary and mandibularincisors, respectively, andpoint P is located on the surface of the skin overlying the fronto-nasal suture.
J. Acoust. Soc. Am., Vol. 61, No. 4, April 1977
Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 130.113.111.210 On: Fri, 19 Dec 2014 10:12:45
1053
McCutcheon, Fletcher,and Hasegawa: Video scanning of lip andjaw motion
1053
secß Maximum instantaneous velocity in the X and Y
directions is 45 cm/sec. For measurements during speech, points on the lower lip are the most critical since the mandible and upper
lip move with slower velocities. The limiting 45-cm/sec velocity exceeds the maximum lower-lip
velocities of 30
cm/sec reportedby AbbsandGilbert• andapproximately 25 cm/secby Sussman et al. •o System linearity is about 2%. Stability tests con-
H!
ducted indicate no measurable instability for stationary reflectors. For moving reflectors, however, there is some variation in position as a result of nonuniformities in lighting and surface conditions of the painted beads. An estimate
of the maximum
error
in this
case is + 0.5
mm, or the bead radius. III.
•H 2 J2
DATA
ACQUISITION
In our kinematic studies of the lips and mandible we have for the most part used a profile view with reflec-
tors in the midsaggital plane of the subject's face. Point
O in Fig. 3 serves as a maxillary reference and point M on the edge of a lower incisor serves to characterize motions of the mandible. The position of point M on the mandibular
incisor
is calculated
from
measures
of re-
flectors located at points J• and J•. on a rigid wire at-
tachedby an orthodonticband to a canine tooth (see Fig. 4). The wire is shapedto exit near the corner of the FIG. 4. Typical location of reflectors used to determine articulator position and motionß
driven disk containing the 1-mm-diam. reflectors. The result of these tests clearly demonstrated that the positions of a reflector moving with maximum velocities up
to 45 cm/sec (1500 lines/sec) can be determined with a 0.3-mm resolution. Figure 5 illustrates measured horizontal and vertical positions of a disk-mounted re-
flector rotating at a constantangular velocity of 30 rad/
mouth at the level of labial margins then curved to place the reflectors in the midsaggital plane. Experiments indicate that vibrations of the short wire are sufficiently low in amplitude to be neglected. To determine geometric relationships between calculated and measured points, data are taken with teeth edge to edge and with a temporary reflector at the anterior edge of their contact. For these reference data, in effect points O and M coincide.
After speech data have been taken, the following calculations are made at each sample interval ø
Vertical
15
ß
o
ß
o
FIG.
-15
5.
Vertical
and horizontal
positions in mm versus time of a reflector rotating with a constant
angular velocity of 30 tad/sec.
Horizontal
Points represent samples taken at 10-msec
o•
ß
intervals.
ß
ß
-15
•o.
J. Acoust. Soc. Am., Vol. 61, No. 4, April 1977
Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 130.113.111.210 On: Fri, 19 Dec 2014 10:12:45
1054
McCutcheon,Fletcher,and Hasegawa:Video scanningof lip and jaw motion
UPPER
LIP
from
1054
the reference
measurements.
Equations (1) and (2) express the calculation in terms of the translational and rotational componentsof jaw
SKULL CANTILEVER
motion.
'U •li
USER DEFINED
I •
L
If the head is not restrained, simultaneous measurements and calculations are needed to determine the po-
POLYGONFOR THE OF
IDENTIFICATION
ß•
sition of point O from the measured positions of H• and //2 on a skull cantilever (see Fig. 5).
J2COORDINATE VALUES _
IV.
MANDIBLE
DATA
The reflector coordinates obtained during data acquisition are stored on a first-found, first-stored basis. Through the use of an off-line interactive program the
CANTILEVER
coordinates
Cumulative plot of reflector
coordinates over time.
The data arefor 15 repetitions ofthetestword
of a reflector
are identified
and sorted.
The interactive feature allows considerable flexibility in placement of the reflectors.
LOWER LIP
FIG. 6.
PROCESSING
[afa].
cosA- Y• sinA,
(1)
In the initial part of the program, reflector coordinate data are cumulatively plotted on a CRT storage screen as shown in Fig.. 6. This display illustrates the range of movement that took place for 15 repetitions of the ut-
Ym=Y•x + Y•u cosA +X•u sinA,
(2)
terance [a/a] from a profile view. Denser regions indi-
X,,,,=X.n +X•
cate slower moving or relatively stationary reflectors. The less-dense regions are associated with less-frequent and more-rapid movements.
where X•, Ymare the X and Y coordinates of the mandible point M, •x is the angle of rotation of the jaw from
iti referenceposition,X•x, Y•x are the measuredX and Y coordinatesfor point Jx; and X•M, Y•M are the X and
To determine the time history of a specific reflector, the associated cluster of data is enclosed by a user de-
Y components of the vector from J• to M as computed
12.5
10.0
HORI ZONTALOI SPLACEMENT (MM.)
o-
5.0
7.5
I
I
UPPER LIP
....... -.............. '--.' .... .... .... ß..............
IO.0
I
7.5
LOWER LIP
-
ß
ø
ø
' ß
.
.
/tOO
LOWER INCISOR
..'"-..
15--
5000
--
•ooo
--
"
LI P-TOOTH
', ....
'-'....
CONTACT
..............
'.:.... :L... :-:... :,•... • ... •...
F I LTERBANK
• •...
.....................................
• '
•...
;
35294
(Received 26 October 1976; revised 20 December 1976)
An instrument for measuringlabial and mandibular positionsduring speechis described.A modified
videosystemis usedto measurethe positions of smallreflectingglassbeadsattachedat selected poihtson the face and to a thin wire anchored on a mandibular tooth. The system allows continuousdata collection from up to 16 reflector points at a sampling rate of 100 Hz. Reflector movementswith instantaneous velocitiesup to 45 cm/sec can be determined with a 0.3-mm resolution. Techniquesfor data acquisition and analysisand displayare described. PACS numbers: 43.70.Ny, 43.70.Bk
camera and monitor modified for a 100-Hz framing
INTRODUCTION
Recent work in speech physiology has seen the development of a number of new research tools for the study of labial and mandibular functions. One reason for this
developmentis the necessityfor large volumesof data to determine strategies used in execution of coordinated movements of the articulators. Because of radiation, use of cinefluorographic technique may be contraindicated for this purpose. Apparatuses developed for dental studies typically are bulky and preclude simultaneous
studyof labial motions•-s andare therefore not well suited for speech physiological investigations.
Strain-gauge devices developed by Sussman and
Smith4 andby AbbsandGilbert• havebeenusedfor the study of labial and mandibular movements.
Also, Ohala
et al. e reportedsomepreliminaryresultsusinga photoelectric technique to transduce mandible movements. More recently, a low-dosage x-ray microbeam system has been developed to track small lead pellets attached
to the speecharticulators.7 It still introducesa definite risk
factor
however.
The instrumentation described in this paper is part of a custom-designed system for synchronous acquisition of data from labial, lingual, mandibular, and acoustic
sources.e As illustratedin theblockdiagramof Fig. 1, the system consistsof (1) a palatometer which detects tongue-to-palate contact with electrodes imbedded in a .very thin pseudopalatal plate that adheres to the
subject'shard palate, (2) a modified video system (gnathometer) that detects the locates the position of small light reflectors attachedto the speaker's face, (3) a
rate, (2) a video processor, (3) a set of X and Y coordinate storage registers interfaced to a P DP 11/40 computer. An internal clock provides a common time reference for all components of the total instrumentation system. ,
During data acquisition, the subject's face is illuminated with a uniform distribution of light and reflections from the small beads on the face form light spots on
the camera sensor. The camera utilizes a scanning process for converting the intensity of the spots into sharp peaks in the signals fed to the video processor. Scanning is effectively horizontal with 256 lines per field, with a framing rate of 100 Hzo
The video processor contains circuitry for detecting the peaks in the video signal, a network of adaptive gates, and two eight-bit X and Y counters synchronized to the system. The adaptive gate network is used to distinguish reflector data from spurious noise peaks in the video signals and to provide a unique set of coordinates for each reflector. During scanning, output signals from the gates cause data in the counters to be shifted to temporary
storage registers.
The registers
can
store data for up to 16 points per frame. The order of storage of coordinate values in the registers is determined by orientation of the points in the field with respect to the path of the scanning process. After scanning is completed, the coordinate values, together with data from other acquistion devices, are
parallel filter bank for spectral analysis of incoming
speechsignals, (4) a P DP 11/40 computerfor the control and acquisitionof data, and (5) data-display devices
.JGNATHO1
RA-[METER 1 ' •
that can be used in real time as monitors or as playback devices driven by the computer.
JIOEO•
The palatometer has been reported previously by
Fletcher, McCutcheon, andWolf.9 This paperwill be
-mGRAPH m m PDP mmy4ømm
concerned primarily with the system used for measuring labial and mandibular positions. I.
SYSTEM
DESCRIPTION PSUEO0PALATE
A block diagram of the video system is shown in Fig.
2. Its major componentsare: (1) a COHU model 6150 1051
J. Acoust. Soc. Am., Vol. 61, No. 4, April 1977
FIG. 1.
LED DISPLAY
Block diagram of total instrumentation system.
Copyright¸ 1977 by the AcousticalSociety of America
1051
Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 130.113.111.210 On: Fri, 19 Dec 2014 10:12:45
1052
McCutcheon, Fletcher,and Hasegawa: Videoscanning of lip andjaw motion
1052
CLOCK
AND
SYNC
TO
COMPUTER COUNTERS COORDINATE
8 BIT X
ADAPTIVE CAMERA
DETECTOR
STORAGE
FIG. 2. Block diagram of the video-scanning system.
8BITY
GATES
REGISTERS VIDEO i
PROCESSOR
FROM COMPUTER
DIGITAL COMPARATORS
transferred to a memory buffer in a PDP 11/40 computer.
The system throughout rate is sufficient to allow continuous data acquisition for up to 20 rain as data are im-
mediately transferred from the buffer memory to either magnetic tape or disk storage. To facilitate adjustments during data acquisition, signals from intermediate processing stages can be selectively mixed with the camera video and displayed as superimposed images on the monitor.
The monitor can also be operated as a display device driven by the computer with the necessary D/A conversion accomplishedby digital comparators (Fig. 2). The data then appear on the monitor screen as intensified
light spots. By controlling the rate at which data are updated by the computer, the display can be operated at a normal rate or in slow motion. Data acquisition and
display can also be carried out in a test mode, independent of the computer. II.
SYSTEM
PERFORMANCE
System performance is determined by a number of factors: lighting intensity and distribution, reflector characteristics, lens optics, camera sensor, scanning characteristics, The minimum
and the electronics. reflector
size that can be detected
de-
pends on the magnification provided by the optical system. The hardware detection scheme requires that the reflector
lines.
image subtend two or more successive scan
The reflectors are l-ram diam. glass beads
coated with glossy white enamel paint and attached to the skin with a medical
adhesive.
Their
small
size and
weight(0.05 g) afford minimummassloadingto the soft tissues of the lips and other structures.
Detection of
moving reflectors dependsprimarily on the dynamic response and scanning characteristics
of the camera sen-
sor. To evaluate the system's ability to locate a moving point, a number of, tests were run using a motor
FIG. 3. Anatomical reference points and coordinate axes in the midsaggitalplane. Points O and M are at the tips of the maxillary and mandibularincisors, respectively, andpoint P is located on the surface of the skin overlying the fronto-nasal suture.
J. Acoust. Soc. Am., Vol. 61, No. 4, April 1977
Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 130.113.111.210 On: Fri, 19 Dec 2014 10:12:45
1053
McCutcheon, Fletcher,and Hasegawa: Video scanning of lip andjaw motion
1053
secß Maximum instantaneous velocity in the X and Y
directions is 45 cm/sec. For measurements during speech, points on the lower lip are the most critical since the mandible and upper
lip move with slower velocities. The limiting 45-cm/sec velocity exceeds the maximum lower-lip
velocities of 30
cm/sec reportedby AbbsandGilbert• andapproximately 25 cm/secby Sussman et al. •o System linearity is about 2%. Stability tests con-
H!
ducted indicate no measurable instability for stationary reflectors. For moving reflectors, however, there is some variation in position as a result of nonuniformities in lighting and surface conditions of the painted beads. An estimate
of the maximum
error
in this
case is + 0.5
mm, or the bead radius. III.
•H 2 J2
DATA
ACQUISITION
In our kinematic studies of the lips and mandible we have for the most part used a profile view with reflec-
tors in the midsaggital plane of the subject's face. Point
O in Fig. 3 serves as a maxillary reference and point M on the edge of a lower incisor serves to characterize motions of the mandible. The position of point M on the mandibular
incisor
is calculated
from
measures
of re-
flectors located at points J• and J•. on a rigid wire at-
tachedby an orthodonticband to a canine tooth (see Fig. 4). The wire is shapedto exit near the corner of the FIG. 4. Typical location of reflectors used to determine articulator position and motionß
driven disk containing the 1-mm-diam. reflectors. The result of these tests clearly demonstrated that the positions of a reflector moving with maximum velocities up
to 45 cm/sec (1500 lines/sec) can be determined with a 0.3-mm resolution. Figure 5 illustrates measured horizontal and vertical positions of a disk-mounted re-
flector rotating at a constantangular velocity of 30 rad/
mouth at the level of labial margins then curved to place the reflectors in the midsaggital plane. Experiments indicate that vibrations of the short wire are sufficiently low in amplitude to be neglected. To determine geometric relationships between calculated and measured points, data are taken with teeth edge to edge and with a temporary reflector at the anterior edge of their contact. For these reference data, in effect points O and M coincide.
After speech data have been taken, the following calculations are made at each sample interval ø
Vertical
15
ß
o
ß
o
FIG.
-15
5.
Vertical
and horizontal
positions in mm versus time of a reflector rotating with a constant
angular velocity of 30 tad/sec.
Horizontal
Points represent samples taken at 10-msec
o•
ß
intervals.
ß
ß
-15
•o.
J. Acoust. Soc. Am., Vol. 61, No. 4, April 1977
Redistribution subject to ASA license or copyright; see http://acousticalsociety.org/content/terms. Download to IP: 130.113.111.210 On: Fri, 19 Dec 2014 10:12:45
1054
McCutcheon,Fletcher,and Hasegawa:Video scanningof lip and jaw motion
UPPER
LIP
from
1054
the reference
measurements.
Equations (1) and (2) express the calculation in terms of the translational and rotational componentsof jaw
SKULL CANTILEVER
motion.
'U •li
USER DEFINED
I •
L
If the head is not restrained, simultaneous measurements and calculations are needed to determine the po-
POLYGONFOR THE OF
IDENTIFICATION
ß•
sition of point O from the measured positions of H• and //2 on a skull cantilever (see Fig. 5).
J2COORDINATE VALUES _
IV.
MANDIBLE
DATA
The reflector coordinates obtained during data acquisition are stored on a first-found, first-stored basis. Through the use of an off-line interactive program the
CANTILEVER
coordinates
Cumulative plot of reflector
coordinates over time.
The data arefor 15 repetitions ofthetestword
of a reflector
are identified
and sorted.
The interactive feature allows considerable flexibility in placement of the reflectors.
LOWER LIP
FIG. 6.
PROCESSING
[afa].
cosA- Y• sinA,
(1)
In the initial part of the program, reflector coordinate data are cumulatively plotted on a CRT storage screen as shown in Fig.. 6. This display illustrates the range of movement that took place for 15 repetitions of the ut-
Ym=Y•x + Y•u cosA +X•u sinA,
(2)
terance [a/a] from a profile view. Denser regions indi-
X,,,,=X.n +X•
cate slower moving or relatively stationary reflectors. The less-dense regions are associated with less-frequent and more-rapid movements.
where X•, Ymare the X and Y coordinates of the mandible point M, •x is the angle of rotation of the jaw from
iti referenceposition,X•x, Y•x are the measuredX and Y coordinatesfor point Jx; and X•M, Y•M are the X and
To determine the time history of a specific reflector, the associated cluster of data is enclosed by a user de-
Y components of the vector from J• to M as computed
12.5
10.0
HORI ZONTALOI SPLACEMENT (MM.)
o-
5.0
7.5
I
I
UPPER LIP
....... -.............. '--.' .... .... .... ß..............
IO.0
I
7.5
LOWER LIP
-
ß
ø
ø
' ß
.
.
/tOO
LOWER INCISOR
..'"-..
15--
5000
--
•ooo
--
"
LI P-TOOTH
', ....
'-'....
CONTACT
..............
'.:.... :L... :-:... :,•... • ... •...
F I LTERBANK
• •...
.....................................
• '
•...
;
