From syllable representation via gesture generation to articulation: A spiking neural model
Organizations: Faculty of Medicine, RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany
Abstract
Speech production requires a transformation from phonological representations into coordinated vocal-tract movements. This paper presents a spiking neural sensorimotor model within the Neural Engineering Framework and Semantic Pointer Architecture. Each syllable occurrence has a neural oscillator whose evolving phase coordinates its gestures and initiates the next syllable. Phase-dependent commands recruit separate neural populations for vocalic, oral-constriction, velopharyngeal, and glottal actions (gestures). Neural ramp integrators shape the rise and fall of gesture activation. Together with articulatory targets, these trajectories determine the extent and time course of articulatory movements. Opponent neural populations provide control signals to a geometric articulatory model. Thirty reference recordings cover an input corpus of eight lexical words, an alternative realization of one word, a syllable sequence, and isolated syllables. Two additional paired simulation subcorpora of sixty runs each, covering two words and one syllable sequence, assess activation reliability across neural realizations. Adjusting gesture command amplitudes increased successful realizations from 57/60 to 60/60 under the specified criteria, with ramp rate factors held constant. The model demonstrates how neural oscillator and ramp dynamics generate articulatory timing under specified phasing rules. It complements a preceding mental-lexicon model through an ordered syllable interface. Continuous online execution across all components and connection to the acoustic extension remain further integration steps. Accessible code and synchronized visualization support inspection of the neural activation trajectories and resulting vocal-tract movements.
Figures & tables
| Gesture position | Start | End |
|---|---|---|
| Single onset consonant | 0.1500 ( ) | 0.4317 ( ) |
| Two-consonant onset: first | 0.1500 ( ) | 0.3848 ( ) |
| Two-consonant onset: second | 0.2439 ( ) | 0.4787 ( ) |
| Stressed single vowel | 0.2927 ( ) | 0.7152 ( ) |
| Unstressed single vowel | 0.2927 ( ) | 0.6683 ( ) |
| Two-vowel nucleus: first | 0.2927 ( ) | 0.6213 ( ) |
| Gesture class | Original | Revised | |
|---|---|---|---|
| Oral consonantal | 1.00 | 1.00 | 0.20 |
| Vocalic | 0.95 | 0.95 | 0.05 |
| Velum opening | 0.70 | 0.90 | 0.05 |
| Glottal opening (syllable onset) | 1.00 | 0.70 | 0.20 |
| Glottal opening (syllable coda) | 0.60 / 1.00 | 0.70 | 0.20 |
| Word / realization | Program | Neurons | Min. peak | |
|---|---|---|---|---|
| Tasse | ta se | 6 | 12,400 | 0.88 |
| Tasche | ta Se | 6 | 12,400 | 0.88 |
| Kanne | ka ne | 6 | 12,400 | 0.60 |
| Katze | ka tse | 8 | 14,800 | 0.88 |
| Teller (short ER) | tE lA | 5 | 11,200 | 0.88 |
| Teller (long ER) | tE lEA | 6 | 12,400 | 0.81 |
| Program | Original | Revised |
|---|---|---|
| Kanne | 20/20 | 20/20 |
| Schrank | 20/20 | 20/20 |
| bap--gum--dip--bap | 17/20 | 20/20 |
| Total | 57/60 | 60/60 |
| Question | Coupled planning oscillators | Spiking Neural Model |
|---|---|---|
| How is timing organized? | Relative phases settle through coupling, conditional on the specified graph, coupling functions, and parameters. | Prescribed windows of syllable phase recruit separate gesture ramps. |
| What linguistic patterns are explained? | Accounts for selected consonant–vowel and cluster patterns, and acquisition preferences under stated learning assumptions. | Realization of supplied syllable programs; emergence and acquisition of their coordination patterns are not yet modeled. |
| How explicit is neural implementation? | Dynamical planning states; the learning model includes abstract neural units and Hebbian attunement, without an explicit spiking circuit. | Timing and recruitment are implemented with spiking populations and specified synaptic transformations. |
| What remains biologically unidentified? | The specific neural populations and connections realizing these planning states and couplings. | The human neural populations and codes corresponding to the implemented states and connections. |
| Aspect | Bekolay (2016) | Present implementation |
|---|---|---|
| Trajectory construction | Multidimensional forcing function decoded from neural oscillator phase. | Phase windows issue amplitude commands; each gesture occurrence has a separate recurrent ramp population. |
| Start mechanism | Default inhibition; a start-gated error population resets neural oscillator state. | Input or controller drive initiates each occurrence neural oscillator; no equivalent target-error reset population. |
| Downstream output | Vector gesture score converted for VocalTractLab; neural articulatory expansion left for future work. | Ramps weight articulatory targets, followed by opponent neural populations and geometric movement generation. |
| Aspect | Modeling advantage | Cost or limitation |
|---|---|---|
| Interpretable control | Onset, duration, gesture command amplitude, and ramp rate factor can be adjusted for each gesture occurrence. | These controls interact; gesture command amplitude also affects rise slope and finite-window peak activation. |
| Gesture state | Each population retains activation history and can continue decaying after its command ends. | Residual activity and transmission delays require assessment; the present response law restricts available curve shapes. |
| Rate manipulation | Command timing can change while gesture response time is retained or adjusted separately. | Shorter commands can reduce activation; changing phase speed alone does not uniformly compress the full trajectory. |
| Neural implementation | Recruitment and dynamics can be inspected and perturbed separately for each gesture. | Additional populations and pathways require reliable drive; a successful neural oscillator start does not ensure every ramp is recruited. |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Value |
| Neuron model | LIF |
| Neural oscillator | 2 dimensions; 2,500 neurons per occurrence |
| Neural oscillator intercept distribution | Uniform on |
| Nominal frequency; recurrent synapse | 1.5 Hz; 25 ms |
| Neural oscillator amplitude threshold | Neural oscillator norm at least 0.3 |
| Gesture pulse / next-syllable population | 1 dimension; 200 neurons |
| Quantity | Value |
|---|---|
| Input pulse (preselected syllables) | 0.15–0.20 s; unit amplitude |
| SPA sequence loading (prescribed) | Four 100-ms syllable-position inputs |
| SPA start command | 0.40–0.45 s |
| Timing-network simulation duration | 2.5 s |
| Integration step | 1 ms |
| Neural oscillator, ramp, initiation probe filters | 10 ms |
Vocal-Tract Acoustics, Aerodynamics, and Overlap-and-Add Synthesis in DYNARTmo
Tasse'', Muse'', and ``Lade'', illustrate the coordination of articulation, pressure, flow, and acoustic sources across vowels, nasals, laterals, fricatives, and plosives. The implementation has been optimized by the author primarily for sound recognizability; independent perceptual evaluation remains outstanding, and fricative-noise naturalness requires further refinement. The model provides an inspectable, articulatorily controlled production plant for future neural simulations of babbling, imitation, and early word acquisition. Further work will improve acoustic realism, reduce empirical source corrections, and extend the sensory information available to developmental learning models.