The Complexities of What We Hear: Robot Mouths and Speaking Pianos

The drive to understand the coordination between the mind and body that produces language is a powerful source of innovation. The following videos represent attempts to replicate the human voice by means of a physical apparatus or robot rather than by digital sampling and recording techniques alone. This requires and stimulates a greater understanding of both how the human voice is produced and how it is heard.

Wien Modern was one of ten organizations asked to make an artistic contribution to the European Environmental Criminal Court Forum 2009 event in the Palazzo Ducale, Venice, Italy. Their project: to perform the text of a proclamation released earlier by the EECC as a musical composition, without setting it in a traditional sense. To that end, Austrian composer Peter Ablinger transferred the frequency spectrum of elementary school student Miro Markus’ voice to his computerized piano. Breaking down the voice into snatches of frequency, much as a computer might break down an image into pixels, Ablinger was able to reconstruct Markus’ recitation into pitch and rhythm on the keyboard. The result is astonishing, although having subtitles really helps clarify the text. The relationship of our speaking voice to pitch and rhythm is quite complex and the “music” of a voice is unique and integral to how that voice is understood.

Video via TheMcphearson on YouTube.

Scientists at Kagawa University in Takamatsu, Japan are trying to achieve a better understanding of how the mind and body work together to produce speech. The talking robot below consists of a silicone mouth, vocal cords and tongue; a plaster nasal chamber for increased resonance; an air pump that acts as lungs and a rudimentary computerized brain which has the capacity to associate movement with sound and learn through auditory feedback. What the human mind and body does subconsciously can only be reproduced with a great deal of trial and error, but the robot can learn and execute an extremely simple song, make vowel sounds, and produce some words. Next step? This mouth is currently being fitted with teeth so that it can produce fricative consonants. Look out, world!

Video via Diginfonews on YouTube.