paper - Department of Speech, Music and Hearing

Proceedings, FONETIK 2004, Dept. of Linguistics, Stockholm UniversityModelling interactive language learning:a project presentationLacerda, F., Sundberg, U., 1 Carlson, R. 2 and Holt, L. 31 Dept. of Linguistics, Stockholm University, Stockholm2 Dept. of Speech, Music, Hearing, KTH, Stockholm3 Dept. of Psychology, Carnegie Mellon University, Pittsburgh, USAAbstractThis paper describes a recently started interdisciplinaryresearch program aiming at investigatingand modelling fundamental aspects ofthe language acquisition process. The workinghypothesis assumes that general purpose perceptionand memory processes, common toboth human and other mammalian species,along with the particular context of initialadult-infant interaction, underlie the infant’sability to progressively derive linguistic structureimplicitly available in the ambient language.The project is conceived as an interdisciplinaryresearch effort involving the areas ofPhonetics, Psychology and Speech recognition.Experimental speech perception techniques willbe used at Dept. of Linguistics, SU, to investigatethe development of the infant’s ability toderive linguistic information from situated connectedspeech. These experiments will bematched by behavioural tests of animal subjects,carried out at CMU, Pittsburgh, to disclosethe potential significance that recurrentmulti-sensory properties of the stimuli mayhave for spontaneous category formation. Datafrom infant and child vocal productions as wellas infant-adult interactions will also be collectedand analyzed to address the possibility ofa production-perception link. Finally, the datafrom the infant and animal studies will be integratedand tested in mathematical models ofthe language acquisition process, developed atTMH, KTH.BackgroundIn the advent of the experimental studies on thelanguage acquisition process the primary focuswas on the infant’s ability to discriminate orproduce isolated speech sounds (e.g. Eimas,Siqueland, Jusczyk, & Vigorito, 1971; Locke,1983). However necessary and important asinitial steps, the study of isolated perceptual orarticulatory phenomena soon falls short of addressingthe general linguistic implications ofsuch initial phonetic abilities. Therefore morerecent attempts have been instead focused oninvestigating, for example, the internal structureof phonetic categories in half-year-old infants(Kuhl, Williams, Lacerda, Stevens, &Lindblom, 1992), the infant’s ability to recognizeword-like patterns from continuous speech(e.g. (Jusczyk, 1999) or to capitalize on statisticalregularities in the speech signal (Saffran,Aslin, & Newport, 1996). Also research on infants’vocal production have shifted from focuson isolated speech sound productions to attemptsto relate patterns of babbling with awider biologically oriented linguistic frame(MacNeilage & Davis, 2000). Yet, many ofthese studies emanate from a perspective ofadult-full-fledged linguistic behaviour wherethe infant is assumed to acquire its ambientlanguage by engaging in a process of findingout its phonemic and other traditional linguisticconstituents. Not surprisingly, the infant’s capabilityof learning the ambient language withall its variability, have been proposed to rely onpre-wired linguistic knowledge in terms ofLanguage Acquisition Device or notions of‘poverty of the stimulus’. In the present researchprogram, on the other hand, the infant isassumed to have no innate language-specificpredispositions and the language acquisition isviewed as a consequence of general sensoryand memory processes and continuity betweenlow-level sensory information processing andlanguage capacity.From the theoretical outline of the present researchproject it is assumed that the initialphases of human language acquisition processis a general purpose process through whichauditory representations of speech sequencesare linked to other co-occurring sensory stimulias an automatic consequence of exposure tocorrelated multi-sensory information input.Since spoken language is often used to refer to

Proceedings, FONETIK 2004, Dept. of Linguistics, Stockholm Universityobjects or actions in the shared outside world,there is an inevitable implicit correlation betweenhearing the sounds of words or phrasesrelating to these objects and seeing, feeling,smelling or otherwise perceiving the referentsof the spoken language.Initially, a sound sequence may be associatedwith any object that happen to be presentedwith temporal contiguity with the sound, butcontinued exposure to richly varying spokenlanguage soon provides enough information tonarrow the scope or review the sound-objectlinks (Lacerda, 2003). As this happens, a moredetailed representation of the initially unanalyzedchunks of spoken sounds starts toemerge, enabling differentiation between identifiable(acquired) sound objects and the remainingunknown sounds.In the present theoretical framework, this representsthe emergence of a general segmentationprocedure that will eventually lead to lexicalrepresentation. At this basic level, the humanrepresentation capacity is likely to be shared bysome other species although humans at somelevel obviously depart to using sound representationsas objects that can be processed as combinatorialelements while keeping their attachedmulti-sensory links. At this level a cognitiveprocess is unparalleled in the animal world beginsto emerge but its ontogenesis is, so far,unaccounted for. By the coordinated study ofthe early segmentation and sound-object associationprocesses in human and animal subjectsand detailed simulation on the experimentalconditions with operative mathematical models,the current project expects to make a contributionto the understanding of this unique humancapacity.Research planThrough a joint effort from three areas of research– early human language development,animal learning and speech technology - datawill be gathered and used to generate and calibratemathematical models that may accountfor language learning in terms of general multisensoryinput, memory processes and the infant’sinteraction with its environment.Infants as models of speech and languagedevelopmentHuman infants typically engage in speechcommunication activities within their first yearsof life and usually achieve a well-developedlinguistic competence by about 10-12 years ofage. Retrospective analyses of the language acquisitionprocess often present language learningas an effortless and smooth developmenttowards the adult language. This approachmisses the important information conveyed bydifficulties and the errors that children actuallydo and detaches the speech signal from its ecologiccontext. Indeed, traditional language developmentresearch fails to address some fundamentalquestions such as:• Why does the sound of speech seem tobe more attractive for the newborn infantthan other sounds? In this neonatalpreference a sign of human specific geneticmechanisms that under normalcircumstances generate behaviour that istaken for an innate preference for listeningto spoken language?• How do representations of phonemes,syllables and words arise and develop inthe human infant? Can these traditionallyassumed units, the very core of theadult linguistic structure, be derivedfrom continuous speech in the absenceof pre-wired linguistic knowledge?• How do adults conceive the infant’s linguisticcompetence and what phoneticand linguistic modifications do they introduceto accommodate the infant?• Are production preferences reflected inperceptual biases? For example, how isthe perception of phonemic contrasts affectedby the child’s production abilities?How does the interplay betweenperception and production develop duringthe first years of life? How does thecombinatorial pressure of lexical representationsrelate to the child’s phonologicalawareness?Questions like these will be investigatedfrom a broad developmental perspective usingwell established techniques such as theHigh-Amplitude Sucking technique, to testthe youngest infants, and the visual preferenceparadigm to study the emergence oflexical and phonological structure in childrenup to 2 years of age. The next step willbe to investigate the stability over time and

Proceedings, FONETIK 2004, Dept. of Linguistics, Stockholm Universitygeneralization ability from the establishedauditory-visual representations, testing thesubjects’ ability to cope with different kindof noise affecting the speech signal, in linewith the phoneme restoration and rhymingstudies.The subject population will consist of largecross-sectional samples (about 100 subjects)to address specific perceptual issuesplus a group of 20 infant-adult dyads, foradditional detailed investigation of productionand interaction aspects.Animal studies of sensory processing andmemoryAs speech scientists began to argue about thespecificity of human speech perception, the resultsfrom auditory experiments with animalsubjects became a crucial contribution to theon-going debate. For instance, animal studiescontributed with important result that certainacoustic continua were categorized in much thesame way by both humans and non-human subjects(Kuhl & Padden, 1983; Kuhl & Padden,1982). More recent studies involving animalsubjects have further exposed general mechanismsaccounting for the internal organizationof categories of speech sounds, the influence ofphonetic context (Lotto, Kluender, & Holt,1997) and sensitivity to acoustic trading relations(Holt, Lotto, & Kluender, 2001; Kluender,Lotto, Holt, & Bloedel, 1998; Kluender &Lotto, 1994; Lotto et al., 1997). Using animalsubjects in perception studies has proven to offera unique possibility to exercise completeexperimental control over experience. The generalprocedure in studies with, for instance,gerbils allows assessing detection, discriminationand identification of speech stimuli.In the present project gerbils will be used inperception experiments since this species hasbeen used successfully in previous studies.Methods of training these animal subjects arebased on a go/no-go paradigm. The gerbils aretrained with positive reinforcement to remainon a little platform in a cage on a “positive”stimulus and to jump off the platform on a“negative” stimulus. The gerbils are “at work”15-20 min. in daily sessions and their performanceis measured in terms of % correct responsesand d’.The current project attempts to widen the scopeof speech perception experiments with animalsby trying to create realistic animal models ofcertain early stages of the language acquisitionprocess. It is expected that the animal modelswill perform more or less like the infant subjectsup to a certain level of initial representation.By studying the levels of complexity forwhich the animal models fall short of the humanperformance the project is expected tocontribute with important insights on the emergenceof early cognitive processes.Testing the hypotheses in mathematicalmodelsAddressing the early stages of the language acquisitionprocess from a broad perspective offersa unique opportunity to understand and designspeech communication systems that hopefullymay be able to capitalize on the key aspectsof the human children’s flexibility andlearning capability rather than attempting tomimic adult stereotypes.The ultimate goal of a speech recognition systemis to be able to handle speech signals withhuman-like efficiency. While the performanceof available systems using speech input may byreasonable under optimal communication situations,the systems’ lack of flexibility and vulnerabilityand noise or moderately adversecommunication situations is a significant hinderto a wider and safer application of such speechinterfaces. There are several factors that accountfor the mismatch between the expectedand the obtained performance of such speechcommunication interfaces. In addition to thedifficulties inherent to the characterization ofthe speech signals per se, these engineered interfaceshave also been developed to mimic andmatch the speech communication performanceof experienced adult speakers, bypassing thedevelopmental process that eventually led tothe adult communicative competence. Andwhile this in an acceptable strategy to quicklyapproach the ultimate communicative goal itrelies on a deterministic view of the speech signaland of the whole human communicationprocess that tends to result into rather rigid systemsthat can hardly cope with realistic variancein, for instance, the speech signal.

Proceedings, FONETIK 2004, Dept. of Linguistics, Stockholm UniversityPart of the problems usually faced by speechrecognition systems is caused by the strong focuson the speech signal itself, as the primarycomponent of the speech communication process.While this is a reasonable starting point forthe development of man- machine speech interfaces,it is obvious that in natural speech communicationsettings the speech signal is onlypart, albeit crucial, of the communication process(Colin et al., 2002; McGurk & MacDonald,1976) and recent developments of language andspeech systems are beginning to take advantageof multimodal information to improve the systems’communication efficiency (e.g. Beskow,2003). However, in spite of the improvement incommunication performance clearly accrued bythe introduction of additional informationsources, the design of these communicationsystems may still be limited by their attempt toengineer a full-fledged human communicationsystem while attempting to bypass the long developmentalpath that eventually led to such anadult competent performance. Thus, the currentproject will attempt to incorporate linguisticand psychological knowledge of the earlystages of the language acquisition process inorder to design a prototypical system that essentiallywill be able to learn from multimodalinformation sources and integrate them toachieve flexible and realistic representations ofits environment.AcknowledgementsThe project is funded by The Bank of SwedenTercentenary Foundation K2003-0867.ReferencesBeskow, J., 2003. Talking heads – models andapplications for multimodal speech synthesis.PhD thesis, TMH/KTH.Colin, C., Radeau, M., Soquet, A., Demolin,D., Colin, F., & Deltenre, P. (2002). Mismatchnegativity evoked by the McGurk-MacDonald effect: a phonetic representationwithin short-term memory. Clin. Neurophysiol.113, 495-506.Eimas, P. D., Siqueland, E. R., Jusczyk, P., &Vigorito, J. (1971). Speech perception in infants.Science 171, 303-306.Holt, L. L., Lotto, A. J., & Kluender, K. R.(2001). Influence of fundamental frequencyon stop-consonant voicing perception: acase of learned covariation or auditory enhancement?JASA 109, 764-774.Jusczyk, P. W. (1999). How infants begin toextract words from speech. Trends CognSci. 3, 323-328.Kluender, K. R. & Lotto, A. J. (1994). Effectsof first formant onset frequency on [-voice]judgments result from auditory processesnot specific to humans. JASA 95, 1044-1052.Kluender, K. R., Lotto, A. J., Holt, L. L., &Bloedel, S. L. (1998). Role of experiencefor language-specific functional mappingsof vowel sounds. JASA 104, 3568-3582.Kuhl, P. K. & Padden, D. M. (1982). Enhanceddiscriminability at the phonetic boundariesfor the voicing feature in macaques. Percept.Psychophys32, 542-550.Kuhl, P. K. & Padden, D. M. (1983). Enhanceddiscriminability at the phonetic boundariesfor the place feature in macaques. JASA 73,1003-1010.Kuhl, P. K., Williams, K. A., Lacerda, F., Stevens,K. N., & Lindblom, B. (1992). Linguisticexperience alters phonetic perceptionin infants by 6 months of age. Science255, 606-608.Lacerda, F. (2003). Phonology: An emergentconsequence of memory constraints andsensory input. Reading and Writing: An InterdisciplinaryJournal 16, 41-59.Locke, J. L. (1983). Phonological Acquisitionand Change. New York: Academic PressLotto, A. J., Kluender, K. R., & Holt, L. L.(1997). Perceptual compensation for coarticulationby Japanese quail (Coturnix coturnixjaponica). JASA 102, 1134-1140.MacNeilage, P. F. & Davis, B. L. (2000). Onthe Origin of Internal Structure of WordForms. Science 288, 527-531.McGurk, H. & MacDonald, J. (1976). Hearinglips and seeing voices. Nature 264, 746-748.Saffran, J. R., Aslin, R. N., & Newport, E. L.(1996). Statistical Learning by 8-Month-OldInfants. Science 274, 1926-1928.