Nonlinear Dynamics of Operant Behavior: A New Approach via the ...

This paper was published in **the** journal:“Reviews in **the** Neurosciences”, 13(2002), 31-57.

**Nonlinear** **Dynamics** **of** **Operant** **Behavior**: A **New** **Approach** **via****the** Extended Return MapJay-Shake Li* and Joseph. P. HustonInstitute **of** Physiological Psychology and Center for Biological and MedicalResearch, University **of** Düsseldorf, GermanyKey words:operant, Skinner-box, reinforcement, rat, nonlinear dynamics, Extended Return Map.Corresponding Author:Jay-Shake LiInstitute **of** Physiological Psychology 1, University **of** DüsseldorfUniversität Str. 1, 40225 Düsseldorf, GermanyTel.: 49-211-8113493Fax: 49-211-8112024e-mail:lijay@uni-duesseldorf.de2

SynopsisPrevious efforts to apply nonlinear dynamical tools to **the** analysis **of** operant behaviorrevealed some promise for this kind **of** approach, but also some doubts, since **the** complexity**of** animal behavior seemed to be beyond **the** analyzing ability **of** **the** available tools. We hereoutline a series **of** studies based on a novel approach. We modified **the** so-called “ReturnMap” and developed a new method, namely, **the** Extended Return Map (ERM) to extractinformation from **the** highly irregular time series data, **the** inter-response-time (IRT) generatedby Skinner-box experiments. We applied **the** ERM to operant lever pressing data from ratsusing **the** four fundamental reinforcement schedules: Fixed-Interval (FI), Fixed-Ratio (FR),Variable-Interval (VI) and Variable-Ratio (VR). Our results revealed interesting patterns inall experiment groups. Especially, **the** FI and VI groups exhibited well organized clusters **of**data points. We calculated **the** fractal dimension out **of** **the**se patterns and comparedexperimental data with surrogate data sets, which were generated by randomly shuffling **the**sequential order **of** original IRTs. This comparison supported **the** finding that patterns inERM reflect dynamics **of** **the** operant behaviors under study. We **the**n built two models tosimulate **the** functional mechanisms **of** **the** FI schedule. Both models can produce similardistributions **of** IRTs and **the** stereotypical “scalloped” curve characteristic **of** FI responding.However, **the**y differ in one important feature in **the**ir formulation: While one model uses acontinuous function to describe **the** probability **of** occurrence **of** an operant behavior, **the** o**the**rone employs an abrupt switch **of** behavioral state. Comparison **of** ERMs showed that only **the**latter one was able to produce patterns similar to **the** experimental results, indicative **of** **the**operation **of** an abrupt switch from one behavioral state to ano**the**r over **the** course **of** **the** interreinforcementperiod. This example demonstrated **the** ERM to be an useful tool for **the**analysis **of** IRT accompanying intermittent reinforcement schedules and for **the** study **of**nonlinear dynamics **of** operant behavior.IntroductionSince **the** pioneering works **of** Skinner /33/, a number **of** efforts have been made toexamine functional mechanisms underlying reinforced behavior from different aspects. Most**of** **the** **the**oretical models dealt with influences **of** different schedules **of** reinforcement onbehavior. For example: Herrnstein’s hyperbola equation describes **the** relationship betweenresponse rate and **the** rate **of** reinforcement /15,16/. Killeen and Fetterman introduced a“**Behavior**al Theory **of** Timing (BeT)”/19/ to describe behavior under **the** control **of** intervalschedules **of** reinforcement. Killeen formulated a ma**the**matical principle for reinforcement asa whole /20/, and developed in his recent work a ma**the**matical model **of** operant behaviorcontrolled by ratio-schedules /2/. Parallel to **the**se efforts, Gibbon introduced **the** “ScalarExpectancy Theory (SET)“ /5,10/ for timing in operant behavior.These models described how operant behavior − in **the** case **of** Skinner-boxexperiments, **the** rate **of** lever pressing − changed with parameters defined in reinforcementschedules. They predicted **the** averaged output **of** behavior over a session or across severalsessions. Key assumptions underlying **the**se kinds **of** approaches were: firstly, **the** existence**of** a stationary state **of** behavior, and secondly, **the** arrival to such a state at **the** moment **of**measurement. Without **the** existence and arrival **of** such a stationary state, averaging oversessions would not make sense. Machado and his coworkers noted this point and madeobservations on changes **of** behaviors from session to session /24,25/. The model **the**yproposed was based on Killeen’s BeT, and was able to describe changes **of** behavior over timewithin **the** inter-reinforcement period for **the** fixed-interval (FI) schedule. Their workrepresented one **of** **the** rare attempts to actually deal with dynamics **of** operant behavior. The3

sparsity **of** such studies is astonishing considering that **the** learning process itself is defined bychanges **of** behavior over time as a function **of** experience, and, hence, is a dynamic process.However, Machado’s approach used values averaged over **the** whole or half **of** asession. This made his model very different from **the** stringent motion equations known inphysics. Earlier, several studies did go into **the** realm **of** studying real time dynamics. Forexample, as early as Skinner’s time, it was well known that stereotypical behavioral patternscould be found in operant experiments, and that **the** type **of** behavioral pattern depends on **the**reinforcement schedule in use. For example, a stereotypical pattern called “scalloped” curveappeared in **the** cumulative records **of** behavior controlled by fixed-interval (FI) reinforcementschedules. Parallel to this stereotypical behavior, rate **of** operant responding varied withinone reinforcement interval, as well as from one interval to **the** next one. A large variability inrate **of** responding was also found from session to session /33/. Numerous efforts, including**the** analysis **of** inter-response-time (IRT), were made to study **the** stereotypical “scalloped”curve and to tackle this “variability”-problem (e.g. /6,7,31,38/). However, an “equation **of**motion” in **the** sense **of** **New**ton’s classical dynamics has not yet been found.In **the** second half **of** **the** past century new findings in **the** study **of** chaos and nonlineardynamics opened a new field and achieved deeper insight into **the** nature **of** variability (for anintroduction see /11/). Similar to traditional models, **the** nonlinear systems approach viewedbehavior as being caused by **the** combined deterministic influences **of** a set **of** controllingvariables. However, in a nonlinear system **the**se variables did not work **the** same way asstandard independent variables described in traditional models. The effect **of** each variabledepended on **the** simultaneous states **of** all o**the**r variables. As a result, **the** combination **of**just a few variables in a nonlinear system was capable **of** generating highly irregular patternsthat were indistinguishable from behavior **of** a real complex system with a huge number **of**variables /26/. These findings provided a new aspect to **the** “variability”-problem found inbehavioral experiments.Fur**the**rmore, nonlinear dynamics implied that separately studying each variable **of** asystem in isolation might not necessarily lead to an understanding **of** that system as a whole.The effects **of** variations **of** a single variable in **the** context **of** a system might be different thanwhen it was altered in isolation /17/. Thus, experimenters were left with an infinitelycomplicated system, whose understanding required studying all **of** **the** potential factors and**the**ir interactions simultaneously. The nonlinear systems approach made use **of** tools thatexplored how multiple variables might interact without identifying and measuring **the**mdirectly /29/. As a result, systems could be understood as a whole, even if **the**ir componentparts had not been identified.Analysis **of** nonlinear dynamics relied heavily on graphical methods. One **of** **the** mostfundamental tools is **the** construction **of** phase space (for an introduction to nonlineardynamical tools see /1/). The phase space **of** a dynamic system is a ma**the**matical space withorthogonal coordinate directions representing each **of** **the** variables needed to specify **the**instantaneous state **of** that system. To construct such a portrait, **the** position **of** each variablemust be known in infinite precision through time. Unfortunately, in most behavioralexperiments, only few **of** **the** variables involved in a particular experimental situation can beknown. Even less understood is how **the**se variables change over time. In order tocircumvent this problem, we can reconstruct a multi-dimensional space with delay-coordinatevectors, which are vectors **of** time series measurement **of** one observable **of** a system.Analyses **of** this type were suggested in physics (e.g. /29/) and **the**ir validity has beendemonstrated ma**the**matically /35/. The delay-coordinate method is widely used in all fields**of** studies employing nonlinear dynamics.4

The Skinner-box experiment faces ano**the**r problem when applying **the** delayedcoordinatemethod. The data set recorded here, **the** inter-response-time (IRT), is in fact a timeseries with time itself as **the** observable, and it is sampled in an irregular period along **the**time-coordinate. This problem is common for all systems that can be classified as pointprocess, such as **the** inter-spike-interval **of** neuronal firing (e. g. /34/), or **the** R-R interval fromelectrocardiogram (ECG) studies (e. g. /13/). Although it is possible to first convert **the** IRTdata into response rate, thus making it uniform along **the** time axis, **the**re are many settingswhere it is more interesting to analyze **the** IRT data directly /18/. Instead **of** time-coordinate,**the** basic idea **of** a delayed-coordinate method can be extended to **the** order-coordinate.Diagrams drawn in this manner are **the** so-called “return maps” (e.g. /4,32/). In some earlyapplications to operant experiments similar tools were in use, and such diagrams were said toshow **the** sequential dependence **of** IRTs /30,38/. If IRTs are bounded by some kind **of**dynamic mechanism with low dimensionality, **the** variation **of** IRTs should be deterministicand **the** return map should be able to capture **the** structure **of** **the** function that describes **the**dynamic mechanism /18/.Previous applications **of** **the** return map to operant data indicated some promise **of** **the**nonlinear systems approach, but also some doubts, since **the** complexity **of** behavioralexperiments seemed to demand improvement **of** **the** analyzing ability **of** this tool /30,38/.Therefore, we modified **the** original definition **of** **the** return map and developed **the** “ExtendedReturn Map (ERM)” /21,22/. Instead **of** using single IRTs directly, we first calculated **the**running average **of** several IRTs and **the**n used **the**m to plot a diagram. We applied thismethod to study operant behavior **of** rats controlled by Fixed-Interval (FI) , Fixed-Ratio (FR),Variable-Interval (VI) and Variable-Ratio (VR) schedules, and found interesting patterns in**the** resulting ERMs. We fur**the**r calculated **the** correlation dimension and informationdimension from **the**se patterns and compared **the**m with maps generated using surrogate datasets, which were generated by randomly shuffling **the** sequential order **of** **the** original IRTs.The random shuffling procedure destroyed **the** sequential dependency **of** **the** original timeseries data. This dependency is essential for **the** dynamics **of** **the** system. If **the** results from**the** surrogate data sets do not match **the** experimental results, **the**n **the** patterns shown in **the**ERM taken from **the** experiment must reflect **the** dynamics **of** **the** system /23,37/. Our resultsshowed that this is indeed **the** case. **Dynamics** **of** operant behavior are **of** deterministic nature,and this can be analyzed using ERM.We **the**n developed two models to simulate **the** operant behavior controlled by FIschedules and generated artificial IRT sets using a computer program. We adjusted **the**parameters **of** **the** models so that both sets **of** IRTs showed curves similar to **the** experimentaldata in **the** frequency distribution **of** IRT. Fur**the**rmore, both IRT sets could successfullyreproduce **the** “scalloped” patterns **of** response shown in experimental data as represented incumulative records. However, both models differed in one feature **of** **the**ir formulation.While one model produced a probability function that changed continuously after eachreward, **the** o**the**r one employed an “abrupt switch **of** responding modus”. The two modelsshowed different patterns in **the**ir ERMs, and only **the** one employing **the** abrupt switch led toERMs that were like those **of** **the** experimental results. This indicated that an abrupt switch **of**responding modus was essential for **the** dynamics **of** operant behavior under **the**sereinforcement schedules. These corresponded to a change **of** behavioral states during **the** timebetween two adjacent rewards. This idea has been suspected in early operant conditioningstudies (e.g. /9,31/), and Killeen and Fetterman also indicated switching between behavioralstates in **the**ir BeT /19/. However, it has not been possible so far to quantitatively identify thisphenomena. Now, with **the** ERM as an analyzing tool, we can easily monitor **the** procedure**of** this switching between behavioral states.5

Ma**the**matical definitionDefinition **of** **the** Original and **the** Extended Return MapData suitable for **the** application **of** **the** return map consist **of** sequences **of** timeintervals between events, **the** inter-event-time (IET), or, in **the** case **of** Skinner-Boxexperiments, **the** inter-response-time (IRT). In **the** simplest case **the** nth inter-event-time(IRT n ) is plotted against **the** value that precedes it (IRT n-1 ), resulting in a 2-dimensional, lag 1return map. In **the** more general case, a d-dimensional lag L return map is plotted by usingpoint set:{ (IRT n , IRT n-L , IRT n-2L , ... IRT n-(d-1)L ) | where n indices **the** order **of** data points } (1)In order to improve **the** analyzing ability **of** return maps for data strongly contaminatedwith noise, and for systems with both deterministic and probabilistic features, we havemodified **the** above definition and have invented **the** “extended return map” /21,22/. In **the**first step a new data set consisting **of** **the** running average (RV) **of** **the** original IRT wasconstructed according to **the** following equation:fRVn=n∑ + f − 1k = nIRTk/ f (2)Here a new parameter “f” was introduced in order to specify **the** range within which**the** running average was computed. We call this new parameter **the** “fold” **of** **the** extendedreturn map. Then a procedure similar to **the** original return maps can be applied. A d-dimensional, lag L extended return map with fold f, is constructed by **the** following point set:{( f RV n , f RV n-L , f RV n-2L , ... f RV n-(d-1)L ) | where n indexes **the** order **of** data points } (3)Fractal DimensionOne way to specify **the** property **of** patterns found in a phase diagram is to calculate**the** fractal dimension. Unlike **the** topological dimension, which is always an integer value,**the** fractal dimension can be an integer or a fraction. This value gives a quantitative measure**of** **the** fractal properties, including self-similarity and scaling /23/. Several ma**the**maticaldefinitions **of** fractal dimension have been suggested, and procedures were developed tocalculate **the**m from experimental data /12,14,36/. In **the** present study, we used **the**“correlation dimension” and **the** “information dimension” to interpret our experimental data.We chose **the**se because **the** distribution **of** our data points in **the** extended return map was notuniform and **the**se two definitions can take this factor into account.Suppose **the** data points from an experiment consist **of** a time series {X 1 , X 2 , ..., X M },**the**n **the** correlation dimension **of** this set as defined by Grassberger is /12/:ν=ln C( ε )(4)ln ( ε )limε → 0Where C(ε) is defined as2C(ε) = limM− × { number **of** pairs (X i , X j ) with | X i - X j | < ε} (5)M→∞M is **the** amount **of** data in **the** time series; ε is a value predefined to calculate C(ε).6

The information dimension is defined according to Grassberger as /12/:=E(ε)(6)ln(ε)δ limε→0Where E(ε) is defined asN( ε )E(ε) = ∑ Pi ln(P i)(7)i=lHere P i is **the** probability for a data point to fall into **the** cube i **of** **the** partitioning with size ε.For **the** analysis **of** experimental data, **the** frequency **of** occurrence in each cube **of** **the**partitioning was counted and was taken as **the** probability P.In practice, several ε values between 0.01 sec and 10 sec were chosen. The former is**the** resolution **of** **the** IRT and **the** latter is in **the** order **of** **the** size **of** **the** patterns in **the** ERMs.The corresponding ln C(ε) and E(ε) were calculated, and **the**n plotted against ln (ε.) Theslope **of** **the** resulting regression line was taken as **the** correlation dimension ν and **the**information dimension δ.Although it is not certain, whe**the**r or not **the** extended return map is a topologicallyequivalent reconstruction **of** **the** phase diagram, it is still possible to calculate **the** fractaldimension, because **the** ma**the**matical definition in equations (4), (5), (6) and (7) is applicablefor any time series. The only constraint is **the** number **of** data points. Theoretically infinitedata points are required in order to obtain an exact value. In practice, **the** amount **of** datashould be reasonably large /28/. For Skinner-Box experiments, this condition is relativelyeasy to fulfill.Even when **the** meaning **of** this value calculated from **the** extended return map differsfrom **the** “dimension” **of** attractors found in a real phase diagram, it is still interesting todetermine, whe**the**r or not it can serve as a quantitative measure **of** behavioral patterns inanimals receiving different reinforcement schedules and/or physiological treatments.Testing **the** Results with Surrogate DataSince experimental data is always contaminated with noise, **the** differentiation **of**information **of** systems dynamics from disturbance by noise becomes a critical point in a nonlinearanalysis. We wish to determine whe**the**r our findings in **the** extended return map reallyreflect **the** system’s characteristics, or whe**the**r **the**y are simply caused by **the** analyzingprocedure. One way to clarify this, is to test **the** results with surrogate data sets which have**the** same statistical properties as **the** original data, such as mean, variance and power spectra,while **the** dynamic information, that is, **the** deterministic correlation **of** **the** system’s presentstate with its history, is destroyed. If **the** patterns in **the** extended return map from surrogatedata sets differ from those **of** **the** original data, **the**n **the** differences must be due to **the** fact that**the** dynamic information presented in **the** original data was removed.There are several different ways to generate surrogate data sets. The simplest one is torandomly reorganize **the** sequential order **of** **the** original data set. This reorganization is verysevere. Theiler suggested that it might be enough to generate surrogate data sets which have**the** same first order correlation as **the** original one, but all higher order correlations areremoved /23,37/. For simplicity, only **the** reorganization **of** **the** original data was applied in**the** present study.7

In **the** present study, 5000 and 2500 surrogate data sets were generated for **the**calculation **of** **the** surrogate information and correlation dimensions respectively. Thus afrequency distribution **of** **the** surrogate fractal dimensions could be established. The location**of** **the** experimental fractal dimensions on **the** diagram **of** **the** surrogate distributionquantitatively shows **the** probability **of** acquiring **the** experimental IRT data by chance.ExperimentsAnimalsFive male, experimentally naive white Wistar rats obtained from **the**Tierversuchsanlage, University **of** Düsseldorf, were used in **the** present study. They were 2months old and weighed 220 ~ 260 g at **the** beginning **of** **the** experiment. The animals werehoused in individual cages with free access to water. Illumination was controlled artificiallyunder a 12hours day/night period with light on at 7:00 AM and **of**f at 7:00 PM. Experimentswere conducted between 10:00 AM and 5:00 PM. During shaping and test sessions, **the**animals were given about 50 % **of** **the** food **the**y consumed during **the** free feeding days. Thefood pellets consumed through rewards in **the** Skinner-box study were also taken into account.ApparatusThree Skinner-boxes (28 cm × 27 cm × 29 cm) made **of** metal were placed into threeseparated wooden sound attenuating chambers (130 cm × 59 cm × 103 cm). A 40 W bulbilluminated each chamber, resulting in about 12 LUX measured in **the** middle **of** **the** Skinnerbox.Experiments were executed and controlled by a personal computer. A program runningunder MS DOS 6.2 was developed to control **the** hardware and to record data. With **the** aid **of**this s**of**tware, one PC can run up to four Skinner-Boxes at a time. IRTs were saved in a textfile with tabular format, and a second program running under Micros**of**t Windows 95/98 waswritten to perform **the** nonlinear dynamical analysis. This program was also capable **of** doingsimulations using models discussed later. The DOS-program was written and compiled with**the** programming language “Pascal”. Design and development **of** **the** apparatus, including **the**s**of**tware, was accomplished by Manfred Mittelstaedt, and one **of** **the** authors (J.-S. Li). TheWindows 95/98-program was written by J.-S. Li and was compiled with **the** programminglanguage “C++ Builder”.ProcedureAfter five days (15 minutes per day) **of** manual shaping followed by a three day pause,**the** effect **of** changing reinforcement schedules on **the** dynamics **of** operant lever pressingbehavior was observed for 8 days (details see. Table 1). One daily session was run for eachanimal. In **the** first session, every rat received 90 minutes continuous reinforcement (CRF).In **the** following 7 sessions, each animal received CRF during **the** first minute, **the**n **the**reinforcement schedule was changed abruptly to one **of** **the** following schedules: FI20, FI40,FR20, VI40/20, and VR20/20. Each rat received **the** same intermittent reinforcementschedule in all 7 sessions during **the** testing periods. Data from **the** first day are indicatedbelow as “CRF”. They served as a kind **of** baseline behavior before different reinforcementschedules were applied. Data from **the** remaining 7 days are designated as “sessions 1through 7” throughout this paper. Before **the** beginning **of** shaping and during **the** 3 daypause, **the** weight **of** food consumed by each rat was recorded individually and served asreference for food deprivation.8

Table 1Experimental proceduresDays Reinforcement schedules Length **of** Run time1 ~ 5 Shaping 15 min6 ~ 8 Pause9 CRF 90 min10 ~ 16(Session 1 ~ 7)CRF⇓VI, FI or FR1 min⇓90 minFor variable ratio and interval schedules, a random number generator provided by **the**compiler “Borland Turbo Pascal 7.0” was used to construct a homogeneous probabilitydistribution. The numbers x/y in **the** notation **of** variable interval and ratio schedules stand for**the** mean (**the** first number x) and **the** range **of** variation (**the** second number y). That is, **the**intervals and ratios used in **the** VI40/20 and VR20/20 schedules varied within **the** range (30,50) and (10, 30). The probability **of** occurrence **of** each value within **the** range was **the** same.ResultBefore presenting **the** results **of** nonlinear dynamic analysis, some traditional analysis,including **the** overall lever pressing behavior across sessions, **the** frequency distribution **of**IRTs, and **the** cumulative records were examined in order to ensure that it conformed toconventional responding generated by Skinner-box experiments. The nonlinear dynamicanalysis began with a comparison between **the** original “Return Map (RM)” and **the**“Extended Return Map (ERM)” from session 7. The results confirmed **the** expectedimprovement **of** **the** analyzing ability **of** **the** ERM over **the** RM. The comparison **of** fractaldimensions **of** **the** ERMs between **the** experimental and **the** surrogate data sets demonstrated**the** consequence **of** destroying **the** sequential order **of** **the** IRT data. Since **the** animals under**the** control **of** interval-schedules showed very special patterns in **the** ERMs **of** session 7, wedocumented **the** ERMs **of** **the**ir behavior from o**the**r sessions, in order to investigate **the**development **of** **the**se special patterns.Lever pressing behavior across sessionsFig 1 shows **the** total number **of** lever presses per session throughout **the** wholeexperiment. Data points in **the** first column show **the** number **of** lever presses over 90minutes. On this day **the** animals were under a CRF schedule. As indicated by **the** diagram,all animals produced roughly **the** same number **of** lever presses. For sessions 1-7 only **the**lever presses from **the** 2 nd to **the** 91 th minute were taken into evaluation. Records **of** **the** firstminute were discarded, because all animals were still under CRF during this time. From **the**second minute on different reinforcement schedules were introduced. Thus, only behaviorunder **the** influence **of** testing schedules in sessions 1-7 were shown in **the** diagram, and **the**length **of** recording time was also 90 minutes, as in **the** first column. As indicated by **the**figure, number **of** lever presses increased rapidly over **the** first session as **the** scheduleschanged, and **the**y remained more or less constant over **the** following six sessions. Only **the**data from **the** FR20 rat exhibited ano**the**r rapid increase in responding in sessions 4 and 5.9

Fig. 1: The total number **of** lever presses per session throughout **the** whole experiment.The first column (CRF) shows **the** number **of** lever presses over 90 minutes. Forsessions 1-7 only **the** lever presses from **the** second to **the** 91th minute wereevaluated.Frequency distribution **of** IRTsThe frequency distributions **of** IRTs from session 7 are shown in figure 2. Each bar in**the** diagrams was 0.1 second wide. All IRTs longer than 12 seconds were accumulated in **the**last bar. Roughly speaking, **the** IRT distributions can be divided into three parts: firstly, anextremely sharp and narrow peak consisting **of** very short IRTs (

Fig. 2: The frequency distributions **of** IRTs from session 7. Width **of** bars = 0.1 sec.; allIRTs longer than 12 sec. are accumulated in **the** last bar. The IRT distribution curvesare similar to **the** distribution curve **of** a Poisson process.Fig. 3: Part **of** **the** cumulative records in session 7, reconstructed using IRT data. Typicalcharacteristics for operant behavior under intermittent reinforcement schedules can beseen.11

Fig. 4: The averaged rate **of** responding after rewards **of** FI20, FI40 and VI40/20 rats insessions 1. Column width = 1 second for FI20; 2 seconds for FI40 and VI40/20.The last column summarizes all responses that appeared later than 20, 40 and 50seconds after rewards.Fig. 5: The averaged rate **of** responding after rewards **of** FI20, FI40 and VI40/20 rats insession 7. Setup for **the** diagrams is **the** same as in **the** fig. 4.The high variability shown in fig. 3 made it difficult to generalize any findings to be acommon characteristic **of** **the** operant behavior for a given animal. One possible way toovercome this problem was to average **the** responses across rewards. Although different,previous studies have indicated that this kind **of** analysis might be problematic and should notbe used to represent **the** “average” **of** scalloped patterns found in cumulative records under FIschedules /3,7,9/, this procedure is a convenient way to produce an easily recognized overallcumulative record.Figures 4 and 5 show **the** averaged rate **of** responding over time after rewards underFI20, FI40 and VI40/20 schedules from sessions 1 and 7. The width **of** all but **the** lastcolumns is 1 sec for FI20 and 2 sec for FI40 and VI40/20. The last column summarizes allresponses that appeared later than 20, 40 and 50 sec after rewards. In **the** first session **the**averaged response rate was roughly constant except shortly (less than 4 sec) after rewards, asindicated by **the** shorter second columns in **the** records **of** **the** FI20 and FI40 rats, and **the** firstcolumn in **the** records **of** **the** VI40 animal. The patterns **of** responding in records **of** session 7show typical characteristics **of** operant behavior under interval-schedules: except for **the**overshooting in **the** first column, rate **of** responding dropped abruptly after reward, and **the**nincreased gradually until **the** administration **of** **the** next reward. The maximum response rate12

Fig. 6: The original “Return Maps (RMs)” from session 7. Here only **the** region with Xand Y values smaller than 10 seconds is shown. Most **of** **the** data points are locatednear **the** zero point and along **the** two axes. No structure can be clearly identified.Fig. 7: The “Extended Return Maps (ERMs)” from session 7. “RA **of** IRT(k)” and “RA **of**IRT(k+L)” represent running average **of** IRT at position k and k+L, where L is **the**time lag. Data used was **the** same data as in fig. 6. Parameters: F=15, L=15.Lattice structure can be seen in **the** maps on **the** left side. Arrows indicate **the**cluster with highest concentration **of** data points.13

in **the** record **of** VI40/20 was about 36-38 seconds, which was slightly lower than **the** mean **of****the** pre-defined inter-reinforcement-interval. These patterns **of** responding are typical foroperant behavior controlled by an interval-schedule. Also **the** differences between behavior insession 1 and in **the** final session 7 are consistent with previous reports (e.g. /25/).Results **of** **the** original “Return Maps” and **the** “Extended Return Map”Figures 6 and 7 show **the** original “Return Maps (RMs)” and **the** “Extended ReturnMaps (ERMs)” respectively. They were constructed using IRTs from session 7. In order tohave a closer look at **the** fine structures and to have a better comparison between **the** twodiagrams, only **the** region with X and Y values smaller than 10 seconds is shown. The fold(F) and time lag (L) used for **the** construction **of** **the** ERMs were F=15 and L=15.In figure 6, most **of** **the** data points are located near **the** zero point and along **the** twoaxes. No o**the**r structure could be found in any **of** **the** RMs. Nei**the**r could we find cleardifferences between behavior controlled under different schedules. This finding is consistentwith a previous study using pigeon /30/. In figure 7, we can easily differentiate betweenanimals under control **of** interval- and ratio-schedules by comparison **of** **the** ERMs. The 3ERMs in **the** left column show results **of** **the** rats under interval-schedule control. Severalclusters **of** points form well organized lattice structures. The arrows indicate **the** cluster withmaximal concentration **of** data points. The two ERMs in **the** right column show results **of** **the**animals under ratio-schedule control. A very dense cluster, which contains most **of** **the** datapoints, is located in **the** region bounded by **the** four coordinates: (0.5, 0.5), (2, 0.5), (0.5, 2)and (2, 2) for FR20 rat: (1, 1), (3, 1), (1, 3) and (3, 3) for VR20/20 rat. O**the**r data pointsformed several small clusters which line up along both axes. Data points were very rare in **the**central region **of** **the** maps.Fig. 8: The surrogate ERMs from session 7. Compared with fig. 7, **the** lattice structures havealmost vanished.14

Comparison between experimental and surrogate data setsThe ERMs **of** surrogate data sets from session 7 are shown in figure 8. The data setswere generated by randomly shuffling **the** sequential order **of** **the** experimental IRT data. Theparameters used for **the** construction **of** **the** surrogate ERMs were F=15 and L=15, which were**the** same as in figure 7. The comparison between figures 7 and 8 gives a clear impression **of****the** change **of** patterns caused by **the** random shuffling. The well organized lattice structure in**the** ERMs **of** animals under interval-schedule control almost vanished in **the** ERMs **of** **the**surrogate data. Although most **of** **the** data points still concentrated in roughly **the** same regionas in **the** ERMs **of** **the** experimental data, **the** distribution **of** data points was morehomogeneous. The ERMs **of** **the** surrogate data from animals under ratio-schedule control arevery similar to those from **the** experimental data, except that for **the** FR20 rat, **the** smallclusters aligned along **the** axes in **the** experimental ERM become more diffuse in **the** surrogatedata. Hardly any difference can be identified in **the** surrogate ERM **of** VR20 rat.Since **the** direct comparison **of** **the** maps cannot quantitatively describe **the** differencebetween **the** experimental and **the** surrogate data sets, we first had to calculate an entity thatcould reflect **the** topological features **of** **the** patterns found in ERMs, and **the**n compare thisvalue. We chose **the** correlation dimension and **the** information dimension to accomplish this.We repeated **the** random shuffling procedure many times and calculated **the** correlation andinformation dimensions from each **of** **the** resulting surrogate data set; hence, after partitioning,we could count **the** occurring frequency **of** **the** surrogate dimension in each partition and forma frequency distribution diagram for each rat. These are shown in figure 9 for **the** correlationdimension and figure 10 for **the** information dimension. The number **of** surrogate data setsgenerated in **the** random shuffling procedure was 2500 for **the** correlation dimension and 5000for **the** information dimension. The parameters used to construct **the** ERMs were F=15 andFig. 9: The frequency distribution **of** **the** surrogate correlation dimension. The bar indicates **the**position **of** **the** experimental value. The percentage value represents **the** relative size **of****the** dark gray area (outside **the** experimental dimension ). Except for **the** VR20 rat, allo**the**r experimental values are clearly far from **the** central area **of** **the** distribution.15

Fig. 10: The frequency distribution **of** **the** surrogate information dimension. The barindicates **the** position **of** **the** experimental value. The percentage value represents**the** relative size **of** **the** dark gray area. There are two peaks in each distributioncurve. The experimental FI40 and VI40 values are clearly apart from **the** centralarea. The experimental FR20 value is very close to **the** center **of** **the** surrogatedistribution. Owing to **the** existence **of** **the** second peak, it is not so easy to evaluate**the** situation for FI20 and VR20.L=15. The experimental correlation and information dimensions are shown in figs. 9 and 10as light gray bars. The percentage shown in **the** diagrams represents **the** relative surfacecovered by **the** dark gray area.It can be clearly seen that **the** frequency distribution **of** **the** surrogate correlationdimension **of** all rats was more or less normally distributed. Except for **the** VR20 rat, all o**the**rexperimental correlation dimensions were located far apart from **the** center maximum. Thisindicated that **the** sequential order **of** IRTs was important for **the** operant behavior **of** **the**serats, so that its randomization resulted in data sets that generated ERMs that had distinctivelydifferent values **of** correlation dimension. The VR20 rat, on **the** contrary, produced IRTswhose sequential order could be formed by chance as well.The frequency distribution curves **of** information dimension did not look like a normaldistribution, because **the**re were two maxima in **the** occurring frequency. For **the** FR 20 rat,one **of** **the** two peaks was extremely small. Although **the**y were not normally distributed, wecould never**the**less judge from **the** percentage **of** **the** gray area how probable it was to obtain**the** sequential order **of** **the** experimental IRTs simply be chance. According to **the** presentresults, **the** rats under **the** interval-schedules showed operant behavior that is far**the**r fromrandom than that **of** **the** ratio-schedule controlled rats.16

Formation **of** **the** lattice structures in **the** ERMs **of** rats under interval-schedulesThe ERMs constructed using IRT-data **of** **the** FI20, FI40 and VI40 rats from all testingsessions are shown in figs. 11, 12 and 13. The gradual formation **of** **the** lattice structures canbe seen. The parameters used in **the** construction **of** ERMs were for FI20: F=12, L=12; forFI40: F=15, L=15; for VI40/20: F=15, L=15. In **the** beginning **of** **the** learning process **the**ERMs **of** all three rats (left diagram in **the** first line) looked very similar: A dense cluster **of**points near **the** bottom-left corner **of** **the** diagram can be seen, and **the** density **of** pointsdecreases when we move from bottom-left to top-right. The data points in **the** ERM **of** **the**FI20 rat are slightly more concentrated on **the** bottom-left side **of** **the** cluster as revealed by**the** slightly denser tip **of** **the** cluster. The lattice structure in **the** ERMs from **the** FI20 ratbegins to appear in session 3 (right diagram in **the** first line) and becomes very clear insessions 4 and 5 (left and middle diagrams in **the** second line). Thereafter, **the** lattice structurein **the** ERMs becomes slightly diffuse in sessions 6 and 7 (right diagram in **the** second lineand left diagram in **the** third line). The formation **of** lattice structure in **the** ERMs **of** **the** o**the**rtwo animals begins at about session 6 (right diagram in **the** second line). In comparison to **the**FI20 rat, this occurred much later. Prior to **the** formation **of** **the** lattice structure all **the** ERMslooked very similar to that **of** session 1. The only exceptions were **the** ERMs from session 4**of** **the** FI40 rat and session 5 **of** **the** VI40/20 rat, which showed more diffuse distributions andhad fewer data points.DiscussionSummery **of** **the** traditional analysisThe goal **of** **the** traditional analysis was to confirm that **the** behavior generated in **the**present study corresponded to **the** conventional responding generated by similar Skinner-boxexperiments. This object was accomplished through **the** following findings: Firstly, **the**overall number **of** lever presses across sessions showed that **the** shaping **of** **the** operantbehavior and **the** training **of** different reinforcement schedules were successful. Secondly: **the**Fig.11 The ERMs **of** **the** FI20 rat from all sessions. There is already weak organization in **the**ERMs from sessions 2 and 3. Clear lattice patterns appear from session 4 on.17

Fig.12 The ERMs **of** **the** FI40 rat from all sessions. Weak organization appears in sessions 5and 6. Clear lattice patterns do not form until session 7. In session 4 **the** distribution **of**data points is very diffuse.Fig.13 The ERMs **of** **the** VI40 rat from all sessions. Clear lattice patterns appear in sessions 6and 7. In session 5 **the** distribution **of** data points is very diffuse.analysis **of** **the** frequency distribution **of** IRTs revealed characteristics **of** a Poisson process,which is consistent with previous reports (e.g. /38/). Finally, both **the** reconstructedcumulative records from individual reinforcements and **the** averaged responses over allrewards showed **the** expected “scalloped” curve and **the** post reinforcement pause that aretypical for behavior under interval-schedules. In conclusion, it is justified to regard **the**18

present study as a standard conventional Skinner-box experiment. Thus, **the** results **of** **the**nonlinear dynamic analysis can be generalized to o**the**r operant studies under comparableexperimental conditions.Summery **of** **the** nonlinear dynamic analysisOne achievement **of** **the** present work is **the** improved analyzing ability **of** **the**“Extended Return Map” over **the** original “Return Map”. Comparison between figures 6 and7 reveals that **the** well organized lattice structures in **the** ERMs **of** rats under **the** control **of**interval-schedules are not visible in RM. Also **the** difference **of** behavioral dynamics betweenanimals under interval- and ratio-schedules becomes visible only in **the** ERMs. The patternsseen in **the** maps is comparable to **the** trajectory in **the** phase space and reflects dynamicproperties **of** **the** system under study /18,32/. Thus, we can state that **the** patterns in ERM aretraces **of** **the** behavioral state **of** **the** animals during **the** experiment. The regular latticestructures in **the** ERMs **of** interval-schedule controlled rats suggest a kind **of** periodicity **of** **the**operant behavior.However, since time series data from an experimental study contain only limitednumber **of** elements and are **of**ten contaminated with noise, **the** absolute results **of** a nonlineardynamic analysis could be erroneous and might indicate false deterministic relationships **of****the** system’s dynamics. To exclude this possibility, it is necessary to look at **the** surrogatedata sets, which share **the** same first order correlation with **the** experimental data, but have **the**higher order correlation destroyed /23,37/. In **the** present work, we produced surrogate datasets using exactly **the** same elements as **the** experiment. We simply randomized **the**irsequential order. Hence, both **the** surrogate and **the** experimental data sets had **the** samemean and frequency distribution. However, if **the**re were any sequential dependency in **the**experimental IRTs, this relationship would be destroyed.The direct comparison between **the** surrogate and **the** experimental ERMs shows that,after randomization, **the** patterns found in **the** interval-schedule controlled rats almostvanished. This indicates that **the** lattice structures seen in **the** ERMs **of** **the**se rats really reflectdynamics **of** operant behavior under this condition. There were also differences betweensurrogate and experimental ERMs **of** **the** FR rat, however, **the**se are not so obvious. Although**the** direct graphical comparison is very impressive, we cannot be certain, that **the** surrogateERMs shown in fig. 8 are representative, nor can we make any statistical test to quantify **the**difference. To overcome this problem, we first had to calculate an entity able to representsome geometrical features **of** **the** patterns found in **the** ERMs, and **the**n obtain **the** same entityfor **the** surrogate ERMs in order to perform a statistical comparison. In **the** present work, wechose **the** correlation and **the** information dimensions as entities to be calculated andcompared. The results showed that **the** probability to obtain **the** experimental sequence **of** IRTdata from interval-schedule controlled rats as well as **the** FR20 rat is very low, since **the**experimental correlation dimensions are very far apart from **the** central area **of** **the** distributioncurve **of** **the** surrogate dimension. Similar results in **the** information dimension were als**of**ound for **the** interval-schedule controlled rats. The VR20 schedule, on **the** contrary,produced an IRT sequence which was indistinguishable from **the** surrogate data from **the**standpoint **of** both **the** correlation and information dimensions **of** **the** ERM.Fractal dimensions reflect very important geometrical features **of** **the** patterns understudy. In phase spaces **of** systems with complicated dynamics, **the**y usually indicate **the**chaotic properties **of** **the** attractors generated by **the** systems. However, it has not been provenma**the**matically that **the** reconstructed diagrams using ERMs are equivalent to **the** originalphase space. Thus, **the** meaning **of** **the** correlation and **the** information dimensions calculatedin **the** surrogate studies is unclear. However, this deficiency does not impair **the** goal **of** **the**19

surrogate study, because **the** fractal dimensions **the**mselves still reflect **the** differencesbetween patterns seen in **the** experimental and **the** surrogate ERMs.Ano**the**r interesting application **of** **the** ERM is to trace **the** course **of** **the** learningprocess in an operant experiment. As mentioned previously, **the** learning process is bydefinition a dynamic process. With **the** ERM as an analyzing tool, we can follow **the** change**of** **the** animal’s behavioral dynamics over time. In figs. 11-13 we can see when **the**stereotypical lattice patterns in **the** ERM **of** an interval-schedule controlled rat began to occur,and we can fur**the**r observe how **the**se patterns gradually developed, and which intermediatestate would appear before **the** final formation **of** **the**se patterns.Models for operant behavior under fixed-interval reinforcement schedulesOur next step was to construct a modelthat can produce artificial IRTs which match **the**experimental results found in session 7 from **the**fixed-interval schedule controlled rats. Themodel was to be basically a kind **of** “pointprocess”; that is, it was intended to describe **the**time dependent changes **of** **the** probability **of**lever pressing behavior. The first idea for **the**construction **of** such models came from **the**“scalloped” curve in **the** cumulative records.Similar to **the** lattice structures in **the** ERMs,this stereotypical pattern in **the** cumulativerecords also developed over **the** course **of** **the**learning process, as indicated in figs. 4 and 5.In addition to that, **the** IRT distribution is animportant condition that a point process modelmust fulfill. Theoretically, **the**re are an infinitenumber **of** different ways to build models thatcan generate IRTs to match such experimentalcumulative records and IRT distributions. Thequestion is, whe**the**r **the**se two conditions aresufficient to completely describe **the** operantbehavior under **the** interval-schedule. If, forexample, we can build a model that cansuccessfully simulate **the** cumulative recordsand IRT distributions, but fails to generateERMs that can match **the** experimental results,**the**n we can be sure that **the** ERM is able toshow some properties that are not visible in **the**cumulative records and **the** IRT distributions.Fur**the**rmore, if a small modification can makethis model fulfill all **the** three conditions, **the**nthis modification might represent a veryimportant feature for **the** dynamics **of** **the**operant behavior under this conditions.Table 2Parameters for **the** simulationsModel IModel IIA 00.0042 0.0044M 00.2 0.2O 00.0016 0.00005A 10.7 0.9M 10.65 0.16O 10.015 0.02A 20.45 0.62M 20.31 0.166O 20.02 0.03A 30.02 0.06M 30.01 0.05O 30.002 0.01K 2 ×10 - 9 6 ×10 - 11ReinforcementScheduleTotal RunTime (sec.)FI 40 FI 405400 5400Following this strategy, we designed two models, both **of** which were able to generateIRT data sets that matched **the** IRT distributions and **the** cumulative records. The mainconcept for **the** design **of** **the** models can be divided into two parts. The first part describes**the** probability for an operant behavior to occur, and it can be fur**the**r divided into two20

Fig.14 Results **of** **the** simulations. Upper panel: model I; lower panel: model II; leftcolumn: IRT distributions; middle column: averaged responding rate over timeafter rewards; right column: ERMs. Parameters used for ERM: fold = 15, time lag=15. Compared to figs. 2, 5 and 7, both models I and II can produce IRTdistribution and averaged response after rewards similar to **the** experimental results,but only model II produces ERM that can match **the** experimental results.subunits. The first subunit is a function that depends on **the** time after **the** delivery and **the**consummation **of** rewards. The second subunit consists **of** three sigmoid functions which aredependent on **the** time after each response. How **the**se two subunits are interconnected,represents **the** main difference between **the** two models. While in **the** first model **the** twosubunits are interconnected with a multiplication-operation, in **the** second model **the**y areinterconnected with a switch-operation. That is, after **the** reward is consumed and before **the**first operant response occurs, **the** final probability for an operant behavior to occur is decidedby **the** first subunit. After **the** first response, it is decided by **the** second subunit. We canregard this switch between reacting modus as a switch between behavioral states. The firstmodel, on **the** contrary, is a continuous function.The second part **of** **the** two models is designed to account for **the** “overshooting”phenomena. Since **the** feeding machine required some time to execute **the** command for“reinforce” from **the** s**of**tware, **the**re were always small delays between **the** time **of** reward and**the** time at which **the** animals could actually consume **the** reward. Besides, **the** animalssometimes did not respond for **the** presence **of** **the** reward at once, and continued to leverpress for a while. This resulted in a number **of** operant responses immediately after rewards,as indicated in **the** first column **of** **the** averaged cumulative records (fig. 4). Therefore wedesigned a time dependent function to describe **the** probability that **the** animal actually took or21

consumed **the** reward after it was recorded as “available” by **the** s**of**tware. This part wasactive only during **the** time between **the** availability and **the** consuming **of** a reward.The detailed formulation **of** **the** models is outlined in **the** appendix. The parametersused in **the** simulations are outlined in table 2. The parameters were carefully adjusted so that**the** simulated IRT distributions and **the** averaged rate **of** responding after rewards could match**the** experimental results by visual examination. The simulated IRT distributions, **the** averagedrate **of** responding after rewards, and **the** ERMs are shown in figure 14, from which it isapparent that both models could produce simulations by which **the** IRT distributions and **the**averaged response rate after rewards were well matched with **the** experimental results. Thefirst model (left column), however, produced an ERM that is completely different from that **of****the** experimental data. The second model (right column), on **the** contrary, produced an ERMthat looks very similar to **the** experimental ERM in figure 7.In summary, this simulation reveals that **the** cumulative records and **the** IRTdistribution alone are not sufficient to completely describe **the** dynamics **of** **the** operantbehavior under fixed-interval schedule, whereas **the** ERM accomplishes this task.Fur**the**rmore, **the** simulation also suggests that a switch between two behavioral states playsan important role in **the** operant behavior under interval-schedules.ConclusionsThe present work shows that **the** ERM has, in comparison with **the** original RM, animproved analyzing ability for **the** dynamics **of** operant behavior. Especially in **the** datagenerated by interval-schedules, **the**re are patterns in **the** ERMs that were not visible in **the**original RMs. The examination with **the** surrogate data procedure in combination with **the**computation **of** correlation and information dimensions fur**the**rmore revealed that **the** patternsfound in **the** ERMs characterize dynamics **of** **the** operant behavior under interval- and fix-ratioschedules, because if **the** sequential order **of** **the** IRT data was destroyed, **the** patterns shownin **the** ERMs vanished and **the** ERMs had different correlation as well as informationdimensions.We fur**the**rmore studied **the** time course **of** **the** whole learning process involved inestablishing behavioral control with intermittent reinforcement schedules with ERMs. Cleardifferences could be seen in **the** ERMs from different sessions. Of course, since **the**re wasonly one rat used under each condition, it is not proven whe**the**r or not this finding can begeneralized to operant behavior under **the**se conditions as a whole. From ano**the**r point **of**view, **the** ERM is designed to study **the** behavioral dynamics **of** individual animals. Thus, itshould not be surprising if we should find individual differences in **the** behavior **of** animalsunder **the** same experimental conditions, including a distinctively different ERM. Thequestion is, whe**the**r we can find an entity that can quantify **the** patterns found in **the** ERM andwhe**the**r this entity can be used to represent **the** behavioral dynamics **of** an experimentalgroup. The fractal dimension used in **the** surrogate data study might be suitable for thispurpose. In **the** present work, we did not compare between fractal dimensions from rats underdifferent reinforcement schedules, because **the**re was only one rat run under each conditionand this kind **of** comparison would not have made much sense.A next possible improvement would be a systematic study **of** **the** parameters used in**the** reinforcement schedule; e.g., **the** length **of** **the** inter-reinforcement interval defined in **the**FI schedule. In **the** present study, we implemented only 20 and 40 seconds for **the** FIschedule, and **the** simulation for **the** FI40 schedule using model II suggests that **the** switchbetween two behavioral states is essential for **the** dynamics **of** operant behavior under thiscondition. If we use a longer period in **the** FI schedule, for example 120 seconds, will **the**rebe a third or even more behavioral states in **the** period between two adjacent rewards?22

The study on simulation models also revealed very interesting results. As mentionedpreviously, **the** comparison between **the** two models suggests that a nonlinear switch betweentwo behavioral states plays an essential role in **the** operant behavior under **the** control **of**interval-schedules. Although similar observations were reported previously /9,19,27,31/, it is**the** ERM and **the** study using two simulation models that definitely demonstrated **the**necessity **of** this nonlinear switch **of** behavioral states. In **the** present work we performed onlysimulations for **the** FI40 rat. It might be easy to simulate **the** FI20 and VI40 rats as well byusing different sets **of** parameters, since **the** cumulative records, **the** IRT distributions, and **the**ERMs **of** **the**se rats looked very similar.Ano**the**r direction for fur**the**r study is to implement drug and/or lesion manipulations.Although **the**re are many reports about **the** influence **of** drug and lesion treatments, **the**reexists hardly any work on such effects on dynamics **of** operant behavior. Most availablestudies have dealt solely with changes in “averaged” rate **of** responding. Since **the** averagedoperant response is **the** consequence **of** **the** behavioral dynamics **of** **the** animals, it will be veryinteresting to see what kind **of** change in **the** dynamics can be produced through **the**se kinds **of**treatments.In addition to Skinner-box experiments, **the** ERM can also be applied to o**the**rparadigms in behavior research. For example, self stimulation **of** **the** brain using electricalstimulations as a reinforcer is a good candidate, since **the** type **of** data from this experiment isalso a kind **of** point process. There would be no need to make any modification **of** **the** ERM,and **the** possibility for **the** experimental setups is even more manifold. Theoretically, **the**ERM can be applied to any kind **of** point process. Of course, **the** number **of** data pointsrepresents an important constraint for this method. For a reliable analysis, **the** number **of** datapoints required is in **the** order **of** hundreds.Finally, a thorough study **of** **the** ma**the**matical properties **of** **the** ERM itself would bean important fur**the**r step. As mentioned in **the** introduction, **the** ma**the**matical foundation **of****the** delay-coordinate method is based mainly on **the** work by Takens /35/. The assumptionsmade in his **the**ory, such as **the** continuity **of** **the** function, cannot be fulfilled, stringentlyspeaking, by any real experiments. Owing to **the** irregular measurements, **the** extension **of**this method to **the** RM presents even more problems regarding **the** ma**the**matical validity **of**Takens **the**ory /18/. Here, in **the** definition **of** ERM, we introduce **the** computation **of** **the**running average, and **the**reby complicate **the** situation even more. In **the** present work, wesimply empirically demonstrated how to use **the** ERM in **the** analysis **of** behavioralexperiments. To what extent we can still, ma**the**matically, rely on **the** Takens **the**ory, was notdiscussed at all, and this would be a very important question for **the** future application **of** **the**ERM.Appendix:The detailed formulation **of** **the** models is outlined as following:Model I : Φ (t) = F 0 (t a ) × F 1 (t b ) (8)Model II: Φ (t) =F 0 (t a ) after **the** taking or consummation **of** a reward andbefore **the** occurrence **of** **the** first responseor F 1 (t b ) after **the** first response (9)Where Φ (t) is **the** probability for an operant response to occur. The definition **of** **the** twomodels differs only in **the** equations (8) and (9). All o**the**r equations below are valid for bothmodels. The functions F 0 (t a ) and F 1 (t b ) are defined as follows:23

F 0 (t a ) = F 0 (t a - 1) +A 0 × F 0 (t a - 1) × [M 0 - F 0 (t a - 1)] (10)with initial condition:F 0 (0) = O 0 (11)andF 1 (t b ) = R 1 (t b ) – R 2 (t b ) + R 3 (t b ) (12)The functions R 1 (t b ), R 2 (t b ) and R 3 (t b ) are defined as follows:R 1 (t b ) = R 1 (t b - 1) +A 1 × R 1 (t b - 1) × [M 1 - R 1 (t b - 1)] (13)R 2 (t b ) = R 2 (t b - 1) + A 2 × R 2 (t b - 1) × [M 2 - R 2 (t b - 1)] (14)R 3 (t b ) = R 3 (t b - 1) + A 3 × R 3 (t b - 1) × [M 3 - R 3 (t b - 1)] (15)with initial conditions:R 1 (0) = O 1 , R 2 (0) = O 2 and R 3 (0) = O 3 (16)Here t a and t b were discrete counters which represented **the** time after **the** taking **of** or **the**consummation **of** a reward (t a ), and **the** time after an operant response (t b ), respectively.When a reward was consumed, and when an operant response was emitted, **the** counters t a aswell as t b would be reset to 0. The probability for a reward to be taken was determinedthrough **the** following formula:Ψ(t) = k × t c3 × tb2where **the** Ψ (t) was **the** probability that a reward would be taken after it was available. Here**the** t c was again a discrete counter which represented **the** time after **the** presentation **of** areward. The definition for **the** counter t b was **the** same as above. The units **of** all timecounters t a , t b and t c was 0.01 second. The parameters A 0-3 , M 0-3 , O 0-3 and k can be adjusted,so that **the** IRT distribution and **the** averaged cumulative records generated from **the** simulatedIRT data sets can match **the** experimental results.Acknowledgement:References:Supported by a grant (Hu.306/11-3) from **the** German National Science Foundation.1. Baker GL, Gollub JP. Chaotic dynamics - an introduction. **New** York: CambridgeUniversity Press,1990.2. Bizo LA, Killeen PR. Models **of** ratio schedule performance. Journal **of** ExperimentalPsychology: Animal **Behavior** Processes 1997; 23(3):351-367.3. Branch MN, Gollub LR. A detailed analysis **of** **the** effects **of** d-amphetamine onbehavior under fixed-interval schedules. Journal **of** **the** Experimental Analysis **of****Behavior** 1974; 21:519-539.4. Braun T, Lisbôa JA. Characterization **of** homoclinic chaos in a glow discharge throughreturn maps. International Journal **of** Bifurcation and Chaos 1994; 4(6):1483-1493.(17)24

5. Church RM, Meck WH, Gibbon J. Application **of** scalar timing **the**ory to individualtrials. Journal **of** Experimental Psychology: Animal **Behavior** Processes 1994;20(2):135-155.6. Dews PB. The **the**ory **of** fixed-interval responding. In: Schoenfeld WN, editors. The**the**ory **of** reinforcement schedules. **New** York: Appleton-Century-Cr**of**ts, EducationalDivision, Meredith Corporation, 1970; 43-61.7. Dews PB. Studies on responding under fixed-interval schedules **of** reinforcement: IIThe scalloped pattern **of** **the** cumulative record. Journal **of** **the** Experimental Analysis **of****Behavior** 1978; 29:67-75.8. Ferster CB, Skinner BF. Schedules **of** reinforcement. **New** York: Appleton-Centrury-Cr**of**ts, 1957.9. Gentry GD, Weiss B, Laties VG. The microanalysis **of** fixed-interval responding,Journal **of** **the** Experimental Analysis **of** **Behavior** 1983; 39:327-343.10. Gibbon J. Scalar expectancy **the**ory and Weber’s Law in animal timing. PsychologicalReview 1977; 84(3):279-325.11. Gleick J. Chaos - making a new science. **New** York: Penguin Books, 1987.12. Grassberger P. Generalized dimensions **of** strange attractors. Physics Letters, 1983;97A(6):227-230.13. Guzzetti S, Signorini MG, Cogliati C, Mezzetti S, Porta A, Cerutti S, Malliani A. **Nonlinear**dynamics and chaotic indices in heart rate variability **of** normal subjects and **the**heart-transplanted patients. Cardiovascular Research 1996; 31:441-446.14. Hentschel HGE, Procaccia I. The infinite number **of** generalized dimensions **of** fractalsand strange attractors. Physica D 1983; 8:435-444.15. Herrnstein RJ. On **the** law **of** effect. Journal **of** **the** Experimental Analysis **of** **Behavior**1970; 13:243-266.16. Herrnstein RJ. Formal properties **of** **the** matching law. Journal **of** **the** ExperimentalAnalysis **of** **Behavior** 1974; 21:159-164.17. Hoyert MS. Order and chaos in fixed-interval schedules **of** reinforcement. Journal **of****the** Experimental Analysis **of** **Behavior** 1992; 57:339-336.18. Kantz H, Schreiber T. **Nonlinear** time series analysis. **New** York: CambridgeUniversity Press, 1997.19. Killeen P, Fetterman G. A behavioral **the**ory **of** timing. Psychological Review 1988;95(2): 274-295.20. Killeen P. Ma**the**matical principles **of** reinforcement, **Behavior**al and Brain Sciences1994; 17:105-172.21. Li JS, Huston JP. The extended return map: a new method to analyse **the** non-lineardynamics **of** inter-response-intervals in Skinner-box experiments. In: Fourth EuropeanMeeting for **the** Experimental Analysis **of** Behaviour, Amiens, France, July 9-13, 2000.22. Li JS. The extended return map: a new method to analyze **the** nonlinear dynamics **of**inter-response-intervals in operant conditioning studies. In: Tenth Annual InternationalConference **of** **the** Society For Chaos Theory in Psychology & Life Sciences, University**of** Pennsylvania; Philadelphia, Pennsylvania; July 20-23, 2000.25

23. Liebovitch LS. Fractals and chaos: simplified for **the** life sciences. **New** York: OxfordUniversity Press, 1998.24. Machado A. Learning **the** temporal dynamics **of** behavior. Psychological Review 1997;104(2):241-265.25. Machado A, Cevik M. Acquisition and extinction under periodic reinforcement.Behavioural Processes 1998; 44:237-262.26. May RM. Simple ma**the**matical models with very complicated dynamics. Nature 1976;261:459-467.27. Nevin JA, Baum WM. Feedback functions for variable-interval reinforcement, Journal**of** **the** Experimental Analysis **of** **Behavior** 1980; 34:207-217.28. Ott E, Sauer T, Yorke JA. Coping with chaos: analysis **of** chaotic data and **the**exploitation **of** chaotic systems. **New** York: John Wiley & Sons, Inc., 1994.29. Packard NH, Crutchfiel JP, Farmer JD, Shaw RS. Geometry from a time series.Physical Review Letters 1980; 45:712-716.30. Palya WL. **Dynamics** in **the** fine structure **of** schedule – controlled behavior. Journal **of****the** Experimental Analysis **of** **Behavior** 1992; 57:267 – 287.31. Schneider AB. A two-state analysis **of** fixed-interval responding in **the** pigeon. Journal**of** **the** Experimental Analysis **of** **Behavior** 1969; 12:677-687.32. Shaw RS. The dripping faucet as a model chaotic system. Santa Cruz CA: Aerial Press,1984.33. Skinner BF. The behavior **of** organisms. **New** York: Appleton-Century-Cr**of**ts 1938.34. Suzuki H, Aihara K, Murakami J, Shimozawa T. Analysis **of** neural spike withinterspike interval reconstruction. Biological Cybernetics 2000; 82:305-31135. Takens F. Detecting strange attractors in turbulence, In: Rand DA, Young LS, editors.Lecture notes in ma**the**matics: dynamical systems and turbulence 898. Berlin: Springer–Verlag, 1981; 336-381.36. Takens F. On **the** numerical determination **of** **the** dimension **of** an attractor. In:Braaksma Boele LJ, editors. Lecture notes in ma**the**matics: dynamical systems andbifurcations 1125. Berlin: Springer–Verlag, 1985; 99-106.37. Theiler J, Eubank S, Longtin A, Garldrikian B, Farmer JD. Testing for nonlinearity intime series: **the** method **of** surrogate data. Physica D 1992; 58:77-94.38. Weiss B. The fine structure **of** operant behavior during transition states, In: SchoenfeldWN, editor. The **the**ory **of** reinforcement schedules. **New** York: Appleton-Century-Cr**of**ts, Educational Division, Meredith Corporation, 1970; 277-311.26