Gabor-Type Filtering in Space and Time - Department of Electronic ...

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998 121Gabor-Type Filtering in Space and Timewith Cellular Neural NetworksBertram E. Shi, Member, IEEEAbstract— Gabor filters are preprocessing stages in imageprocessingand computer-vision applications. One drawback isthat they are computationally intensive on a digital computer.This paper describes the design of cellular neural networks(CNN’s) which compute the outputs of filters similar to Gaborfilters. Analog VLSI implementations of these CNN’s might eventuallyrelieve the computational bottleneck associated with Gaborfiltering image-processing algorithms. The CNN’s compute boththe real and imaginary parts of the filter outputs simultaneously,which is an important feature in applying them in algorithmsutilizing the phase of the Gabor output.Index Terms—Analog circuits, cellular neural networks, filtering,image processing, neural networks.I. INTRODUCTIONGABOR filters [1] have been used as preprocessors fordifferent tasks in computer vision and image processing.These approaches to image processing and computer visionhave been motivated partially by the discovery that the responsesof orientation selective cells in the visual cortex canbe modeled using Gabor filters [2], [3]. Because image velocitycan be considered as an orientation in the space–time domain,three-dimensional (3-D) Gabor filters have been used to modelcortical cells’ velocity and directional sensitivity [4]. Initialevidence indicates that approaches based upon Gabor filteringcan outperform previously developed approaches [5].One drawback of Gabor filtering approaches is that they arecomputationally intensive. Here we describe how to implementfilters similar to the Gabor filter using cellular neural networks(CNN’s) [6]–[8]. The advantage of CNN’s is that they can beimplemented in analog VLSI alongside photosensors whichsense the image [9]–[11]. The filter outputs can be computedin less time than required by serial digital computer implementationsand be read off the chip directly, relieving thecomputational bottleneck of preprocessing with Gabor filters.The remainder of Section I reviews Gabor filters and definesthe class of “Gabor-type” filters which we believe capture theimportant properties of Gabor filters exploited by many imageprocessingand computer-vision applications. It concludes witha short review of previous related work. Section II shows thatany low-pass spatial filter implemented on a CNN can betransformed into a corresponding Gabor-type filter and givesManuscript received September 11, 1996; revised February 26, 1997. Thiswork was supported by the Hong Kong Research Grants Council under GrantHKUST675/9E. This paper was recommended by Associate Editor J. Pinedade-Gyvez.The author is with the Department of Electrical and Electronic Engineering,Hong Kong University of Science and Technology, Kowloon, Hong Kong.Publisher Item Identifier S 1057-7122(98)00889-7.several illustrative examples of the types of filters which canbe achieved on CNN’s. Section III extends the results to spatio–temporalfilters. Finally, Section IV summarizes our resultsand outlines ongoing research in designing and fabricatinganalog VLSI chips implementing these CNN’s and applyingthem in computer vision.In the following, small letters denote space and time waveforms,e.g., , ,or where represents continuousspace, represents discrete space, and represents continuoustime. Capital letters denote Fourier transforms. Transforms ofa continuous waveform or will be written asor , while transforms of a discrete waveform willbe written as .A. Gabor FiltersGabor filters exist for signals of arbitrary dimension wherean -dimensional signal is defined to be a mapping fromto or . For one-dimensional (1-D) signals, the impulseresponse of a Gabor filter is a complex exponentialfunction modulated by a Gaussianwhere is the angular frequency of the complex exponentialand is the standard deviation of the Gaussian. Fig. 1(a)and (b) plot the real and imaginary parts of (1). Although theimpulse response is defined for continuous , since the Gabortypefilters implemented by CNN’s are inherently discretespace, we plot the impulse responses at discrete integer valuesof to facilitate comparison with later results.By the Fourier shift theorem, the frequency response of aGabor filter is a Gaussian function centered at :The Gabor filter is a bandpass filter tuned to frequencies near[see Fig. 1(c)]. Generalizing to -dimensional signals, theconvolution kernel and frequency response arewhere , the covariance matrix is positive(1)1057–7122/98$10.00 © 1998 IEEE

122 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998(a) (b) (c)(d) (e) (f)Fig. 1.(g) (h) (i)A comparison of the impulse and frequency responses of the 1-D Gabor filter and Gabor-type filters implementable on a CNN. The filter parametershave been chosen so that the squared errors between the impulse responses are minimized. (a)–(b) The real and imaginary parts of the impulse response ofa Gabor filter with = 3:32 and !xo =0:93 are sine and cosine functions modulated by a Gaussian (dotted line). The responses are plotted at integervalues of x to facilitate comparison with the other plots. (c) The frequency response of the Gabor filter is a Gaussian centered at !xo. (d)–(e) The real andimaginary parts of the impulse response of the Gabor-type filter in Example 1 for =0:3and !xo =0:93 are sine and cosine functions modulated bya function which decays exponentially away from the origin (dotted line). (f) The frequency response of the filter in Example 1 is the Fourier transformof the modulating function shifted to !xo. Since the filter is defined for discrete space, the frequency response is periodic with period 2. (g)–(h) Thereal and imaginary parts of the Gabor-type filter in Example 2 for =0:352 and !xo =0:93 more closely approximate the Gabor. (i) The frequencyresponse of the filter in Example 2 is approximately Gaussian, but is flatter at the peak at !xo.definite, and. The covariance matrix is oftenchosen to be a scalar multiple of the identity matrix.Two-dimensional (2-D) Gabor filters are orientation selectiveand have been used to model receptive fields of orientationselective neurons in the visual cortex. For example, the Gaborfilter with impulse responseis tuned to spatial frequencyThis filter respondsmaximally to edges which are oriented at an anglewhere is defined to be the angle betweenthe horizontal axis and the line perpendicular to theedge (see Fig. 2). 2-D Gabor filters have found applicationsin computer-vision algorithms for stereo vision [12]–[18],binocular vergence control [19], texture segmentation [20],and face recognition [21].Three–dimensional spatio–temporal Gabor filters have beenused for image-motion analysis [22]–[24]. Image motion canbe characterized as an orientation in the space–time domain[4]. The energy spectrum of a 1-D image undergoing uniformtranslation at velocity is nonzero only along the line. For 2-D images translating with velocity ,the spectrum is nonzero only along the plane. A spatio–temporal bandpass filter tuned to frequencycan be considered to be tuned to the velocity

SHI: GABOR-TYPE FILTERING IN SPACE AND TIME WITH CELLULAR NEURAL NETWORKS 123filter’s output, its magnitude and phase are(a)The phase of the output at a given pixel is related to thelocation of edges and other features in the input image nearthat pixel. Translating the image input results in a phase shift inthe Gabor output at a given pixel. This property has motivatedthe development of “phase-based” approaches to stereo-visionand image-motion analysis. Phase differences between theGabor filter outputs from two stereo images can be used toestimate disparity [12]–[18] and control binocular vergence inactive-vision systems [19]. Fleet [24] showed that the temporalvariation of phase is a robust indicator of the local imagevelocity. Barron et al.’s comparison [5] of algorithms foroptical-flow estimation indicates that Fleet’s algorithm usingGabor phase is the most accurate among those tested. Manyphase-based algorithms use the amplitude as a confidencemeasure for the reliability of the phase measurement.B. Gabor-Type FiltersWe extend Gabor filters to Gabor-type filters by allowingmodulating functions other than the Gaussian. Formally, definea filter to be Gabor-type if its impulse response can beexpressed as a complex exponential modulated by a real valuedenvelope , which is the impulse response of a low-passfilterAll Gabor filters are Gabor-type. The frequency responseof a Gabor-type filter is equal to the Fourier transformof the modulating function shifted to :(b)Fig. 2. The (a) real and (b) imaginary parts of the impulse response of a2-D Gabor filter which is tuned to the orientation = =4.Spatio–temporal filtering approaches to estimating the opticalflow often compare the outputs of filters tuned to differentregions in the spatio–temporal frequency domain. The opticalflow is the projection of the 3-D motion vector field resultingfrom relative motion between a camera and its environmentto a 2-D motion vector field in the image plane. Estimation ofthe optical flow is one of the first steps in many algorithmswhich extract the shape of surfaces being imaged by a movingcamera [25].For computer-vision applications, the output of a Gaborfilter is often expressed in terms of its magnitude and phase. Ifand are the real and imaginary parts of a GaborGabor-type filters do not necessarily optimize the uncertaintyrelation between the resolution in space and spatial frequency,a commonly cited advantage of Gabor filters resulting fromtheir Gaussian modulating function. However, they can bedesigned to satisfy the requirements of filters used for phasebasedalgorithms in computer vision. In their phase-baseddisparity study, Westelius et al. [18] examined the use of filtersother than the Gabor and found that the overall performanceof their algorithm “was not critically dependent upon thetype of disparity filter used.” They identified several desirableproperties for 1-D filters used to compute phase.1) The filter has no dc component.2) The filter is sensitive to positive frequencies only.3) The filter is insensitive to singular points.4) The phase of the impulse response does not wrap around(i.e., exceed ).5) The filter has small spatial support.6) The phase of the impulse response is monotonous.

124 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998In order for a Gabor-type filter to possess the first threeproperties, its bandwidth should be small in comparison withthe center frequency , which we assume to be positive. Ifis identically zero outside the range ,the response of the filter to dc and negative frequenciesis identically zero. In general, the frequency response of aGabor-type filter may have infinite support. However, becauseof the low-pass nature of , it is possible to make themaximum response to dc and negative frequencies smallerthan any fixed positive threshold by decreasing the bandwidthsufficiently. In addition, even if the dc response is not zero,an extra processing step can easily be added to remove thedc component. Westelius et al. point out that the sensitivity tosingular points also decreases with bandwidth.On the other hand, the fourth and fifth properties imply alarge bandwidth since they constrain the width of the impulseresponse. The fourth requirement is satisfied by Gabor-typefilters if is identically zero outside the range. For filters which do not satisfy this condition, theeffects of phase wraparound can be minimized by ensuringthat width of is small enough that the impulse responseis close to zero outside this range. In general, the fourth andfifth properties are less critical than the first three. Westeliuset al. state that the fourth requirement is “desirable, but notabsolutely necessary.” The fifth property, small spatial support,was included since it determines the computational cost ofthe filter. However, one goal of this paper is to decreasethe computational cost of Gabor-type filters by enabling theirimplementation in analog VLSI.The sixth property can be satisfied by Gabor-type filterswith both large and small bandwidths. Any Gabor-type filterin which is strictly positive satisfies this property. Forfilters that do not satisfy this property, the effect of phasenonmontonicity can be alleviated by ensuring that the absolutevalue of the low-pass envelope is small when it changes sign.In summary, a Gabor-type filter can be used effectively inphase-based disparity estimation if its bandwidth is chosento be small enough and if the modulating function is strictlypositive or at least positive over the region where its magnitudeis significant. Although Westelius et al.’s study was confinedto disparity estimation, we expect that the properties theyhave identified to be applicable to filters for other phase-basedalgorithms, such as for image-motion analysis. For example,the Gabor-type filter in Example 1 has been used in a phasebasedalgorithm for extracting time-to-contact from a movingimage taken by a camera translating toward a planar surface[26].C. Previous WorkRaffo has demonstrated that a resistive network similar tothat shown in Fig. 5 followed by an additional processing stagecan implement a real valued Gabor-type filter with arbitraryphase [28]. In particular, for 1-D signals the convolution kernelhas the formwhere can be chosen arbitrarily. For , this is thereal part of convolution kernel implemented by the networkdescribed in Example 1. For , it is the imaginary part.In some ways, the two networks are similar in complexityand capability. Raffo’s network requires connections to secondnearest neighbors. Although the network of Example 1 requiresonly nearest neighbor interconnections, each pixel requirestwo nodes. The network of Example 1 computes both realand imaginary parts of the Gabor-type outputs simultaneously,but this could be done by Raffo’s network by adding twoadditional processing stages rather than one. The arbitraryphase achieved by Raffo’s network could be obtained bya linear combination of the two outputs of the network ofExample 1.One advantage of the network of Example 1 is that thedependency of the conductances and transconductances onthe parameters and , which determine the shape ofthe convolution kernel, are simpler than for Raffo’s network.For example, the resistor between nearest neighbor nodes forRaffo’s network has conductancewhereThe transconductances in the second processing stage are alsocomplex functions of , , and . On the other hand,for the network of Example 1 is uniquely specified bythe ratio of the conductances and . In addition, if. The methodologypresented here can also be used to implement a wide class ofGabor-type filters, of which Example 1 is only one example.In addition, this paper discusses not only spatial filtering,but also spatio–temporal filtering. Previous work in this areaincludes Delbruck’s successful VLSI implementation of avelocity-sensitive filter using a delay line consisting of acascade of first-order filter stages [29]. Unlike the Gabor-typefilters proposed here, the spatio–temporal filters implementedby the delay line are not separable in space and time likethe Gabor filters. Space–time separability can be exploited incomputing the optical flow [23]. In addition, the delay linedoes not generate the complex valued filter output critical tophase-based algorithms.II. SPATIAL FILTERSThis section introduces CNN architectures for Gabor-typespatial filtering. For simplicity, we introduce architectures for1-D images before generalizing to two dimensions.A. 1-D ImagesThe CNN is a neural-network architecture consisting ofan array of neurons, called “cells.” Each cell is a first-ordercontinuous-time dynamical system. To filter an -pixel 1-Dimage, where , we use a 1-DCNN array of cells where the state at the th cellsatisfies(2)

SHI: GABOR-TYPE FILTERING IN SPACE AND TIME WITH CELLULAR NEURAL NETWORKS 125The dot denotes differentiation with respect to time. Theand are complex coefficients called thefeedback and feedforward cloning templates and is definedto be the connection radius. We represent the feedback cloningtemplate using amatrix where the center elementequals . For example, for , the cloning template matrixisThis CNN equation is slightly different than that presentedin [6] and [7]. The key differences are that here the state iscomplex rather than real and the bias and nonlinearity areexcluded. However, (2) can be considered to be a special caseof a two-layer CNN as presented in [6], where: 1) the cells inthe first layer represent the real parts of the complex valuedstate; 2) the cells in the second layer represent the imaginarypart; 3) the bias terms are identically zero; and 4) the outputalways operates in the linear region of the output nonlinearity.Thus, the hardware complexity of the CNN defined here iscomparable to that of a two-layer CNN.Since the dynamics of the CNN in (2) are purely linear, thestability of any template can be evaluated theoretically [30]. Ifthe CNN is stable, then for an input which is constant in time,the state settles to a unique equilibrium point which isa spatially filtered version of the input . The steady-statevalue of is the output of the “computation” performed bythis CNN. The length of the transient is the time required toperform the computation. Assume that the CNN array consistsof an infinite number of cells indexed by .Define the discrete-space Fourier transforms of the input andoutput to bedesigning CNN’s for low-pass filtering in [31]. The followingfocuses on mapping a template implementing a low-pass filterto the template which implements the corresponding Gabortypefilter.Suppose that the cloning templates and are chosen sothat the corresponding CNN implements a low-pass filter withfrequency response . By shifting such that itis centered around , we obtain the frequency response ofthe corresponding Gabor-type filter tuned to :This filter can be implemented by a CNN with the same feedforwardcloning template and the complex valued feedbackcloning templateUsing the approach in [30], it can be shown that if the originallow-pass filter template is stable, then the correspondingGabor-type template is also stable.Example 1: Consider the low-pass filter implemented bythe CNN with cloning templateswhere . Substituting into (2)(4)The frequency response of the spatial filter is(3)Assuming both and are real, the linear resistive grid[32] in Fig. 3(a) implements this CNN. At steady state,is a low-pass filtered version of . Fig. 3(b) and (c) plotsthe impulse and frequency responses of the filter which aregiven by [33]:A similar result using the discrete Fourier transform (DFT)holds for finite arrays with periodic boundary conditions wherethe cells at the ends of the array are considered to be nearestneighbors. With other boundary conditions, the analysis herewill be approximate, but for stable filters the effect of theboundary conditions decays as the distance from the boundaryincreases. By changing the values of the cloning templatecoefficients, different filters, e.g., low-pass, bandpass, andhigh-pass, can be constructed. Since the CNN consists of adiscrete array of cells, the filters are defined only for discretespaceinput.For every low-pass filter implementable on a CNN, theGabor-type filter obtained by multiplying the impulse responseof that low-pass filter by a complex exponential is also implementableon a CNN. Crounse and Chua describe methods forwhere. This system is a discreteapproximation to a continuous space system with impulse andfrequency responseswhere the subscript emphasizes that the function is definedon continuous space. The width of the impulse responseincreases linearly with . The approximation improves asdecreases.By (4), the corresponding Gabor filter [34] is implementedby the CNN with templates(5)(6)

126 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998(a)(b)(c)Fig. 3. (a) This resistive grid implements the low-pass filtering CNN of Example 1. The resistor labels denote conductances. The (b) impulse and (c)frequency responses of the filter for = 0:3 confirm the expected low-pass behavior.Fig. 4. A circuit implementation of two cells of a CNN array which implements the Gabor-type filter corresponding to the resistive grid discussed in Example1. Resistor labels denote conductances. Trapezoidal blocks represent transconductance amplifiers labeled by their gains.Its impulse and frequency responses, plotted in Fig. 1(d) and(e) arecomplex state variable has been replaced by two real valuedstate variablesFor a circuit implementation of this network, the complexvalued state is represented by the voltages across twocapacitors representing its real and imaginary parts and. Substituting (6) into (2) and separating the real andimaginary parts, we can express the time evolution of thecomplex valued state in (2) as an equivalent system where the(7)

SHI: GABOR-TYPE FILTERING IN SPACE AND TIME WITH CELLULAR NEURAL NETWORKS 127Fig. 5. This resistive network better approximating Gaussian filtering adds negative resistance connections to second nearest neighbors. Only the fullconnections for node n are shown. The resistor labels denote conductances.The circuit in Fig. 4 implements two cells of the CNN andtheir interconnections with their nearest neighbors. In thefollowing, we refer to each circuit node by its labeled nodalvoltage. The voltages across each pair of vertically alignedcapacitors represent the real and imaginary parts of the stateof a cell. The resistances and transconductance amplifiersimplement the interconnection between cells. The input to thenetwork is provided by current sources which supply currentsproportional to the input image intensity. By writing KCL atnodes and and assuming a unit capacitance, onecan verify that the time evolution of satisfies the topequation in (7) and that satisfies the bottom equation.The entire CNN array can be constructed by replicating thiscircuit to add more cells and connections. At steady state,the voltages across the lower capacitors are the result ofconvolving the spatial distribution of the input currents withthe real part of . The voltages across the upper capacitorscorrespond to convolving with the imaginary part. This mixedtransconductance amplifier and resistor implementation canbe proven to be more robust to parameter variations thanimplementations based on either resistors or transconductanceamplifiers alone [35].The circuit implementation also gives good intuitive understandingof the CNN’s operation. Assume that the inputimage is an impulse at pixel . In the circuit, this correspondsto setting the current source to amps and settingthe remaining current sources to zero. If the gains and conductanceswere chosen so that and , thenthe steady-state voltages across the lower capacitors wouldfollow the spatial distribution shown in Fig. 1(d) where thecenter peak occurs at cell and the voltages across the uppercapacitors would follow the distribution shown in Fig. 1(e).To see how this would arise in the circuit, consider thecurrent supplied by the source . Part of the currentflows through the conductance (which is positive) pushingthe voltage positive. As this voltage increases, the tworesistors with conductancecause a smoothingeffect which pulls the voltages and uptoward . Current also flows through the diagonal resistorwith conductance pulling positiveas well. At the same time, the transconductance amplifierwith input draws current from node pushingnegative. The larger , the more the voltages atnodes and are pushed negative and positive.On the other hand, the larger , the greater the smoothingbetween nodes. Thus, the larger the ratiothe higher the spatial frequency at which the impulseresponse oscillates. This is consistent with our theoreticalpredictions since tan increases with .Example 2: Based upon Kobayashi et al.’s resistor networkfor Gaussian filtering [36], it is possible to design a CNNwhich implements a filter with an impulse response which iscloser to the Gabor, albeit at the price of increased complexity.For 1-D images, the resistor network in Fig. 5 smooths theinput current distribution with an approximately Gaussianconvolution kernel. The corresponding CNN cloning templatesareThe filter implemented by this network is a discrete approximationto a continuous space filter with impulse and frequencyresponses given by [37]The width of the impulse response increases linearly with .The approximation improves as decreases.The template for the corresponding Gabor-type filter isFig. 1(g)–(i) plots the impulse and frequency responses of thisfilter.To compare how well the filters in the preceding examplesapproximate Gabor filters, we define the normalized squarederror (NSE) to be the total energy in the difference betweenthe Gabor impulse response and the impulse response of thecontinuous space filter approximated by the CNN filter dividedby the total energy in the Gabor impulse responsewhere is the Gabor impulse response in (1) and isgiven either by (5) or (8). The frequencies of the complexexponentials in the Gabor and Gabor-type filters are(8)

128 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998chosen to be identical. The results of analyzing the discretespacefilters will be more complex, but very similar to theresults here, especially as and approach zero.Using the fact that, it can be shown thatthe error is independent of and depends only upon thedifference between the Gaussian envelope of the Gabor filterand the low-pass envelope of the Gabor-type filter. Forthe filter in Example 1, the minimum error of 15 dB isachieved forTABLE ICOMPARISON OF PHASE MEASUREMENT PROPERTIESOF GABOR AND GABOR-TYPE FILTERSFor the filter in Example 2, the minimum error ofachieved for21 dB isThe values of and have been estimated numerically.Simulation results in Fig. 1 where the parameters chosenminimize the NSE provide visual confirmation that the impulseand frequency responses of the filter in Example 2 are closerto the Gabor filter’s than are those of the filter in Example 1.To compare the three filters with respect to the desirableproperties for computing phase discussed in Section I-B, wedefine the following criteria. The criteria are based upon theassumption that the input image is white.Negative Frequency Rejection (NFR): This is the ratio betweenthe energy in the output due to positive frequencies inthe input and the energy due to negative frequencies. Ifis the frequency response of the filter, thenDC Rejection (DCR): This is a measure of the extent towhich the dc component is suppressed compared with the peakresponse. If is the frequency response of the filterPhase Wraparound Rejection (PWR): The output at a givenpixel can be considered as the sum of two components: thefirst (desired) component due to convolving the image with thepart of the impulse response where the phase varies fromto and the second (undesired) component due to convolvingthe input image with the part of the impulse response wherethe phase has wrapped around. Since we assume the imageinput is white, these two components are uncorrelated. Wedefine the PWR to be the ratio of their energies. If is theimpulse response of the filter, thenwell as whether or not the phase of the impulse responseis monotonous. The results were estimated numerically usingthe same filter parameters used to generate the plots. Sincethe parameters were chosen to minimize the energy in thedifference between the filter impulse responses, their spatialsupport is similar. Since the PWR is defined in terms of theenergy in the filters, their PWR is also similar. However, thereare large differences between the NFR and DCR. As can beseen from Table I as well as the plots in Fig. 1, the NFR andDCR is largest for the Gabor filter and smallest for the filterin Example 1. The rejection can be improved at the cost ofdecreased PWR by decreasing the bandwidth of the filters.Only the filter in Example 2 does not have monotonous phase,since its low-pass modulating function is not strictly positive.Although the degree to which the filters exhibit the propertiesdescribed in Section I-B varies, we would expect any of thefilters to be effective in phase-based algorithms because thesealgorithms have been found to be fairly insensitive to the exactform of the filter.B. 2-D ImagesThe 2-D generalization of (2) simply extends the summationto two dimensions:The feedback cloning templatebe represented by acan nowmatrix, e.g.,If the CNN filter is stable, its frequency response isAs in the 1-D case, for any 2-D CNN low-pass filter therecorresponds a 2-D CNN Gabor filter tuned to frequencyobtained by replacing the feedback cloning templatewithTable I compares the Gabor filter and the Gabor-type filtersin Examples 1 and 2 with respect to these three criteria asIts impulse response is the impulse response of the low-passfilter modulated by. Its frequency response is.

SHI: GABOR-TYPE FILTERING IN SPACE AND TIME WITH CELLULAR NEURAL NETWORKS 129(a)(b)Fig. 6. (a) The 2-D version of the resistive grid connects each node in the array resistively to its top, bottom, left, and right nearest neighbors. (b) For! yo = 0, the corresponding network for Gabor-type filtering consists of rows of nodes interconnected horizontally, as in Fig. 4, and coupled verticallyby resistors. To avoid clutter, we have omitted the capacitor at each node.Example 3: The 2-D extension of the resistive grid ofExample 1 is shown in Fig. 6(a). The CNN cloning templateswhich implement the 2-D grid areThe corresponding Gabor-type filter tuned tocloning templateshaswhich is approximately circularly symmetric around. The shape of the passband can be stretched in thedirections perpendicular to the and axes by scaling thevalues of the horizontal and vertical connections. Althoughthey are restricted to nearest neighbors, the additionalconnections can be quite complex. However, if , thearray reduces to a set of 1-D filters which are resistivelycoupled to the rows above and below [see Fig. 6(b)]. Withthese simpler connections, arbitrary orientations could beobtained by rotating the cell array.and frequency responseIII. SPATIO–TEMPORAL FILTERSThe previous section detailed the construction of CNNspatial filters tuned to arbitrary spatial frequencies .Cascading a CNN spatial filter tuned towith atemporal filter tuned to results in a spatio–temporal filtertuned to. Since the output of the spatial filter iscomplex, we must distinguish between positive and negativespatio–temporal frequencies. A temporal filter tuned to

130 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998results in a spatio–temporal filter tuned to velocities with thesame magnitude in the opposite direction.CNN spatio–temporal Gabor-type filters constructed in thisway are separable in space and time. 3-D spatio–temporalGabor filters are also space–time separable if their covariancematrices have the formAn important difference between Gabor filters and the CNNfilters is that Gabor filters are noncausal in space and time,while the CNN filters are noncausal in space, but causal intime.In order to ensure space–time separability, the spatial filteringCNN’s must settle much faster than the time scales of theimage motion so that their outputs at any time are essentiallythe result of spatial filtering the input at time . Fortunately, itis easy to design a VLSI implementation of the CNN spatialfilters with settling times on the order of microseconds orfaster. The time scale of image motion is usually on the orderof milliseconds. For example, video frame rates are about oneframe every 30–40 ms.There are several advantages to this spatio–temporal filteringarchitecture. First, multiple filters tuned to different velocitiescan be obtained by cascading the same spatio–temporalfiltering stage with different temporal filters. Since all of thefilters share the same spatial filtering stage, the differencesbetween their outputs are purely a function of the temporalvariation of the image, i.e., the motion. Second, the temporalfiltering at each pixel is independent of the filtering at the restof the pixels. Since the output of the spatial frequency stageis bandlimited, it can be subsampled without distortion due toaliasing. Therefore, it is not necessary to build a temporal filterat every pixel of the output. This can result in a significantsaving in chip area.Example 4: Denote the output of the spatial filtering stageat a pixel and time by . One possible continuoustime temporal filter [38] tuned to has outputwhich satisfiesFig. 7.(a)(b)(a) Contour plot and (b) mesh plot of the magnitude of the CNNspatio–temporal filter’s frequency response for 1-D images. The parametersused were = = 0:3; ! xo =2atan0:5, and ! to =0:3:This filter istuned to velocities v = 00:32.impulse response is the product of the spatial and temporalimpulse responseswhere. It has frequency responseThe combined spatio–temporal frequency response is the productof the spatial and temporal frequency responses. Using thespatial filtering stage of Example 1, the frequency response iswhose magnitude is plotted in Fig. 7. Since the correspondingimpulse response is complex valued, the frequency responseis not symmetric with respect to the origin. Due to spatialsampling, the spatio–temporal frequency response is periodicin with period . There is no periodicity in sincethe temporal filter operates in continuous time. Similarly, theIntuitive insight into the operation of the spatio–temporal filtercan be gained by considering the effect of the temporal filteron one pixel in isolation since the temporal filtering stageprocesses each output pixel of the spatial filter independently.Assuming uniform translation, the output of the spatial filteringstage at pixel and time can be written aswhere is constant and varies slowly in and .The output of the spatial filter at a fixed pixel rotates aroundthe origin of the complex plane with frequency . Thespeed of rotation is proportional to the speed of translation. Thedirection of rotation depends on the direction of translation.Assuming , the direction of rotation is clockwisefor positive velocities and counterclockwise for negative (seeFig. 8).

SHI: GABOR-TYPE FILTERING IN SPACE AND TIME WITH CELLULAR NEURAL NETWORKS 131Fig. 8.The output of the spatial filtering stage at a given pixel rotates around the origin of the complex plane. This figure shows the outputs at pixel n of thespatial filtering stage in Example 1 with = 0:3 and ! xo =0:93 for image inputs consisting of impulses located at different pixels near n. If the impulseis translating uniformly, the output rotates around the origin clockwise for positive velocities and counterclockwise for negative velocities.Fig. 10. A circuit implementation of the second-order differential (9) usingthree operational amplifiers. Resistors are labeled by their conductances.and imaginary parts of the temporal filter output. These satisfythe following real valued differential equation:(9)Fig. 9. The derivative of along trajectories of the unforced temporal filterfor = 00:7 and ! to =1illustrates that the system is a damped linearoscillator. Trajectories naturally tend to rotate counterclockwise about theorigin.The differential equation satisfied by the temporal filtercorresponds to a damped linear oscillator forced by the outputof the spatial filter. The derivative along different trajectoriesof the unforced system are shown in Fig. 9 for .Ifthe velocity is negative, the input rotates counterclockwise,facilitating the natural motion of output trajectories and leadingto a large response. For positive velocity, the input rotationopposes the natural motion, leading to a small response.In an analog circuit, the filter can be implemented with twocapacitors whose voltages represent the real and imaginaryparts of the output. Let and denote the realWriting KCL at the inverting inputs of the first and thirdoperational amplifiers and assuming unit capacitance, it canreadily be shown that the circuit shown in Fig. 10 implementsthis differential equation. To cascade this temporal filter withthe spatial filter in Example 1, the inputs andshould be equal to the voltages and from thebottom and top rows of Fig. 4.IV. CONCLUSIONThis paper has defined a class of Gabor-type spatial andspatio–temporal filters which posses the important propertiesof filters used in phase-based algorithms for computer vision.Gabor-type filters can be implemented using CNN’s. It ispossible to convert any low-pass spatial filtering CNN into acorresponding Gabor-type filtering CNN by a simple transformationof the cloning template coefficients. Spatio–temporal

132 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 45, NO. 2, FEBRUARY 1998filters tuned to different velocities can be obtained by cascadingthe spatial filter outputs with appropriately designedtemporal bandpass filters.Ongoing work includes the design and fabrication of thearchitecture described in Example 1 using the 2.0- m processprovided by MOSIS [39]. Future work includes incorporatingthis into a chip implementing a spatio–temporal filter,as described in Example 4. We are also investigating theapplication of these chips to various computer-vision taskssuch as binocular stereo vergence control and computation oftime-to-contact [26].REFERENCES[1] D. Gabor, “Theory of communication,” J. Inst. Elect. Eng. London, vol.93, pp. 429–457, 1946.[2] S. Marceljia, “Mathematical description of the responses of simplecortical cells,” J. Opt. Soc. Amer., vol. 70, pp. 1297–1300, Nov. 1980.[3] J. G. Daugman, “Two-dimensional spectral analysis of cortical receptivefield profiles,” Vision Res., vol. 20, pp. 847–856, 1980.[4] E. H. Adelson, and J. R. Bergen, “Spatiotemporal energy models forthe perception of motion,” J. Opt. Soc. Amer. A, Opt. Image Sci., vol.2, pp. 284–299, Feb. 1985.[5] J. Barron, D. S. Fleet, S. S. Beauchemin, and T. A. Burkitt, “Performanceof optical flow techniques,” in Proc. IEEE CVPR, Champaign,IL, 1992, pp. 236–242.[6] L. O. Chua and L. Yang, “Cellular neural networks: Theory,” IEEETrans. Circuits Syst., vol. 35, pp. 1257–1272, Oct. 1988.[7] , “Cellular neural networks: Applications,” IEEE Trans. CircuitsSyst., vol. 35, pp. 1273–1290, Oct. 1988.[8] L. O. Chua and T. Roska, “The CNN paradigm,” IEEE Trans. CircuitsSyst. I, vol. 40, pp. 147–156, Mar. 1993.[9] J. M. Cruz and L. O. Chua, “A CNN chip for connected componentdetection,” IEEE Trans. Circuits Syst., vol. 38, pp. 812–817, July 1991.[10] S. Espejo, R. Carmona, R. Dominguez-Castro, and A. Rodriguez-Vazquez, “A CNN universal chip in CMOS technology,” Int. J. CircuitTheory Applicat., vol. 24, pp. 93–109, 1996.[11] P. Kinget and M. Steyaert, “An analog parallel array processor for realtimesensor signal processing,” in IEEE Int. Solid-State Circuits Conf.Dig. Tech. Papers, San Francisco, CA, 1996, pp. 92–93.[12] T. Sanger, “Stereo disparity computation using Gabor filters,” Biol.Cybern., vol. 59, no. 6, pp. 405–418, 1988.[13] R. Wilson and H. Knutsson, “A multiresolution stereopsis algorithmbased on the Gabor representation,” in 3rd Int. Conf. Image ProcessingApplicat., Warwick, U.K., July 1989, pp. 19–22.[14] A. D. Jepson and M. Jenkin, “The fast computation of disparity fromphase differences,” in Computer Vision-ECCV90, O. Faugeras, Ed.Berlin, Germany: Springer-Verlag, 1990, pp. 50–55.[15] K. Langley, T. J. Atherton, R. G. Wilson, and M. H. E. Larcombe,“Vertical and horizontal disparities from phase,” in Computer Vision-ECCV90, O. Faugeras, Ed. Berlin, Germany: Springer-Verlag, 1990,pp. 315–325.[16] D. J. Fleet, A. D. Jepson, and M. R. M. Jenkin, “Phase-based disparitymeasurement,” CVGIP Image Understanding, vol. 53, no. 2, pp.198–210, Mar. 1991.[17] C. Chang and S. Chatterjee, “Ranging through Gabor logons—A consistent,hierarchical approach,” IEEE Trans. Neural Networks, vol. 4,pp. 827–43, Sept. 1993.[18] C.-J. Westelius, H. Knutsson, J. Wiklund, and C.-F. Westin, “Phasebaseddisparity estimation,” in Vision as Process, J. L. Crowley and H.I. Christensen, Eds. Berlin, Germany: Springer-Verlag, 1995, Ch. 11,pp. 157–178.[19] W. M. Theimer and H. A Mallot, “Phase-based binocular vergencecontrol and depth reconstruction using active vision,” CVGIP: ImageUnderstanding, vol. 60, no. 3, pp. 343–358, Nov. 1994.[20] M. Porat and Y. Y. Zeevi, “Localized texture processing in vision:Analysis and synthesis in Gaborian space,” IEEE Trans. Biomed. Eng.,vol. 36, pp. 115–29, Jan. 1989.[21] N. Petkov, P. Kruizinga, and T. Lourens, “Biologically motivatedapproach to face recognition,” in New Trends Neural Computation, Int.Workshop Artificial Neural Networks, Seville, Spain, 1993, pp. 68–77.[22] D. J. Heeger, “Model for the extraction of image flow,” J. Opt. Soc.Amer. A, vol. 4, pp. 1455–1471, Aug. 1987.[23] N. M. Gryzwacz and A. L. Yuille, “A model for the estimate of localimage velocity by cells in the visual cortex,” in Proc. R. Soc. Lond. B,vol. 239, 1990, pp. 129–161.[24] D. J. Fleet, Measurement of Image Velocity. Norwell, MA: Kluwer,1992.[25] B. K. P. Horn, Robot Vision. Cambridge, MA: MIT Press, 1986.[26] B. E. Shi, “Second-order CNN arrays for estimation of time to contact,”in Proc. 4th IEEE Int. Workshop Cellular Neural Networks, Seville,Spain, June 1996, pp. 427–432.[27] R. N. Bracewell, The Fourier Transform and Its Applications. NewYork: McGraw-Hill, 1986.[28] L. Raffo, “Resistive network implementing maps of Gabor functions ofany phase,” Electron. Lett., vol. 31, no. 22, pp. 1913–1914, Oct. 1995.[29] T. Delbruck, “Silicon retina with correlation-based, velocity-tuned pixels,”IEEE Trans. Neural Networks, vol. 4, pp. 529–541, May 1993.[30] B. E. Shi, “Spatio–temporal filtering with cellular neural networks,”Ph.D. dissertation, Dept. Elect. Eng. Computer Sci., Univ. CaliforniaBerkeley, 1994.[31] K. R. Crounse and L. O. Chua, “Methods for image processing andpattern formation in cellular neural networks: A tutorial,” IEEE Trans.Circuits Syst. I, vol. 42, pp. 583–601, Oct. 1995.[32] C. Mead, Analog VLSI and Neural Systems. Reading, MA: Addison-Wesley, 1989.[33] B. E. Shi and L. O. Chua, “Resistive grid image filtering: input/outputanalysis via the CNN framework,” IEEE Trans. Circuits Syst. I, vol. 39,pp. 531–548, July 1992.[34] B. Shi, “Gabor-type image filtering with cellular neural networks,” Proc.IEEE Int. Symp. Circuits Syst., vol. 3, pp. 558–561, May 1996.[35] K. F. Hui and B. E. Shi, “Robustness of CNN implementations forGabor-type filtering,” in IEEE Asia–Pacific Conf. Circuits Syst., Seoul,Korea, Nov. 1996, pp. 105–108.[36] H. Kobayashi, J. L. White, and A. A. Abidi, “An active resistor networkfor Gaussian filtering of images,” IEEE J. Solid-State Circuits, vol. 26,pp. 738–748, May 1991.[37] T. Poggio, H. Voorhees, and A. Yuille, “A regularized solution to edgedetection,” MIT, Cambridge, A.I. Memo 833, May 1985.[38] B. E. Shi, “Spatio–temporal image filtering with cellular neural networks,”Proc. IEEE Int. Conf. Neural Networks, vol. 3, Washington,DC, June 1996, pp. 141–1415.[39] , “An analog neural network for Gabor-type image filtering,” inProc. Int. Conf. Neural Inform. Processing, vol. 1, Hong Kong, Sept.1996, pp. 417–422.Bertram E. Shi (S’93–M’95) received the B.S.and M.S. degrees in electrical engineering fromStanford University, Stanford, CA, in 1987 and1988, respectively, and the Ph.D. degree in electricalengineering from the University of California atBerkeley, in 1994.Since 1994, he has been an Assistant Professorin the Department of Electrical and Electronic Engineering,Hong Kong University of Science andTechnology, Kowloon, Hong Kong. His researchinterests are in CNN’s, image processing, computervision, and speech recognition.