Intersection Safety Using Lidar and Stereo Vision Sensors - (DDS ...

ATTACHMENTS AND SPARES FORMULTISPINDLE AUTOMATIC SCREW MACHINESmachine: TORNOSmodel: AS14 - SAS16catalog :tornos_2009_3

each segment using Cartesian coordinates of its constitutingpoints. This information will be used to perform a polarfusion between lidar and stereo vision.D. Lidar Data Processing OutputThe output of lidar data processing consists of the localgrid map and list of detected moving objects (we do notinclude the static objects in this list). Each object in this listis represented by its centroid and set of points correspondingto the laser hit points. Grid map is only used to display onthe screen whereas list of dynamic objects is used furtherfor fusion. An example of laser processing result is shownin the Figure 3.Fig. 5.Output of classification: the predicted class.A. IntroductionV. STEREO VISION PROCESSINGThe main roles of the stereo-vision sensor in an intersectiondriving assistance system are related to the sensingand perception in the front of the ego vehicle in a regionup to 35m in depth and a 70 ◦ horizontal field of view.The current field of view was established as an optimumcompromise between the maximum reliable depth range andthe field of view. Static road and intersection environmentperception functions are: Lane markings detection and 3Dlocalization; Curb detection and 3D localization; Current andside lanes 3D model estimation based on lane delimiters(lane markings, curbs); Stop line, pedestrian and bicyclecrossing detection and 3D localization; Painted signs (turnright, turn left, and go ahead) detection and 3D localization;Static obstacle detection, 3D localization and classificationincluding parked vehicles, poles and trees. Dynamic road andintersection environment perception functions are: Preceding,oncoming and crossing vehicles detection, tracking and classification;Preceding, oncoming and crossing vulnerable roadusers detection, tracking and classification.B. Stereo sensor architecture for intersection assistanceBased on the requirements analysis a two level architectureof a 6D stereo sensor was proposed [5] (Figure 4 and Figure??). The low level architecture controls the image acquisitionprocess and provides, after the sensor data processing, theprimary information needed by the high level processingmodules: 6D point information (3D position and 3D motion),ego motion estimation and intensity images at a rate of 20frames per second. Using the rich output of the low-levelarchitecture the two environment descriptions (structured andunstructured) are generated.C. Obstacle detection1) 3D Points pre-processing: Using information from thedigital elevation map the 3D points are classified accordingto their position with regard to the detected road/navigablearea plane. Optical flow provides motion information for asubset of these points (corner points)2) Obstacle detection: An improved obstacle detectiontechnique was developed based on the fusion 3D positioninformation with 3D motion information [7]. The obstacledetection algorithm extends the existing polar occupancygrid-based approach by augmenting it with motion information.The benefits gained from the integration of motioninformation are threefold. First, by use on at grid cell level,object boundaries are more accurately detected. Secondly, byexploiting motion at obstacle level, the obstacle’s orientationis more accurately and naturally determined. And finally,each obstacle carries speed information, a valuable cuefor tracking and classification. For non-stationary obstacles,motion can provide additional cues for orientation computation.Theoccupied areas are fragmented into obstacleswith cuboidal shape without concavities and only with 90oconvexities. There are 2 types of objects: 3D Non-Orientedand 3D Oriented. The Obstacles are represented as bothoriented and non-oriented 3D boxes (cuboids) circumscribingthe real obstacles in the scene. The Non-Oriented Obstaclesare described by the minimum and maximum X, Y and Zcoordinates in the ego vehicle reference frame. The OrientedObstacles are characterized by the X, Z coordinates of thecorners and the minimum and maximum Y coordinate.3) Relevant obstacles classification: The goal of obstacleclassification is to recognize the relevant objects in an intersection.We have identified three classes of relevant objects:Pedestrian, Pole, Car. A generic classification system able torecognize in real-time each of the three classes of objectshas been developed [4] (Figure 5 ).4) Obstacle representation: The obstacle are representedas cuboids carrying the following information:• the cuboid’s position, orientation and size,• lateral and longitudinal speed,• variance of the object center coordinates orientation andspeed,• tracking history (number of frames in which this objectwas previously seen),• The detected obstacles are classified into: pedestrians,cars, poles and unknown.To perform fusion between data from lidar and stereovision objects, we project the objects detected by stereovision processing onto the laser plane.865

Fig. 4.Stereo-vision data processing: left) low-level architecture; right) high-level architectureThe output of stereo vision system consists of a list ofobjects with 2D position and classification information foreach object. In the next step, we calculate the centroid of allthe objects: it is the middle point of the front line segmentof the object rectangle obtained after projection (this pointgives better results than the center of gravity of this rectanglebecause laser readings also belong to the front end of theobject). In the final step, we calculate the range and bearingof this centroid for each object from the origin of lidar frameof reference. So the object is represented as a point withrange and bearing along with classification properties.VI. LASER AND STEREO VISION DATA FUSIONInputs to the fusion process are two lists of objects: list ofdynamic objects detected by lidar processing and representedas centroid points, and list of dynamic objects detected bystereo vision processing represented as points along withclassification information. We believe that an object detectionlevel fusion between these two lists can complement eachother thus giving more complete information about the statesof objects in the environment. This fusion process consistsof following two steps:A. Object associationIn this step we determine which stereo objects are to beassociated to which lidar objects from the two object lists,using nearest neighbor technique. We have defined a distancethreshold function based on the depth of the stereo objectfrom the origin. Using this threshold value given by thisfunction we associate the current stereo object to the nearestlaser object lying within this threshold distance from thestereo object. We use this distance threshold function insteadof a hardcoded value because depth uncertainty of stereoobjects increases with the distance from the origin.B. Position information fusionThis step works on the pair of objects associated with eachother in the previous step and fuses their position (rangeand bearing) information. We model the position uncertaintyusing 2D guassian distribution for both objects. SupposeP L = [r L , θ L ] T is the centroid position of laser object andP V = [r V , θ V ] T is the centroid position of associated stereovision object. If X is the true position of the object then theprobability that laser detects this object at point P L is givenas:P (P L |X) = e −(P L −X)T R −1L(P L −X)2√|RL | 8 2πand similar probability for stereo object is given as:P (P V |X) = e −(P V −X)T R −1V2√|RV | 8 2π(P V −X)Here R L is the 2X2 covariance matrix of range and bearinguncertainty calculated from the uncertainty values providedby the vendor. Whereas R V is the covariance matrix forstereo vision and depends on the depth of the object fromorigin In general range and bearing uncertainty for stereoobjects is much higher than the corresponding objects detectedby laser and increases with distance from the origin.Also, range uncertainty for stereo is greater than the bearinguncertainty in general. Using Bayesian fusion the probabilityof fused position P is given as:P (P |X) = e −(P −X)T R −1 (P −X)2√|R|8 2πwhere P and R are given as:andP = P L/R L + P V /R V1/R L + 1/R V1/R = 1/R L + 1/R Vrespectively.The result of this fusion process is a new list of fusedobjects. This list also has all the laser objects which couldnot be associated with stereo objects and all the stereo866

objects which could not be associated with some laserobjects. We keep unassociated stereo objects because theymay correspond to dynamic objects which may not have beendetected by laser either because they may are occluded orthey may are transparent for laser.C. Fusion OutputThe output of fusion process consists of fused list ofobjects. For each object we have position (centroid) information,dynamic state information, classification informationand a count for number of sensors detecting this object. Foreach fused object we also have a pointer to the originallaser or stereo object to use segment or rectangle informationwhile displaying the tracked object.VII. TRACKINGIn general, the multi objects tracking problem is complex:it includes the definition of tracking methods, but also associationmethods and maintenance of the list of objects currentlypresent in the environment. Bayesian filters are usually usedto solve tracking problem. These filters require the definitionof a specific motion model of tracked objects to predicttheir positions in the environment. Using the predictionand observation update combination, position estimation foreach object is computed. In the following we explain thecomponents of our tracking module.A. Data AssociationThis step consists of assigning new objects of fusedlist to the existing tracks. Since in the current work weare more concerned with tracking multiple objects in anintersection like scenario so it is important to choose a moreeffective technique of data association. In an intersectionscenario there may be many objects moving in differentdirections. They may be crossing or wating to cross ina direction perpendicular to the oncoming vehicles, forexample a vehicle waiting to turn left etc. We have usedMHT [8] approach to solve the data association problem.An important optimization that we have achieved here dueto fusion process mentioned above is related to classificationinformation provided by stereo vision. While generatinghypotheses we ignore all those hypotheses which involveobjects from different classes. For example a hypothesistrying to involve a pedestrain with a vehicle in a track will beignored, this significantly reduces the number of hypotheses.To further control the growth of tracks trees we need touse some pruning technique. We have chosen the N-Scanspruning technique to keep the track trees to a limit of N.B. Track ManagementIn this step tracks are confirmed, deleted or created usingthe m-best hypotheses resulting from the data associationstep. New tracks are created if a new track creation hypothesisappears in the m-best hypothesis. A newly created trackis confirmed if it is updated by objects detected in currentframes after a variable number of algorithm steps (one stepif the object was detected by both laser and stereo visionotherwise in three steps). This implies that the spuriousmeasurements which can be detected as objects in the firststep of our method are never confirmed. To deal with nondetectioncases, if a non-detection hypothesis appear (whichcan appear for instance when an object is occluded byan other one) tracks having no new associated objects areupdated according to their last associated objects and forthem next filtering stage becomes a simple prediction. Inthis way a track is deleted if it is not updated by a detectedobject for a given number of steps.C. FilteringSince in an intersection like scenario there may be differenttypes of objects (vehicles, motor bikes, pedestrainsetc) moving in different directions using different motionmodes, a single motion model based filtering technique is notsufficient. To address the tracking problem in this scenariowe have used an on-line adapting version of InteractingMultiple Models (IMM) filtering technique. The details ofthis technique can be found in our other published work [9].We have seen that four motion models (constant velocity,constant acceleration, left turn and right turn) are sufficientto successfully track objects on an intersection. We use fourKalman filters to handel these motion models. Finally themost probable trajectories are copmuted by taking the mostprobable branche and we select one unique hypothesis forone track tree.D. Tracking OutputThe output of tracking process consists of position andvelocity information of ego vehicle alongwith a list of tracks.A track is a moving object with its position, orientation,velocity and classification information as well as a referenceto its instance in the previous frame.VIII. RISK ASSESSMENTThe risk assessment module provides an evaluation of therisk of a potential collision between the host vehicle andthe objects that may be present in the driving environment.Our approach follows the work previously presented in [1].This evaluation consists in the prediction of the environmentfor the future time instants and quantification of the riskassociated to the detected collision situations: potential futurecollisions. It is considered that the driver has full controlof the host vehicle and that the future driver behavior isunknown. The risk assessment is performed in the followingsequential steps:• scenario interpretation• trajectory prediction• collision detection• risk quantification• risk managementA. Scenario interpretationThe scenario interpretation consists of a representationof the current host vehicle state and local map of thesurrounding environment composed of dynamic and static867

objects. This interpretation, that in most cases is incompleteand not very accurate, will influence the performance of therisk assessment. The host vehicle state provides informationabout the position, heading, steering angle, velocity, acceleration,yaw rate of the host vehicle. The dynamic objectscan be of 2 types: vehicles and pedestrians. The informationabout the static and dynamic objects is limited and it leads tosome assumptions that will influence the trajectory predictionprocess:• The objects of type vehicle keep current speed anddirection: no information about steering angle, acceleration,yaw rate or blinkers is provided by the highlevel fusion or communications.• The host vehicle and other dynamic objects trajectoriesare not constraint to follow the road: there is no informationabout the static objects like the road geometryand lanes description.B. Trajectory predictionGiven the current scenario interpretation the host vehiclestate and dynamic objects are modeled and integrated intime to provide a projection of the future environmentrepresentation. This integration time consists in predictingthe trajectories of the dynamic objects, including host vehicle,just by using the current scenario interpretation. Thefuture driver behavior is unknown and will not be predictedalthough it may affect the future trajectories. A trajectory ofa dynamic object is a temporal sequence of object statesfor the future time instants. For each object a trajectorypredicted from the current time t0 until a given time horizont0+h where h is the total prediction time. The modelingof the trajectories is done taking into account the objecttype, vehicle or pedestrian and associated information. Theprediction of the vehicles trajectories, including the one ofthe host vehicle, is performed by using a bicycle dynamicmodel [3] integrated in time using the 4th order Runge-Kuttamethod that uses acceleration and steering rate commandsas inputs. The initial vehicle state used to integrate theprevious model is the one obtained at the time t0. Predictthe movement of pedestrians is a substantially more difficulttask [6] than vehicles. Since a pedestrian can easily changedirection no assumptions are made regarding the directionof its movement. The pedestrian is then modelled as a circlewith a predefined radius centred at the initially detectedpedestrian position at time t0, that will increase its radiusin time proportionally to its initially estimated speed.C. Collision detectionThe host vehicle and dynamic objects are represented ascircles: the circle centre is given by the object position, andthe circle radius is set according with the object type, at agiven moment in time. The position uncertainty of the objectsis represented by an increase of the circles radius in functionof the estimated travelled distance by the object. A potentialcollision is detected when the host vehicle circle intersects atleast one circle of the dynamic objects at the same momentin time. Figure 6 gives an illustration of this process.Fig. 6. Example of a potential collision between the host vehicle (redcircle on the bottom) and another vehicle (green circle on the bottom right).Fig. 7.D. Risk quantificationRelation between TTC and risk indicator.The risk of collision is calculated for the nearest potentialcollision situation in time. For calculating this risk it is usedthe parameter time-to-collision (TTC) that corresponds to theduration between the current time t0 and the instant of whenthe first detected collision will occur. A consideration is takenthat all objects keep their initial speeds until the moment ofthe collision. The TTC is an important parameter becauseit can be compared to the driver and vehicle reaction timesand provide a collision risk indicator. In our implementationthe obtained TTC is compared with the commonly used totalreaction time of 2 seconds: driver (1 sec) [10] and vehicle (1sec). The risk estimation performed until a predefined timehorizon t0+h and the risk indicator is given by the relationshown in figure 7.E. Risk managementBased on quantification of the collision risk two strategiescan be adopted to avoid or minimize the potential accident:Information or warning: advices are provided to the driverthrough the appropriate HMI visual, audio, or haptic feedbackto avoid or reduce the risk of accident. Intervention:the automation takes momentarily control of the vehicleto perform an obstacle avoidance or collision mitigationmanoeuvre. In our implementation only visual informationis provided to the driver with periodical estimation of thecollision risk for the given scenario interpretation.868

Fig. 8.Tracking results for a pedestrian and a cyclist.Fig. 9.Tracking results for two cars.IX. RESULTSExamples of tracking results are shown in Figures 8 and 9along with the images of corresponding scenarios. In Figure8 is a situation at the intersection where the ego vehicleis waiting for the traffic signal, a cyclist and a pedestriancrossing the road in opposite directions are being tracked. Inaddition, a truck which is partially occluded by the cyclistis also well tracked. Figure 9 shows two cars crossing theintersection which are detected and tracked successfully.X. CONCLUSIONIn this paper, we describe our approach for the safety ofvehicles at the intersection developed on the Volkswagendemonstrator. A complete solution to this safety problemincluding the tasks of environment perception and risk assessmentare presented along with interesting results whichcould open potential applications for the automotive industry.XI. ACKNOWLEDGEMENTSThis work was conducted within the research projectINTERSAFE-2 that is part of the 7th Framework Programme,funded by the European Commission. The partnersof INTERSAFE-2 thank the European Commission for allthe support.REFERENCES[1] Samer Ammoun and Fawzi Nashashibi. Real time trajectory predictionfor collsion risk estimation between vehicles. In IEEE InternationalConference on Intelligent Computer Communication and Processing,2009.[2] Q. Baig, TD. Vu, and O. Aycard. Online localization and mappingwith moving objects detection in dynamic outdoor environments. InIEEE Intelligent Computer Communication and Processing (ICCP),Cluj-Napoca, Romania, August 2009.[3] T.D. Gillespie. Fundamentals of Vehicle Dynamics, Society of AutomotiveEngineers. 1992.[4] S. Nedevschi, S. Bota, and C. Tomiuc. Stereo-based pedestriandetection for collision-avoidance applications,. IEEE Transactions onInt, 10:380–391, 2009.[5] S. Nedevschi, T. Marita, R. Danescu, F. Oniga, S. Bota, I. Haller,C. Pantilie, M. Drulea, and C. Golban. On Board 6D Visual Sensorsfor Intersection Driving Assistance Systems. Advanced Microsystemsfor Automotive Applications, Springer, 2010.[6] G. De Nicolao, A. Ferrara, and L. Giacomini. Onboard sensorbasedcollision risk assessment to improve pedestrians safety. IEEETransactions on Vehicular Technology, 56(5):2405–2413, 2007.[7] C. Pantilie and S. Nedevschi. Real-time obstacle detection in complexscenarios using dense stereo vision and optical flow. In IEEEIntelligent Transportation Systems, pages 439–444, Madeira, Portugal,September 2010.[8] D. B. Reid. A multiple hypothesis filter for tracking multiple targetsin a cluttered environment. Technical Report D-560254, LockheedMissiles and Space Company Report, 1977.[9] TD. Vu, J. Burlet, and O. Aycard. Grid-based localization and localmapping with moving objects detection and tracking. InternationalJournal on Information Fusion, Elsevier, 2009. To appear.[10] Y. Zhang, E. K. Antonsson, and K. Grote. A new threat assessmentmeasure for collision avoidance systems. In IEEE InternationalIntelligent Transportation Systems Conference, 2006.869