DaimlerChrysler Computer-Assisted Situation Analysis

Press Release

Photo Caption:
Interplay between two sensors: A radar sensor recognizes the vehicle in front and marks it with a red cross. A video camera serves as the lane-recognition system, with the detected side and center lines marked in green.

Seeing More…Understanding Better

Bringing the vision of accident-free driving a step closer to reality will require electronic assistance systems capable of keeping a close eye on the traffic situation. What's more, these systems will also have to be able to detect an imminent hazard. To meet these requirements, researchers at DaimlerChrysler are adopting new approaches to sensor utilization and the interpretation of data.

Sensor fusion and situation analysis systems are two basic technologies that we will need to master if we are to gradually turn the vision of accident-free driving into reality. The associated issues and tasks are very complex, both from a scientific point of view and from the perspective of how such systems can be created for vehicles," says Gabi Breuel from the Active Safety and Driver Assistance Systems Laboratory at DaimlerChrysler Research. However, immersing oneself in this complex field is well worth the effort. It's amazing how sensors can help researchers not only analyze current traffic conditions but also predict potential hazards.

"Using a variety of sensors, assistance systems will eventually be able to 'understand' a vehicle's entire surroundings," says Hans-Georg Metzler, Breuel's boss and the head of the lab. "This means they will be able to interpret the current driving situation far more reliably than was previously possible. And that, in turn, will greatly improve traffic safety."

Broad array of sensors

All driver assistance systems are based on sensors that collect relevant information from the vehicle and monitor the immediate traffic environment. A broad array of high-tech "sense organs" is available for this purpose: Video cameras recognize the traffic lane ahead and traffic signs. Infrared scanners are particularly adept at perceiving lateral movements, while radar sensors � depending on their frequency range � can detect objects at great distances and monitor the vehicle's immediate surroundings.

Every type of sensor has its strengths but also its weaknesses. A camera with image evaluation electronics, for example, can differentiate between two objects � such as a stop sign and a speed limit sign. But like the human eye, these systems have limitations when the lighting conditions are difficult. Thanks to their operating principle, radar sensors also function at night, and in this regard are superior to optical sensors. But they're unsuitable for classifying objects, which is a must when it comes to recognizing traffic signs.

One way out of the dilemma involves sensor fusion � the use of several sensors, each of which registers only one aspect of the surroundings. The sensors' combined data, however, yields a comprehensive image. The monitored area is enlarged, for example, when a long-range sensor with a narrow beam is combined with a short-range, wide-angle sensor (see the illustration above).

Fusion also has two additional advantages that aren't so obvious. In the example above, the monitored region can be divided up into three parts: two segments that are each exclusively monitored by just one sensor and a fusion zone in which both sensors can provide measurements and data on objects detected there. As a result, the electronics of the assistance system receive two independent measurements of each object in the fusion zone. These two measurements can then be processed to produce a reliable � in other words, a "correct" � #result.

Each individual measurement � for example, the distance to the vehicle in front � is subject to uncertainties, which are characterized by the variance of the respective measurement. That's why it's so important to ensure that the quality of the sensor information is correctly assessed.

The radar might, for example, show that a vehicle in front is 105 meters away. However, if the sensor-related spread or variance is +/-5 meters, the actual distance to the other vehicle could also be 100 or 110 meters. By combining the measurements made by two different sensors at the same time, the fused result becomes more reliable. In this way, the weaknesses of the two sensors can be offset, and activities in the fusion zone better assessed. By combining various sensor data, it is therefore possible to not only take more parameters of objects in the vicinity of the vehicle into account, but also expand the monitored region. In other words, the assistance system's "visual acuity" is significantly improved.

Complex mathematical models are used to link the data. These models take a wide range of factors into account, including whether a specific sensor type is very good or only average at measuring particular parameters. The data is therefore weighted in a sensor-specific manner when the fused measurements are calculated.

Sensor fusion also has another advantage: The results can be used with mathematical models to predict how the measured objects will probably move in the future. To make such forecasts, researchers need a model of how objects observed in the past will probably behave the very next moment. To ensure that the model does not deviate from the actual situation, the predicted result is continuously compared with the fused measurement data. The difference between the two is then fed into the data processing system, where it is used to correct the predictions for the next moment in time (see illustration on page 30).

Danger assessment

But how good is the system at actually "understanding" the situation? "Using a term like 'understanding' implies that we are providing the assistance systems #with human faculties," says Breuel. "And that, of course, would be completely unrealistic. Instead, we are trying to combine all available information into a single consistent picture that will help us analyze potential dangers and offer recommendations on how to proceed." In this respect, using the term "understanding" is definitely justified. A system that has been conceived to not only observe but also trigger a warning or even intervene in the event of a hazard has to be able to assess the level of danger associated with a specific registered situation.

From a technical standpoint this means that, first and foremost, each object's current situation must be registered. In addition, developments and activities observed up to that point must also be combined to form a model of the situation. By analyzing this situation, it is possible to detect indications that danger is imminent. And this is precisely what DaimlerChrysler researchers are trying to do with the situation analysis method they have developed. This method can be easily explained by taking a lane-changing vehicle as an example.

The data from the sensor fusion forms the input data for the situation analysis. The result of this process is, for example, a prediction of the probability that a vehicle in front will change from the inside lane to an outside lane. "This forecast is based on the best information that we have regarding the current situation and on the results of a model that tells us how the current situation will probably change in the future," says Breuel. A decisive factor in the process of determining whether a vehicle will change lanes is the way the data is interpreted. In this case, a probabilistic network analyzes the characteristics of the driving situation to determine the probability of a lane change on the basis of the object's previous behavior.

An important driving situation characteristic here is, for example, the course the vehicle involved has traveled up to now. The information for this process is provided by fused sensor data. The various vehicle-localization measurements made over time can be combined to show the course the vehicle has traveled thus far, which the researchers call a trajectory.

The shapes of these trajectories can be compared with model trajectories, which cover a set of typical lane-changing situations or courses. The more precisely a measured course conforms to a model trajectory, the more likely it is that the driver is about to change lanes. On the basis of the trajectory's measurement data and its interpretation, it is also possible to determine where a lane-changing vehicle will reach the line marking one's own lane. What's more, the vehicle's angle relative to the lanes can also be calculated.

The first of these values can be determined by means of a prediction � an extrapolation into the future. The result is a so-called polynomial regression � a curve connecting all points measured so far. This curve is lengthened by including the points that will probably be measured in the future. Using the polynomial regression, researchers can determine the point at which the expected course and the lane marking will meet. The angle of the vehicle's course can, in turn, be determined by measuring the angle between the tangents of the polynomial regression and the lane marking.

In the probabilistic partial network for the trajectory, the results of the three separate measurements are then weighted with a probability value. The greater the angle of the vehicle's course and the closer this course corresponds to a projected lane-change trajectory, the more likely an actual lane change becomes.

However, if the driving situation characteristic is used without the input of additional data, the system runs the risk of falsely thinking that a minor course change is the prelude to an imminent lane change. Perhaps, however, the driver in the neighboring vehicle has inadvertently swerved a little while turning on the radio.

That's why it's crucial that the system also take other driving situation characteristics into account. These can, of course, also be characterized with the help of the data provided by the fused objects. In the case of lane changes, such characteristics include not only the trajectory, but also the potentially lane-changing vehicle's lateral movements and its movement toward the gap between the vehicle ahead of it and any vehicles in the outside lane. Probability values can also be computed for these characteristics, thus generating three independently determined probability characteristics. These are then mathematically linked, creating an overall probability value of how likely an imminent lane change really is. If the overall probability value exceeds a certain limit for the lane-change situation, the assistance systems can either warn the driver or, if necessary, intervene.

Determining how high this limit should be is an important and critical step. Setting the limit's value lower means that the the system can provide a warning or intervene at an earlier stage. However, this also increases the number of incorrectly analyzed driving maneuvers. Conversely, when the threshold is set high, the system issues its warnings later, but avoids false assessments.

"Three factors are crucial when using this approach," says Breuel. "For one thing, we use the most accurate information available to determine how high the danger potential is in any given situation. We take the measurement accuracies of each stage in the process into account and weight the data according to quality. Here, the situation analysis with probabilistic networks, which are based on driving situation characteristic models, operates independently of the number and type of sensors used. In this process, the analysis doesn't only factor in the sensor data's margin of error; it also takes into account that the driving behavior of the surrounding motorists isn't 100-percent predictable."

This leads to an important consequence: Even if a sensor is replaced by another of a higher specification, or if an additional sensor for measuring further parameters is added, the situation analysis in the assistance system functions as normal. In fact, its quality of judgment improves, so that the system can correctly foresee a lane-changing maneuver more quickly.

In the first trials, Gabi Breuel's project team tested just such a lane-changing maneuver in an experimental vehicle. "The Distronic didn't recognize that the vehicle ahead was about to change lanes until it was already well into the test vehicle's own lane. With the situation analysis for changing lanes, we have every chance of recognizing such a vehicle 1.5 to 2 seconds earlier."

Recognizing Lane Changes

In order to recognize whether the red car is going to change lane, a "probabilistic network" analyzes the driving situation in terms of a trajectory, lateral motion and motion towards the gap. The values for the three characteristics are based on the data provided by the fusion of two radar sensors in the white car. For each driving situation characteristic, the system checks to see if certain conditions apply, and uses its findings to calculate the individual probability values. These values show how the vehicle will probably behave in the near future � its likely course, lateral motion and motion toward the gap. In the final step, the lane-change recognition system combines the three separate values into an overall probability that the red car will change lanes.

The benefits of sensor fusion: In the left part of the graphic, a radar sensor with a narrow angle (orange color) measures the distance to distant objects with great accuracy. The wide-angle sensor (green), in contrast, is good at measuring the lateral distance to objects at close range.

In the right half of the illustration, the measured values have been mathematically combined. In this way, it possible to more accurately determine the longitudinal and lateral directions of the vehicle in front.