Feature Extraction and Recognition for Road Sign Using Dynamic Image Processing

In the research field of the intelligent transportation system (ITS) and the roving robot, the driver-support system auto-recognizing road signs and the robot control system auto-recognizing behavior-indication signs are variously suggested. Both systems use the dynamic image processing to acquire features such as the shape and design of signs. Artificial signs valid for the machine vision are sometimes used for these researches. However, in this study, the general road signs adopted in public roads are used, because machines such as robots and vehicles are supposed to live together in real environments. In this paper, nine kinds of signs one third in size such as "STOP," "NO ENTRY" and "NO PASSAGE" are prepared, and then some experiments were carried out to recognize these signs under various kinds of measurement condition.


Introduction
In the research field of the intelligent transportation system (ITS), the studies for the auto detection and auto recognition of the road sign are variously suggested [1][2][3].If any useful information acquired through the road signs are utilized, the effective driver support system can be constructed.On the other hand, in the research field of robot control for the robot navigation and autonomous behavior, it is required that the robots are controlled based on the sign in the static or dynamic image [4].For the signs, there are two types.One type is the general road signs that presuppose to be utilized by humans, and regulate the road traffic based on the rule of the road.Other type is the artificially shaped signs that presuppose to be auto-recognized by the computer vision.The detection and recognition system above mentioned use mainly the dynamic image processing to acquire the features such as the shapes and the design of signs.The artificial signs valid for the machine vision are sometimes used in these research fields.However, this study uses general road signs adopted in the public roads, because the robots or vehicles are supposed to live together in real environments.In the future, this system that detects and recognizes the signs is also expected to apply to the network robot to recognize the ambient environment, and to behave autonomously at a remote place.
In this paper, nine kinds of sings one third in size such as "STOP," "NO ENTRY" and "NO PASSAGE" are prepared, and then some experiments are carried out to recognize these signs under various kinds of measurement condition.These road signs consisted of maximum four colors and relatively simple designed are adopted.The signs are detected based on the color information of the sign, and are recognized based on the ratio of the composing colors of the sign.

Program configuration of detection and recognition
Three kinds of sign, "white on a red ground," "white on a blue ground," and "black on a yellow ground" are adopted.The program configuration of detection and recognition is explained below, by giving the example of "white on a red ground" such as "STOP," "NO ENTRY," and "NO PAS-SAGE" as shown in Figure 2.

Figure 2. Example of sign "white on a red ground."
The program consists of three main function blocks "Capture of the dynamic image," "Detection," and "Recognition and Display of result" as shown in Figure 3.

Capture of the dynamic image
In the "Capture of the dynamic image" block in Figure 3, the RGB signals are to the color-difference signals YCbCr as shown in Figure 4, where Y= 0.299 R+0.587 G+0.114 R is intensity, Cb=0.5 B-0.169 R-0.331G is a color-difference signal for blue, and Cr=0.5 R-0.419 G-0.08 B is a color difference signal for red, and are outputted to the "Detection and recognition" block.The upper and lower sides of the original image size 320 by 240 are trimmed away to be the size 320 by 180 because it rarely seems that any signs exist at upper and lower area in the image.

Detection
The sings "white on a red ground" have one white pattern in the red area.In the "Detection" block, the white and red areas, which are larger than the threshold value, are extracted from the image by using the Y and Cr signals.The ratio of the height of red area to that of white area, and the ratio of the width of red area to that of white area are calculated.The calculation procedure of these ratios is concretely shown below.
The images by Y and Cr signals are shown in Figure 5.By using the red and white area, the ratio of the height of red area to that of white area, which is called the height ratio, and the ratio of the width of red area to that of white area, which is called the width ratio, can be calculated as shown in Figure 7.For signs of "STOP," "NO ENTRY" and "NO PASSAGE," the height and width ratios are shown in Table 1.These values have the range because the image is reflected by the light circumstances and/or the image processing operations.

Recognition and Display of result
"Recognition and Display of result" block distinguishes between these three kinds of signs by using the height and width ratios from the previous block.If either of the ratios, for example, the height ratio, is used, the range of the ratio for "STOP" and "NO ENTRY" overlaps partially.For the width of these three signs, they are different.So, the signs can be distinguished by using only the width ratio.However, these ratios are not always different, and if the information is restricted to just one kind of ratio, the signs may be unrecognizable.We are sure that both ratios are useful to distinguish or recognize any signs.

Figure 7. Definition of height ratio and width ratio
For the recognition algorithm, the following idea is adopted.As shown in Table 2(a), three thresholds for each of the height and width ratios are set, and arbitrary numbers are assigned to each of the thresholds so that the code "a" times the code "b" corresponds to the particular sign shown in Table 2(b), where three ranges are determined based on the measured range values in Table 1.

Table 1. Height and width ratios for sign of "White on a red ground."
Table 2 Code assigned to each sign.

Experiment
Nine kinds of signs one third in size are prepared.These road signs are "STOP," "NO ENTRY," "NO PASSAGE" as "white on a red ground" sign, "ONLY BICYCLE," "CROSSING," "NO ENTRY EXCEPT DESIGNATED DIRECTIONS" as "white on a blue ground" sign, "INTER-SECTION," "RAILROAD CROSSING," "UNDER CON-STRUCTION" as "black on a yellow ground" sign.Then some experiments were carried out to recognize these signs under some measurement conditions indoors and outdoors.These road signs consisted of maximum four colors and relatively simple designed are adopted.The signs are detected based on the color information of the sign, and are recognized based on the ratio of the composing colors of the sign.
Following the proposed processing procedure, the sign "RAILROAD CROSSING" was processed to recognize.The result was shown below.The original RGB image shown in Figure 8(a) is first transformed to the YCrCb image.Then, by setting the threshold values for Y, Cr, and Cb signals, and by some kinds of processing, yellow and black areas are extracted as shown in Figure 8(b) and (c).For the yellow area of the sign and the black area inside the sign, the coordinates of above left for the height and width areas are detected during the processing, and each of these areas are surrounded by a square and added color, with light blue for the yellow area and yellow green for the black area.At the above left corner, the name of the sign "RAILROAD CROSSING" is displayed as shown in Figure 9.

Conclusions
This research proposed the processing method to extract the feature of the road sign and recognize what the road sign means.Nine kinds of signs were adopted, which were consisted of maximum four colors and relatively simple designed.This method paid attention to the ratio of the composed colors.However, by using only the ratio of colors, the recognition may be mistaken.Therefore, the height and width ratios of the ground color area to the inside pattern area of the road sign were utilized to recognize.
In future, further considerations below are needed.If two signs have almost the same height and width ratios, the recognition is impossible.In this case, other information such a configuration needs to be added.And, the color state of the image changes depending on the light condition in the daytime and in the nighttime.And Furthermore, the camera angle to the sign affects the shape of the image.Any measure needs to be taken into consideration in the next step.

Figure 1
Figure 1 shows the system configuration of the road sign detection and recognition.The scene including the sign is captured by a camera to send a computer.Then, the sign recognition program written by us analyzes the dynamic image, detects only the sign, and recognizes what the sign is, based on the criterion mentioned in Sec.3.The result of recognition displays at the above left corner, and the recognized sign is

Figure 1 .
Figure 1.System configuration of the road sign detection and recognition, which consists of a camera, a computer, and a display.

Figure 3 .
Figure 3. Programs consist of three function blocks "Capture of the dynamic image," "Detection," and "Recognition and Display of result."

Figure 4 .
Figure 4. Image by YCrCb signals transformed from RGB signals.
result By setting the threshold value of Y to 0.57, the image by Y signal is transformed to the binary image.This binary image includes also many white areas other than the one white pattern in the sign.In other words, it does not mean that only the sign is extracted.And, by setting the threshold value of Cr to 0.56, the image by the Cr signal is also transformed to the binary image.The white area in this binary image represents the red area of the sign as shown in Figure6(a).The dilation and erosion operations are applied to this binary image several times, in which the white area corresponds to the extracted red area, so that there are no black areas inside this white area.Next, this image with no black area inside the sign is ANDed with the binary image by the Y signal.Then, only the white area in the sign is extracted shown in Figure6(b).

Figure 5 .
Figure 5. Two images by the different signals.

Figure 6 .
Figure 6.Color areas of the sign.

Figure 9 .
Figure 9. Result display with colors to easily see where the recognized sign is.