2.1. LiDAR-based 3D Object Detection

A significant portion of the current state-of-the-art 3D object detection methods leverage LiDAR sensors to capture point clouds as input data ^{[2- 5]}. Notable methods in this category include PointRCNN ^[3], which employs a two-stage approach. In the first stage, it focuses on bottom-up 3D proposal generation, followed by the second stage, which refines these proposals in canonical coordinates to yield the final detection results. Another noteworthy method is Frustum PointNets ^[4], which directly utilizes raw point clouds as input to localize 3D objects. This localization is based on the frustum region determined using PointNet ^[5] and 2D detection. LIGA-Stereo ^[21] boosts the performance of stereo-based 3D detectors by learning LiDAR geometry-aware representations. By incorporating geometric cues from LiDAR data, it enhances the accuracy of detection.

2.2. Image-based 3D Object Detection

Several approaches have been developed for 3D object detection using monocular images or stereo RGB images as input ^{[6- 12]}. Disp R-CNN starts by detecting instance masks and 2D bounding boxes for each object. It then estimates disparities for pixels belonging to objects and uses a 3D detector to predict 3D bounding boxes from the instance point clouds. Stereo R-CNN ^[11] matches left and right Regions of Interest (RoIs) by employing the Stereo Region Proposal Network. It further utilizes dense alignments to refine 3D bounding boxes. 3DOP ^[12] generates 3D proposals by encoding object size priors, ground-plane priors, and depth information, such as point cloud density and free space, into an energy function. These 3D proposals are then used to regress the object pose and 2D bounding boxes using the R-CNN methodology. DSGN++ ^[22] enhances the performance of stereo-based 3D detectors by exploiting visual-spatial relations.

2.3. Tone Mapping

In the past decade, various tone mapping techniques have been proposed. There are two main categories of the present method, (1) tone reproduction operator-based methods, and (2) tone reproduction curve-based methods. Tone reproduction operator-based methods involve spatial manipulation of local neighboring pixel values and usually perform on multiple scales. An image can be modeled as $I(x,y) = L(x,y) * R(x,y)$, where $R$ is the reflectance component, and $L$ is the luminance component. Usually, $R$ has a very limited dynamic range. Therefore, if the researchers successfully process the two components separately, they can achieve range compression and color enhancement by processing the $L$. Tone reproduction curve-based methods ^{[15- 17]} work on the pixel distribution, where all pixels use the same mapping function. Larson et al. ^[16] used histogram adjustment of the high dynamic range radiance map. Duan et al. ^[17] achieved fast tone mapping by partitioning the pixel distribution into intervals and subsequently mapping these intervals to match the characteristics of low dynamic range devices. Zhang et al. ^[23] proposes a low-light image enhancement method based on convolutional neural networks, which significantly improves image brightness and contrast by learning the mapping relationship from low-light to normal-light images. Lamba et al. ^[24] presents a real-time dark stereo image restoration method based on GANs, which generates more natural enhanced images through adversarial training.

3.1. Proposed Tone Mapping Technique

The detection accuracy is usually affected by the low illumination. Since the original Disp R-CNN does not work well under low illumination conditions, we applied a new image pre-processing procedure, tone mapping, to the input image before object detection. Fig. 2(a) shows an example of object detection by original Disp R-CNN under low illumination conditions. Since the vehicles on the right-hand side are under the shadow, it is difficult for the detector to detect the vehicles. Therefore, we applied tone mapping to the input image before object detection. Fig. 2(b) shows an example of object detection by using tone mapping. Tone mapping is an effective technique to enhance the sensitivity of low illuminated objects. By enhancing the sensitivity of the low illuminated regions of an image, it is easy to detect these objects under low illumination conditions.

Fig. 2. Examples of object detection results. (a) Detection result without tone mapping. (b) Detection result with tone mapping.

In this study, we introduce a tone mapping technique applied to the input image prior to object detection. Tone mapping refers to the utilization of specific algorithms to compress the dynamic range, such as from 10000 : 1 in high dynamic range (HDR) images, to match the low dynamic range (LDR) display range, typically around 100 : 1, while preserving essential image characteristics. For a given input image, the log-average luminance is computed by the equation as follows.

where $N$ is the total number of pixels in an image, $L(x,y)$ is the input luminance, and $\delta$ is a small constant. The original input luminance $L(x,y)$ can be calculated from the RGB values by using the following equation.

The contrast factor (CF) is defined by the following equation.

Here, $C$ represents a parameter that governs the overall contrast and brightness of the resulting image. A smaller value of $C$ enhances visibility in darker areas while further compressing the brighter regions. The parameter $C$ can be adjusted to achieve the desired sensitivity and visual characteristics. In this paper, the parameter $C$ is set to 0.15.

After calculating the contrast factor, a simple tone mapping operator is given by

where $L(x,y)$ represents the original input luminance of the HDR image and $Ld$ is the displayable luminance. The tone mapping factor $CF$ has the effect of scaling high luminance values close to 1 and low luminance values close to 0. Therefore, it bridges the gap between high and low brightness values, promoting a smooth transition of intensity values in the resulting LDR image.

After calculating the displayable luminance Ld, the new output RGB values are determined using the equations as follows.

where $\gamma$ is the parameter utilized to manage the display color. When we get the new images, we can use them as input for 3D object detection.

3.2. Proposed Fine Disparity R-CNN

In order to boost the detection rate of small objects, we present an improved version of Disp R-CNN called fine disparity R-CNN. As shown in Fig. 1, to learn discriminative point cloud features to describe point clouds, we adopt the PointNet++ ^[18] with multi-scale grouping as a 3D detector backbone network instead of the PointRCNN in the original Disp R-CNN. The reason for choosing PointNet++ instead of PointRCNN used in the original Disp R-CNN is the demand to improve the detection rate of small objects. By reducing the group radius, PointNet++ can better preserve the feature information of small objects and extract more detailed feature information. PointNet++ has the ability of multi-scale grouping, which is helpful to capture the local region patterns at different scales. This is crucial for improving the detection accuracy of small-sized objects. Fig. 3 shows the overall architecture of PointNet++. PointNet++ consists of multiple set abstraction levels, each of which processes and abstracts a set of points to produce a reduced set. Each set abstraction includes three main layers: sampling layer, grouping Layer, and pointNet layer. The sampling layer selects a subset of points from the input point cloud that defines the centroid of the local area. The grouping layer constructs a set of local regions by identifying neighboring points around the centroid. The PointNet layer uses mini-PointNet to encode local area patterns into feature vectors.

Fig. 3. The overall architecture of the PointNet++.

In the Grouping Layer, the input consists of a point set of size and the coordinates of a set of centroids with a size of. The output comprises groups of point sets, each with a size of , where each group represents a local region, and K is the number of points in the neighborhood of the centroid points. After setting the size of the grouping radius, we identify all points within a specified radius of the query point K. The original grouping radius parameters for the set-abstraction layer of PointNet++ are 0.2, 0.5, respectively. In our model, the grouping radius parameters for the set-abstraction layer are changed to 0.15, 0.4, respectively. By reducing the grouping radius, we can get denser information. The features of small objects will be preserved better, so that more refined feature information can be extracted.

4.1. Performance Verification of the Proposed Fine Disparity R-CNN

In Table 1, a comparison of the performance for utilizing different grouping radiuses of two set-abstraction layers is given, respectively. The original grouping radiuses of two set-abstraction layers are 0.2 and 0.5 respectively. We gradually reduce the radius value to get denser information and see the corresponding changes in detection performance. We first keep the grouping radius of the second layer constant and gradually decrease the grouping radius of the first layer. Table 1 shows the average precision (AP) when IoU =0.5 for all cases. As shown in Table 1, the highest AP is obtained when the first grouping radius is 0.15. We gradually decrease the grouping radius of the second layer. One can see that the highest AP of 82.76% is achieved when the second grouping radius is 0.4.

As shown in Table 2, our proposed fine disparity R-CNN (FD R-CNN) outperforms the original Disp R-CNN. At IoU = 0.7, our results demonstrated improved Average Precision (AP) across the easy, moderate, and hard cases by 0.08%, 0.35%, and 0.1%, respectively. At IoU = 0.5, our results indicated enhanced AP across the same cases, with improvements of 0.26%, 0.85%, and 0.06% for easy, moderate, and hard cases, respectively.This indicates that our proposed optimization techniques are effective to improve the detection rate of small sized objects.

Fig. 4 illustrates a visual comparison of the original Disp R-CNN detection results versus our FD R-CNN detection results. In Fig. 4(a), one can see that the small sized object is missed by the original Disp R-CNN. In Fig. 4(b), the proposed FD R-CNN can successfully detect the small vehicle.

Fig. 4. Detection result comparison of Disp R-CNN and the proposed FD R-CNN on the KITTI dataset. (a) original Disp R-CNN and (b) the proposed FD R-CNN.

4.2. Performance Verification of the Proposed Tone Mapping Technique

We implemented the tone mapping method using MATLAB. Fig. 5 shows the detection result comparison of with and without tone mapping. We can observe that after using tone mapping, the vehicles in the shadows become more visible, and the pre-processing using tone mapping is very effective in object detection.

Fig. 5. Detection result comparison on the KITTI dataset. (a) Detection result on the original image (b) Detection result on the image using tone mapping.

To further verify the effectiveness of the tone mapping, we added tone mapping technique in the proposed FD R-CNN. We pre-processed 3769 validation images from the KITTI dataset. Table 3 shows the quantitative results of the Average Precision (AP). TFD R-CNN means the detection result of FD R-CNN adding a tone mapping. Compared with the FD R-CNN, TFD R-CNN results showed improved Average Precision (AP) on easy, moderate, and hard cases, when IoU = 0.7 by 3.86%, 1.9%, and 1.51%, respectively. When IoU = 0.5, TFD R-CNN results showed improved Average Precision (AP) on easy, moderate, and hard cases, by 0.17%, 0.38%, and 0.5%, respectively. This comparison shows that using tone mapping in pre-processing is effective in improving the performance of object detection. LIGA-Stereo ^[21] shows better results than our method because it utilizes LiDAR to provide high-precision depth and geometry information.

There are a total of 2088 low-light images of the KITTI dataset. A quantitative measure of the 2088 of low-light images is conducted. We use three indicators: contrast, information entropy, and average gradient to analyze the image. The contrast reflects the difference between different colors in the image. The greater the contrast value, the greater the contrast between different colors and the more obvious the color distinction. Image information entropy can represent the aggregation characteristics of image grayscale distribution. The greater the image information entropy value, the more image detail information. The average gradient can reflect the tiny detail contrast and texture transformation characteristics in the image. The larger the average gradient value, the clearer the image. The experimental results are shown in Table 4, all values are average of the 2088 images. It can be observed that values of contrast, information entropy, and average gradient are all improved after our proposed tone mapping technique is applied to the original images.

In Table 5, we have made a quantitative result comparison to evaluate the average precision of the proposed FD R-CNN on the original 2088 low-light images and the tone-mapped 2088 low-light images. TFD R-CNN means the detection result of FD R-CNN on the tone-mapped 2088 low-light images. Compared with the FD R-CNN, TFD R-CNN results showed improved average precision when IoU = 0.7 by 3.44%, and when IoU = 0.5 by 2.35%, respectively. This comparison shows that using tone mapping in pre-processing is effective in improving the performance of object detection.

4.3. Performance verification of the Proposed Method on the DrivingStereo Dataset

The proposed method is further evaluated on the DrivingStereo dataset ^[20]. The DrivingStereo dataset is constructed for stereo matching, which includes left and right image pairs under autonomous driving scenes and contains more than 180,000 images. Due to the DrivingStereo dataset not providing label files of objects, we only use this dataset for qualitative evaluation.

We evaluate the proposed FD R-CNN using the 300 demo images from the DrivingStereo dataset. The small car instances among that 300 demo images are miss detected by the original Disp R-CNN and are successfully detected by our proposed FD R-CNN. Fig. 6 shows a visualization of a sample result, we can see that one small car is missed by the original Disp R-CNN and is correctly detected by our proposed FD R-CNN. This result demonstrates that the grouping radius parameters adopted in the proposed FD R-CNN are not specific to the KITTI dataset but also work for other datasets.

Fig. 6. Detection result comparison of Disp R-CNN and the proposed FD R-CNN on the DrivingStereo dataset (a) original Disp R-CNN and (b) the proposed FD R-CNN.

We then evaluate the detection result using the images processed by our proposed tone mapping technique. The qualitative evaluation of the 300 demo images shows that objects with low illumination become more visible and can be detected. One sample result is displayed in Fig. 7, a miss detected car with low visibility in the original image becomes more visible and is successfully detected.

Fig. 7. Detection result comparison on the DrivingStereo dataset. (a) Detection result on the original image (b) Detection result on the image using tone mapping.

4.4. Evaluation of the Computation Time of the Proposed Tone Mapping Technique

Fig. 8 displays the computation time of the proposed tone mapping technique on the 7518 KITTI test images. Our method achieves an average computation time of 0.026 seconds per image on a CPU, while KinD network ^[23] spends an average computation time of 0.05 seconds per image on a 2080Ti GPU and Lamba et al. ^[24] takes 0.35 seconds per image on a CPU. This result demonstrate that our method allows for rapid preprocessing of input images to provide clearer inputs for 3D object detection.

Fig. 8. Computation time of the proposed tone mapping technique on the 7518 KITTI test images.