2.1. Visual SLAM System

Fig. 1 depicts the structure of the Visual SLAM system. It consists of five key components: Camera Sensor, Tracking, Optimization, Loop Closure, and Mapping.

Fig. 1. Visual SLAM flowchart.

• Camera sensor

The camera sensor is responsible for collecting image data. Cameras are categorized into monocular, stereo, and RGB-D types.

Monocular camera utilizes a single lens, offering advantages such as low cost and lightweight design. However, it poses challenges in accurately estimating landmark depth. Additionally, it may lead to scale ambiguity issues during map construction.

Stereo camera utilizes two lenses and involves calibration, rectification, matching, and computation processes to obtain depth information. While it enables depth information to be acquired both indoors and outdoors, it suffers from the drawback of high computational load.

RGB-D camera measures depth information in real-time using structured light or Time-of-Flight (TOF) sensors. However, it may have a limited measurement range and could be challenging to use in outdoor environments.

• Tracking

Tracking includes the concept of visual odometry (VO), which determines the position and orientation of the camera by analyzing camera images. This operates through stages such as feature extraction, matching, motion, and pose estimation. It is primarily divided into two approaches. Table 1 illustrates the differences between the two methods.

The indirect method utilizes features, unique points in images, to perform analysis. It is robust to changes in lighting conditions, object motion, and dynamic environmental changes, demanding fewer computations. However, it may exhibit instability in scenarios with significant motion blur or in environments with texture-less patterns, resulting in relatively sparse reconstructed 3D maps.

The direct method primarily processes images using pixel values themselves, analyzing brightness variations between pixels. This method exhibits robustness to motion blur and environments with low texture, allowing for the generation of denser 3D maps. However, it is vulnerable to changes in lighting conditions and instability in dynamic scenes, demanding higher computational resources due to utilizing all pixel values in the image.

• Optimization

Based on the estimated camera position and orientation information obtained from tracking, Optimization refines the predicted pose more accurately. Two methods used in Optimization are filter-based methods and graph-based methods.

Filter-based optimization: This method deals with probabilistic system state estimation. The most prominent filter is the Kalman Filter. Fig. 2 illustrates a Flow Chart for the Kalman Filter. The Kalman Filter is applied to linear systems. It predicts the state using a linear model after setting the initial estimate and uncertainty and updates using observed data. The Extended Kalman Filter is an extended version applicable to nonlinear systems, approximating nonlinear functions to perform prediction and update. The Particle Filter represents probability distributions with particles for state estimation and is flexible for application to nonlinear and non-Gaussian systems.

Fig. 2. Kalman filter.

Graph-based optimization: This includes Factor Graph Optimization (FGO), Pose Graph Optimization (PGO), and Bundle Adjustment (BA). These methods utilize graph structures composed of nodes and edges to perform optimization, employing nonlinear optimization techniques. FGO utilizes a structure consisting of variables and factors, while PGO models camera poses as nodes and movements as edges. Fig. 3 describes the composition of the Pose Graph.

Fig. 3. Pose graph.

BA adjusts the camera poses and landmark positions using input images and feature points, minimizing reprojection errors.

• Loop closure

In Fig. 4, the role of Loop Closure in closed-loop scenarios is illustrated. Loop Closure verifies whether the current location has been visited previously and corrects accumulated position estimation errors as the robot moves. Typically, Loop Closure is performed using the Bag-of-Words (BoW) model. The BoW model clusters extracted keypoints from images to form groups of similar keypoints, known as “visual words.” Each image is represented by a vector recording the frequency of occurrence of corresponding visual words, allowing measurement of similarities between images. If the current location is close to a previously visited place, it is considered a Loop Closure candidate, and if necessary, the robot’s position estimation is corrected.

Fig. 4. Loop closure.

• Mapping

Mapping is the process of creating a detailed representation of the surrounding environment, known as a map. Maps are broadly classified into Metric maps and topological maps. Fig. 5 illustrates two types of maps.

Metric map is to accurately model the surrounding environment in 3D space, representing actual physical distances and directions. It primarily uses 3D point clouds, grids, or various geometric structures to represent the environment. As it contains detailed spatial information, it is useful for determining precise locations and exploring the surrounding environment. Metric maps can be classified into sparse maps, which contain limited information, and dense maps, which provide detailed and comprehensive information. Typically, the process of creating a map involves setting the intrinsic and extrinsic parameters of the camera, finding corresponding pairs in two images, restricting possible locations using epipolar constraints, calculating 3D coordinates through triangulation, collecting these 3D points to create a 3D point cloud, and reconstructing the structure of the environment.

Fig. 5. Metric and topological map comparison.

Topological map is to represent the surrounding environment using connected nodes and edges, primarily depicting the structure and relationships within the environment. Nodes typically represent places or locations, while edges indicate connections between these places. This allows robots to identify paths for moving from one place to another. Topological maps provide abstracted information about the surrounding environment and do not include detailed spatial information.

Semantic mapping combines geometric information with semantic knowledge of the environment. This involves identifying objects through object recognition and segmenting images into object instances via semantic segmentation, providing object or structural information corresponding to each pixel. This information is integrated with SLAM systems to better understand the environment.

Implicit mapping involves implicitly representing the neural map of the environment to encode both geometric and semantic information. It captures environmental characteristics using two methods: deep autoencoders and neural rendering-based scene representation. Deep autoencoders compress input data into high-level abstract representations to implicitly capture key features of the environment. In contrast, neural rendering learns and models the 3D structure of the environment to reconstruct scenes. These methods are useful for understanding the surrounding environment and extracting valuable information without explicitly representing the environment’s features.

2.2. Degrading factors of Visual SLAM

In the field of SLAM, several challenges primarily arise from factors such as lighting changes, presence of dynamic objects, rapid camera movements, love-texture environments, and unstructured environments.

Natural phenomena or artificial factors can induce lighting changes, causing abrupt alterations in image brightness. These changes can compromise the consistency of feature matching, potentially leading to errors in the SLAM process.

Dynamic objects in images also pose a challenge. While static elements provide a stable, dynamic objects can disrupt this stability with their rapid or unpredictable movements, making camera pose tracking difficult and distorting the map.

Rapid camera motion introduces shaking and distortion in images, leading to data loss and inconsistency between consecutive images, necessitating high computational capabilities to adapt to swift environmental changes.

Low-texture environments, like indoor settings, lack discernible color or brightness differences and are characterized by repetitive or monotonous patterns. Techniques such as line detection are used to overcome these limitations.

Unstructured environments, typically encountered outdoors, lack fixed roads, paths, or structures. Irregular terrain, various obstacles, and absence of structure make it challenging for robots or autonomous systems to navigate or operate. Interpreting sensor data and modeling the environment for SLAM systems become difficult in such environments.

3.1. Improvement Using Various Features

In early research on VSLAM, point features have been widely utilized ^[1-6]. These point features offer simplicity and fast processing speed, making them suitable for real time applications. A well-known feature detector for such point features is referenced in ^[7-14]. However, point features have certain limitations. In particular, they struggle to extract sufficient features in environments with rapid changes in lighting conditions or fast camera rotations.

Accordingly, attempts to utilize line features have been proposed ^[15-20]. Line features enable feature extraction even in texture-less environments. However, several challenges need to be addressed to utilize line features. Processing line features requires a significant amount of time. Some studies introduce the Line Segment Detector (LSD) to minimize the time required for line extraction ^[15,16,20]. Efforts have been made to reduce the number of parameters required for optimization by utilizing the Plücker matrix when employing line features ^[17]. This approach has increased computational efficiency and stability compared to the traditional endpoint representation by representing the geometric information of 3D lines in coordinates. Next, during the mapping process using line features, degeneracy occurs when lines have identical or similar direction vectors, resulting in a loss of diversity in information. Studies have been conducted to address this issue by utilizing vanishing points to correct direction vectors ^[18,19].

Additionally, attempts to incorporate plane features have been proposed ^[21-23]. Planes are relatively stable features, making them effective in indoor or artificial environments. However, plane detection faces limitations due to the irregularities of natural terrain and various elements in the environment. UPLP-SLAM ^[21] considers both homogeneous feature correspondences like point-to-point, line-to-line, plane-to-plane, as well as heterogeneous feature correspondences like point-to-line, point-to-plane, and line-to-plane. Structure PLP SLAM ^[22] proposes various optimization techniques to overcome the challenge of reconstructing geometric elements at a consistent scale. Fig. 6 shows a structured scene with 2d features, orthogonal lines and planes. The results are based on PlanarSLAM [58] execution using the TUM RGB-D [60] dataset.

Fig. 6. Example with point, line, plane features in a structured scene.

The advancement of deep learning technologies has shown remarkable achievements in various tasks related to image processing. Particularly, neural network models like Convolutional Neural Network (CNN) demonstrate outstanding performance in image feature extraction. CNN efficiently learns local patterns and features of images through convolution and pooling operations. Due to these characteristics, CNN exhibits superior performance in various image processing tasks, primarily used in tasks such as image recognition, classification, object detection, and segmentation. With the development of deep learning technologies, attempts have been made to replace the process of extracting keypoints in traditional VSLAM structures. Deep learning-based VSLAM utilizes trained neural networks to extract features from images, enabling the utilization of more information compared to traditional VSLAM’s keypoint extraction process, thereby demonstrating robust performance under various conditions.

There are attempts to replace the feature extraction part in the ORB-SLAM2 ^[6] framework with CNNs ^[24-28]. In DF-SLAM ^[24], a TFeat network is proposed, designed with a thin and efficient structure to extract feature vectors of fixed size from image patches. In DXSLAM ^[25], an HF-Net is proposed, which is robust and efficient in extracting both local and global features. Reference ^[26] introduces SAFT, a learning-based descriptor that uses dynamic feature transformation. Lift-SLAM ^[27] achieves denser and more accurate matching results using the LIFT network. Reference ^[28] proposes a multitask feature extraction network that generates both keypoints and their descriptors simultaneously to enhance performance.

Research has been proposed to improve performance by utilizing artificial structures or information ^[29-32]. This enhances the diversity and intensity of features extracted in challenging environments, enabling the system to recognize and track more accurately. However, it has the drawback that artificial visual information needs to be prepared in advance within the environment. Tagslam ^[29] employs markers called AprilTags, which consist of unique patterns of squares easily detectable by cameras. Spm-slam ^[30] uses square planar markers, with the constraint that at least two markers must be present in a single frame for operation. Ucoslam ^[31] combines natural landmarks (keypoints) with artificial landmarks (square markers), automatically computing the scale of the surrounding environment. Additionally, leveraging text information has been proposed Textslam ^[32], which involves detecting text in images and generating high-quality 3D text maps. Table 2 above details various feature type used in SLAM, including their strengths, limitations, and key techniques.

3.2. Improvement Using Object Detection

In the early stages of Visual SLAM, object-related research mostly relied on methods using optical flow. Optical flow is a technique for tracking the motion of objects at the pixel level in images, primarily used for detecting and tracking moving objects. However, optical flow has the drawback of operating at the individual pixel level, thereby not considering the structure or semantics of objects. The advancement of deep learning has overcome these limitations and introduced new methods for more accurate object detection and tracking. With the introduction of technologies such as semantic segmentation and instance segmentation, objects can now be recognized and distinguished more accurately.

Objects detected in images are classified into static and dynamic objects. Static objects represent entities that do not change, typically providing stability in position estimation and enhancing the accuracy of SLAM. On the other hand, dynamic objects are those that move or change, and the motion of such objects can degrade the consistency of the map and introduce errors in position estimation. Consequently, there are various studies focusing on handling dynamic objects.

Efforts to minimize the influence of dynamic objects by removal have been proposed ^[33-36]. DS-SLAM ^[33] combines a real-time semantic segmentation network called SegNet with an optical flow algorithm to remove dynamic objects. Dynamic-SLAM ^[34] detects dynamic objects using Single Shot MultiBox Detector (SSD) and treats them as outliers, subsequently removing the corresponding keypoints in those areas. DP-slam ^[35] employs a Bayesian probabilistic propagation model to iteratively update probabilities in real-time for dynamic object removal. SimVODIS++ ^[36] estimates the relative poses and depth maps between input image frames to generate static background information, which is then compared with the object detection results of the current frame to remove dynamic objects.

There are attempts to utilize dynamic objects without removing them ^[37-39]. Cubeslam ^[37] improves camera pose estimation accuracy by leveraging motion model constraints that track the movements of individual objects represented as 3D cuboids, without treating dynamic objects as outliers. DynaSLAM II ^[38] tracks dynamic objects using instance semantic segmentation and ORB features, optimizing the structure of static scenes and the trajectories of dynamic objects along with camera poses. Moving SLAM ^[39] assumes the entire scene to be non-rigid but approximates rigid regions in small areas to infer and predict the motion of moving objects.

There are attempts to utilize objects to create maps ^[40-42]. Object maps enable robots to effectively understand and interact with their surrounding environment, accurately represent structured indoor environments, enhance the precision of SLAM systems, and enable efficient data management for accurate localization. Reference ^[40] generates object landmarks using semantic information to obtain consistently maintained points, estimating the positions and depth information of objects through the alignment between images obtained from the camera and the model. Reference ^[41] combines ORB-SLAM2 ^[6] and Mask R-CNN ^[43] to recognize objects in 2D images, model them in 3D space, and create semantic point cloud maps. SO-SLAM ^[42] generates object-level maps representing the environment using quadrics. Table 3 above presents various methods for managing dynamic objects in SLAM, either by removing or utilizing dynamic objects for more robust map creation and object tracking.

3.3. 3D Scene Understanding

In SLAM technology, 3D point clouds have traditionally been employed to model the surrounding environment and estimate one’s own position. However, this approach is limited by the resolution and density of the sensor data and is vulnerable to environmental factors.

Accordingly, various efforts have been made to understand the semantics and structure of 3D space. Initially, attempts were made to comprehend it through semantic segmentation ^[33,44]. Semantic segmentation is a technique that classifies each pixel in an image into specific classes or meaningful regions to segment the environment. This allows for the identification and utilization of elements in space to estimate more accurate positions and orientations. In ^[33], a semantic octo-tree map is used to hierarchically store 3D space, where each cube contains information such as position, color, and texture to create the map. Reference ^[44] generates maps including semantic elements such as lane markings, crosswalks, ground traffic signals, and stop lines on road surfaces, and updates them through crowdsourcing methods.

Recently, neural network-based models like NeRF have emerged to enhance the representation of 3D maps. NeRF offers several advantages compared to traditional 3D point clouds. It can generate high-resolution 3D maps without being constrained by sensor resolution or data density. Additionally, It is more robust to environmental factors such as noise and lighting variations, estimating detailed information about light sources and surface properties for enhanced visual effects. Consequently, incorporating NeRF into SLAM has been proposed to represent scenes. This approach aims to build 3D models and color information while maintaining accurate positional information from consecutive image frames. Such SLAM research utilizing NeRF employs implicit neural representation to construct 3D maps and perform tracking ^[46-57]. Neural implicit representation involves utilizing neural networks to implicitly represent spatial information. It uses MLPs to occupy 3D points or map them to colors and optimizes scenes. To achieve this, grid structures like voxel grids are used, enabling the network to predict attributes for each point. iMAP ^[46] employs a single MLP to represent the entire scene, optimizing computational speed and resource usage by processing only the necessary data in required areas through guided pixel sampling based on dynamic information. This approach enables tracking speeds of up to 10 Hz and global map updates at 2 Hz. Nice-SLAM ^[47] organizes multiple MLPs into a hierarchical structure, with each MLP corresponding to different levels and types of local information, representing scenes at multiple levels of detail. Furthermore, it enhances tracking accuracy and stability by reducing the impact of dynamic objects, filtering out pixels with large depth/color rendering loss. iSDF ^[48] utilizes Signed Distance Fields (SDF) to represent 3D maps, which are then utilized for collision detection and path planning in robotics. By employing neural networks, it estimates signed distances for input 3D coordinates, enabling the understanding of surrounding environment shapes and distances to obstacles. Additionally, it provides a more accurate map through adaptive levels of detail and reducing noise in the observations, thereby improving navigation and tracking performance. Idf-SLAM ^[49] predicts T-SDF(Truncated Signed Distance Function) using a single MLP, considering the 3D positions and viewing directions of input points. T-SDF encodes distances from points and truncates points beyond a certain distance, reducing memory usage while aiding in the creation of accurate 3D models. Additionally, it utilizes pairwise point cloud registration networks to align point clouds captured at different times or locations, enhancing camera tracking accuracy. ESLAM ^[50] enables accurate and fast 3D map generation through a hybrid representation consisting of multi-scale axis-aligned feature planes and shallow decoders. This approach leverages implicit neural representation and TSDF to convert each point’s features into TSDF and RGB values, providing high-quality 3D reconstruction performance. Additionally, it improves tracking performance by utilizing a global loss function and Adam optimizer, while excluding outlier pixels and rays without ground truth depth during optimization. MeSLAM ^[51] introduces a network distribution strategy where multiple MLPs are assigned to different regions to process specialized information. This allows each neural network to update and adjust its designated area independently, enabling adaptation to new information or changes. Moreover, by decomposing large scenes and stitching them during reconstruction, it maintains high accuracy and consistency throughout the map-building process. Vox-fusion ^[52] combines neural implicit representation with traditional volumetric fusion methods to encode and optimize scenes within each voxel, constructing a 3D map. Additionally, it utilizes an octree-based structure to dynamically expand scenes and detects object instances to enhance tracking performance. vMAP ^[53] constructs a SLAM system using an MLP neural field model, enabling efficient and accurate modeling of each object. Additionally, it leverages vectorized training to simultaneously optimize multiple objects. Moreover, the system detects object instances within the scene and continuously updates them to enhance tracking performance. Co-SLAM ^[54] utilizes multi-resolution hash-grid and one-blob encoding to achieve high convergence speed and accurate 3D map generation. Additionally, by combining joint coordinate and sparse-parametric encodings, it performs global bundle adjustment, achieving efficient memory usage and robust tracking performance without the need to maintain keyframe selection.

NeRF-SLAM ^[55] combines Dense monocular SLAM with NeRF. By accurately estimating camera poses and generating depth maps, it constructs a scene’s neural radiance field, providing enhanced 3D scene reconstruction capabilities using associated uncertainty information. Orbeez-SLAM ^[56] integrates ORB features with NeRF to build 3D maps. ORB features contribute to initial pose estimation by identifying and extracting distinctive points in the camera’s surroundings. Subsequently, NeRF predicts color and density values for these points to construct a dense 3D map. Nicer-SLAM ^[57] enables high-fidelity 3D scene reconstruction and novel view synthesis by utilizing hierarchical neural implicit representation to construct 3D maps. Additionally, it enhances tracking performance by incorporating additional information such as monocular geometric cues, optical flow, and warping loss to strengthen geometry consistency. Fig. 7 compares rendering results from NeRF-SLAM, Nice-SLAM, and Nicer-SLAM, highlighting differences in scene reconstruction quality. Table 4 below presents a comparative overview of recent advancements in the integration of SLAM with NeRF. The table categorizes these advancements based on their core functionalities, such as scene representation and optimization, distance fields and collision detection, volumetric and implicit representations, and integration with feature-based methods.

Fig. 7. Novel view synthesis results on the Replica dataset [59].