2.1. Proposed Architecture based on GAN

Fig. 1 presents a detailed Generative Adversarial Network (GAN) architecture ^[26] specifically designed for the concurrent processing and generation of diverse data modalities such as images, radar data, and voice data. This system exemplifies a sophisticated approach to data synthesis, harnessing the power of adversarial training to produce remarkably realistic synthetic data across these varied domains. The structure is meticulously designed with independent pathways for each data type, ensuring optimized handling and evaluation at every stage.

Fig. 1. Proposed architecture.

Each generator network consists of 3 convolutional layers with kernel size 3×3, followed by batch normalization and Leaky ReLU activation. The radar and voice generators operate on 64×64 FFT-transformed feature maps, while the image generator processes 128×128 grayscale inputs. Discriminators use four convolutional blocks followed by a fully connected layer. Loss functions are based on binary cross-entropy with gradient penalty applied during training to stabilize convergence.

At the heart of this architecture lies the core principle of GANs which is the adversarial relationship between a generator network and a discriminator network. The Generator networks, as depicted in the diagram, are responsible for creating synthetic data samples from random noise. Each data stream such as image, radar, and voice has its own dedicated generator, labeled generically as “Generator.” These generators are tasked with transforming the input noise into data that mimics the statistical characteristics of real data within their respective modalities. The quality of the synthetic data generated is crucial, as it directly impacts the effectiveness of the downstream processes. The Discriminator networks, labeled as “Discriminator1,” “Discriminator2,” and “Discriminator3” for image, radar, and voice data respectively, play the role of evaluators. Their purpose is to distinguish between the real data samples (“Real Image,” “Real Radar Image,” “Real Voice data”) and the synthetic data samples generated by their corresponding generators (“Fake Image,” “Fake Radar Image,” “Fake Voice data”). The discriminators are trained to minimize their loss, becoming increasingly adept at identifying subtle differences between real and fake data. The adversarial relationship between the generators and discriminators drives both networks to improve continually.

A key element of the radar and voice data pathways is the inclusion of a Fast Fourier Transform (FFT) block. This block performs a mathematical transformation on the real radar and voice data, converting it from the time domain to the frequency domain. This transformation is particularly useful for analyzing signals, as it reveals the frequency components that comprise the signal. Analyzing data in the frequency domain can expose hidden patterns and features that are not readily apparent in the time domain, which assists the discriminator in distinguishing between real and synthetic signals. Also shows “Min. loss Discriminator1,” “Min. loss Discriminator2,” and “Min. loss Discriminator3” blocks connected after the Discriminator blocks. These illustrate that each discriminator is undergoing training to minimize its loss function. The “loss” represents the error in the discriminator’s ability to correctly classify real and fake data. Minimizing the loss means improving the discriminator’s accuracy.

The outputs of all three discriminators are fed into a Selector component. This selector serves as an intelligent aggregation point, combining the evaluations from all three data streams. The specific function of the selector is application-dependent. For instance, it could be designed to select the data stream with the most rfealistic synthetic data, to generate a consensus opinion based on the outputs of all three discriminators, or to prioritize one data stream over others based on pre-defined criteria. The selector enables a holistic view of the generated data, facilitating informed decisions and actions based on the combined information from all three modalities.

2.2. Discriminator

Fig. 2 presented illustrates the intricate structure of a deep neural network, offering a comprehensive view of its various layers and the complex flow of data between them. This sophisticated architecture is designed to process and analyze data. Following the convolutional layers, we see the application of the RELU (Rectified Linear Unit) activation function. RELU is a cornerstone of modern deep learning architectures, introducing non-linearity into the network. This non-linearity is crucial as it allows the network to learn and represent complex, non-linear relationships within the data. The RELU function works by outputting the input directly if it is positive, and zero otherwise. This simple yet effective approach helps mitigate the vanishing gradient problem that plagued earlier neural network designs, allowing for more efficient training of deep networks.

Fig. 2. Discriminator various layers network.

The diagram also showcases NORM (normalization) layers. These layers play a vital role in stabilizing and optimizing the network’s performance. Normalization techniques, such as batch normalization or layer normalization, help to standardize the inputs to each layer. This normalization process has several benefits: it accelerates training by reducing internal covariate shift, allows higher learning rates, and can act as a form of regularization, potentially reducing the need for dropout in some cases.

POOL (pooling) layers are another critical component ultiple advantages: it reduces the computational load, helps to make the features more robust to small translations in the input, and begins to introduce a level of spatial hierarchy into the network’s understanding of the image.

An interesting variation shown in the diagram is the Leaky RELU activation function. By allowing a small gradient for negative inputs, Leaky RELU ensures that neurons have a chance to recover and contribute to the learning process. The final layers of the network are typically FC (fully connected) layers. In fully connected layers, each neuron is connected to every neuron in the previous layer, allowing the network to combine features in complex ways to make its final decision. The arrows between the layers in the diagram represent the flow of data through the network. This flow is typically unidirectional during the forward pass, where input data is processed to produce an output. However, during training, there is also a backward pass where gradients flow in the opposite direction, allowing the network to update its weights and improve its performance. It’s worth noting that the specific arrangement and number of layers can vary significantly between different network architectures. The diagram likely represents a general structure, and in practice, networks may have many more layers, skip connections, or other architectural innovations.

This hierarchical learning is what enables deep neural networks to achieve remarkable performance on a wide range of complex tasks, from image classification and object detection to natural language processing and beyond. This diagram encapsulates the essence of modern deep learning architectures, showcasing how relatively simple components can be combined to create a powerful system capable of learning and representing complex patterns in data. The interplay between convolutional layers for feature extraction, activation functions for non-linearity, normalization for stability, pooling for dimensionality reduction, and fully connected layers for final decision-making creates a versatile and potent tool for tackling a wide array of challenging problems in artificial intelligence and machine learning.

2.3. Generator

Fig. 3 illustrates the structure of a Convolutional Neural Network (CNN), which is widely used in tasks like image classification and object detection. The flow begins with an input image and progresses through multiple layers designed to extract features and classify the input. Here’s a detailed explanation of each stage:

Fig. 3. Generator various layers network.

The process starts with a convolutional layer, which applies filters (kernels) to the input image. This operation extracts local features, such as edges or textures, while preserving spatial relationships in the data. The network continues with additional convolutional layers, each followed by activation functions and pooling operations. As these layers progress deeper into the network. The filters become more complex, capturing higher-level features such as shapes or patterns. The spatial size of the feature maps decreases due to pooling, but the number of feature maps often increases.

Fully connected layer outputs the classification result. This hierarchical structure allows CNNs to learn both low-level and high-level features effectively, making them powerful tools in computer vision tasks.

2.4. Priority Selection Block Diagram

Fig. 4 illustrates a signal selection process based on minimum loss values obtained from three different input modalities: image data, radar image data, and voice data. Each modality provides a precomputed minimum loss signal, which is first stored in its respective register. These signals are then passed through a normalization block to ensure they are scaled uniformly for fair comparison, regardless of their original range or unit.

Fig. 4. Modality extraction using minimum loss values generator.

Normalization is performed using Min-Max scaling across the three minimum loss signals to [0,1] range for fair comparison. Comparator logic is implemented using a three-way fixed-point comparator module that selects the signal with the smallest normalized loss value as the final output.

Following normalization, the signals are input into a comparator module that evaluates the values and identifies the signal with the lowest loss. The final output is the Min Loss Signal, representing the most reliable or accurate modality at that instance. This architecture can be utilized in multimodal learning systems to dynamically select the best-performing input source, enhancing overall inference robustness.

This system processes three types of signals-image signals, radar signals, and voice signals-by generating, evaluating, and optimizing each one independently. Once processed, the system selects one signal based on a priority determination process.

The process begins with a reset step, which initializes all modules. After this initialization, the system concurrently initiates three separate signal processing paths.

In the image signal processing path, an image signal is first generated. The generated signal is then evaluated by a discriminator designed for image signals. After evaluation, a minimum loss operation is applied to reduce signal degradation or error. The system then checks whether the optimized signal meets a predefined threshold. If the signal does not meet this threshold, the process loops back to the image generation step and repeats. If the threshold is met, the signal proceeds to the next stage.

Fig. 5. Overall control flow including selection logic and parallel operation.

The radar signal processing path follows the same structure. A radar signal is generated, passed through a radar discriminator, and optimized via minimum loss operation. It is then evaluated against the same type of threshold condition. If the condition is not met, the signal is regenerated; if it is met, it moves forward.

The voice signal processing path also mirrors the same process: generation, discrimination, minimum loss optimization, and threshold checking. The signal continues in a loop until the threshold condition is satisfied.

Once one or more of these signals meet their respective thresholds, they are passed to a Priority Selector. This module determines which of the qualified signals should be prioritized and selected based on predefined criteria. After a decision is made, the process concludes with the end step.

Fig. 5 is designed to ensure that only high-quality signals, as determined by the minimum loss operation and threshold check, are considered for final selection, and it resolves conflicts through a priority-based decision mechanism.

To ensure fair comparison between modalities with different units and dynamic ranges, all loss signals are normalized using Min-Max scaling. Each raw loss value represents the set of all three loss values (image, radar, and voice). This ensures all signals are brought to a uniform [0, 1] scale. The normalized values are then fed into a comparator module, implemented using fixed-point arithmetic logic, which selects the minimum among the three. The output of this comparator determines which modality is passed to the system’s downstream logic as the most reliable input at that time.

3.1. Simulation and Synthesis

For radar signal input, simulated returns were generated based on FMCW radar reflections through multi-layered materials resembling rubble. Voice data included synthetic distress calls mixed with ambient disaster noise (e.g., fire, wind). Image data was derived from thermal camera simulations with occlusions. Each modality was processed independently and prioritized based on loss performance.

This MATLAB code provides a straightforward example of how a Generative Adversarial Network (GAN) can be implemented to learn and mimic a one-dimensional Gaussian distribution with a mean of 3 and a standard deviation of 1. In this framework, two neural networks-the generator and the discriminator-are trained in opposition to one another. The generator is responsible for taking random noise as input and producing samples that aim to resemble the real data distribution. On the other hand, the discriminator’s task is to distinguish between real samples, which are drawn directly from the target Gaussian distribution, and fake samples, which are generated by the generator.

During the training process, the discriminator first learns to improve its ability to correctly classify real and synthetic data. It does this by analyzing both genuine samples from the Gaussian distribution and artificial samples produced by the generator, and then updating its parameters to better differentiate between the two. Subsequently, the generator is trained to produce more realistic samples by attempting to fool the discriminator. It receives feedback based on the discriminator’s performance and adjusts its own parameters accordingly, with the goal of generating data that the discriminator cannot easily distinguish from real data.

Table 1 compares the proposed architecture with existing multimodal fusion and prioritization approaches in terms of modality coverage, platform compatibility, real-time capability, hardware implementation, and signal prioritization support. As shown in the table, prior works such as ^{[2, 5, 13, 27]} primarily focus on software-based implementations using CPUs or GPUs, and do not provide real-time processing capabilities or hardware-level deployment. Moreover, none of these approaches support dynamic signal prioritization, which is essential in disaster environments where sensor reliability and relevance may vary rapidly. In contrast, our proposed architecture-based on a Hardware GAN combined with a Minimum Loss Selector-supports real-time operation and is implemented on FPGA with verified RTL logic. It also uniquely handles diverse modalities including image, radar, and voice signals, and incorporates a signal prioritization mechanism to enhance decision-making under uncertain and dynamic conditions. This hardware-based approach not only improves computational efficiency but also ensures robust performance in resource-constrained or time-critical environments.

Fig. 6 shows both the generator and the discriminator are implemented as single-layer linear models. The generator transforms random noise through a linear equation, while the discriminator applies a sigmoid function to its input to produce a probability that the data is real. During each training iteration, real data and noise are sampled in batches. The generator creates fake data from the noise, and the discriminator evaluates both real and fake samples to update its parameters using a loss function based on binary cross-entropy. The discriminator is trained to increase its ability to correctly classify real and fake data, while the generator is updated in a way that encourages it to produce outputs that the discriminator would classify as real. This adversarial process continues over many epochs, gradually improving the quality of the generator’s outputs.

Fig. 6. Generated data distribution.

At the end of training, the generator is used to produce a large number of samples, and a histogram is plotted to visualize their distribution. If training has been successful, the histogram of the generated data should closely resemble the shape of the original Gaussian distribution centered around 3. This simple example captures the essence of how GANs function, albeit in a highly reduced and illustrative form, without the complexity of deep neural networks typically used in real-world GAN applications. In a disaster environment, the main data, such as images, radar, and voice, were input, and a high-level simulation was performed on their weights through an artificial neural network.

Fig. 7. Simulation waveform output obtained from ModelSim.

Fig. 7 are designed by performing Verilog Coding for the hardware implementation, the design is verified using the ModelSim simulation environment. ModelSim offers comprehensive support for simulating digital circuits, allowing for thorough functional verification of the system. In this setup, the platform architecture incorporates a specialized hardware engine, which operates at a clock frequency of 50 MHz. During the simulation process, the hardware description files are compiled and executed within ModelSim, enabling designers to observe and assess the system’s functional behavior in detail. The simulation outputs, typically presented as waveforms, provide valuable insights into the timing and logic correctness of the design. By using this approach, it becomes possible to ensure that the hardware engine performs as intended under the specified clock conditions before proceeding to actual hardware fabrication or deployment. The proposed architecture was simulated and synthesized using Verilog RTL. The logic synthesis was performed and the timing was verified in the logic synthesis VivadoR environment. The logic synthesis results were LUT 2%, FF 2%, IO 13%, and BUFG 1%.

Table 2 presents the performance and resource utilization metrics of the proposed hardware architecture. The system operates at a clock frequency of 50 MHz, with a critical path delay of 8.2 ns, ensuring reliable timing performance. The design supports three sensing modalities-radar, voice, and image-with respective processing latencies of 320, 275, and 415 clock cycles. These values indicate the number of cycles required to process each data type within the architecture. In terms of hardware resource usage, the design is highly efficient, utilizing only 2% of both Look-Up Tables (LUTs) and Flip-Flops. Additionally, only 13% of available I/O pins and 1% of global clock buffers are used, indicating minimal footprint and scalability potential. The estimated power consumption of the system is 0.42 W, making it suitable for low-power applications such as real-time disaster response or embedded edge devices.