2.1. Deep Neural Network (DNN)

The DNN is a specific type of ANN comprising multiple hidden layers, as illustrated in Fig. 1. It has garnered significant attention in tandem with the advancements in machine learning models. DNNs excel in effectively learning complex nonlinear patterns, resulting in high performance across diverse domains.

The input layer receives data, hidden layers extract complex patterns, and the output layer produces results. The input layer’s neuron count matches data characteristics. Hidden layers vary in number and complexity. The output layer’s neuron count aligns with network goals. Activation functions add nonlinearity, vital for learning intricate patterns. Choices impact neuron outputs, each function having pros and cons. Common types include sigmoid, hyperbolic tangent, rectified linear unit (ReLU), and leaky ReLU.

Forward propagation in neural networks involves multiplying input data (e.g., x1) by corresponding weights (e.g., wi1), summing the results, and passing them through an activation function to generate predictions as shown in Fig. 2. Backpropagation is crucial for error reduction, adjusting weights based on output layer errors using methods like gradient descent. This iterative process refines weights, minimizing overall error and enhancing network performance. Gradient descent optimizes weight updates based on loss function gradients but faces challenges like local minima and efficiency in learning speed.

2.2. Convolutional Neural Network (CNN)

CNNs excel in image recognition and processing by learning regional patterns and analyzing spatial structures through convolution and pooling operations. The convolution layer applies filters (e.g., 3×3 or 5×5) to the input data to create a feature map. These filters, smaller than the input data, are adjustable and are used to detect local patterns in the image. In the convolution operation, the filter moves across the input image, multiplying and summing overlapping data regions to produce results. This output then feeds into the activation function, commonly ReLU in CNNs. ReLU outputs zero for inputs less than zero and the input value itself for non-negative inputs.

The pooling layer processes data from convolutional operations and activation functions to reduce the feature map size. This reduction decreases computational complexity, speeds up model convergence, and enhances robustness against overfitting. Additionally, the pooling layer maintains recognition of patterns despite shifts in their positions within the feature map, making it robust to spatial variations in the image.

Two common types of pooling in CNNs are max-pooling and average pooling. Max-pooling extracts the maximum value from a specified region (e.g., 2×2, 3×3) of the input feature map, capturing significant features and reducing image size while preserving important information. Average pooling, on the other hand, takes the average value from a region, retaining smaller-scale features and being more robust to noise.

The pooling layer has key parameters: pooling size and stride. The pooling size determines the region’s dimensions, while the stride indicates the step size of the pooling operation. Proper tuning of these parameters helps control the output size, retain relevant information, and manage model complexity. Pooling reduces computational load, increases learning rates, and mitigates overfitting by reducing dimensionality. However, it may result in some information loss and decreased accuracy, and its fixed size can limit applicability to various image sizes.

2.3. Binary Neural Network (BNN)

CNNs demand substantial computation and memory, making them unsuitable for low-end devices or energy constrained environments. To address this, researchers have focused on binary neural networks (BNNs). BNNs reduce computational and memory requirements by binarizing weights and activation values into +1 or -1 using a sign function. This binarization allows efficient processing with bitwise operations like XOR, AND, and bit shifts, replacing traditional multiplication and addition.

BNNs operate similarly to conventional CNNs but use binarized weights and activation values, enabling faster hardware computations. They include convolutional, pooling, and fully connected layers, with computations performed on binarized values. BNN learning relies on backpropagation but requires alternative binarization functions and approximation methods, such as the straight-through estimator, for weight updates. Standard optimization techniques like SGD, Momentum, AdaGrad, and Adam can be used.

BNNs achieve computational efficiency by binarizing weights and activation values, but this can reduce accuracy due to information loss and limited network expressiveness. Applying standard learning algorithms directly to BNNs is challenging, requiring modifications to handle binarized values. Therefore, finding suitable optimization methods is crucial to address these issues and improve the learning process.

3.1. Pipelining and Loop-Flattening

Fig. 4 provides an overview of the Vivado HLS software. Vivado HLS allows for verification through register transfer level (RTL) coding and simulations using high-level languages such as C, C++, and SystemC. Once the verification is complete, the design is generated in an RTL language (either Verilog hardware description language (HDL) or very high-speed integrated circuit HDL (VHDL)) and is transformed into an intellectual property (IP) core.

Fig. 4(b) shows the flow of HLS. It is similar to the traditional RTL design flow but includes an additional HLS coding flow. The advantages of HLS include a significant reduction in design time and cost-effectiveness. Compared to the relatively high entry barrier to using Verilog HDL, HLS using C-based optimization methods allows for a faster design implementation. It also helps reduce the design time in critical areas such as synthesis, place and routing (P&R), verification, and timing closure.

Overall, Vivado HLS offers a more efficient and streamlined design flow, allowing designers to achieve faster and more cost-effective design implementations than those from traditional RTL design methods.

Insofar as the coding time reduction, a significant difference in the code content can exist between designing in Verilog and in Vivado HLS. For example, if we consider designing the pipelining, Vivado HLS allows for a simple implementation with just one line by using macros. In contrast, in Verilog HDL, the entire design needs to be manually coded, resulting in a substantial amount of code content. The reduced coding effort in Vivado HLS not only reduces the design time but also decreases the amount of code that needs to be debugged. Consequently, within the same time frame, Vivado HLS enables the design of a wider range of architectures compared to Verilog HDL.

Vivado HLS offers higher-level abstractions and optimization techniques that simplify the design process and reduce the required amount of manual coding. By utilizing macros and built-in libraries, designers can express their intentions in a more concise and efficient manner. This leads to faster design exploration and parameter tuning, as well as reduced debugging efforts.

Therefore, Vivado HLS provides a more productive and streamlined development experience, allowing designers to achieve a broader range of architectural designs within the same time constraints compared to traditional Verilog HDL coding.

Achieving timing closure, especially when working with high-frequency clocks, can be challenging. However, Vivado HLS addresses this issue by incorporating library information and target frequency details during the generation of the Verilog code from the C description. This allows for the creation of RTL code able to operate at the desired timing for high-frequency designs.

By providing library information, Vivado HLS ensures that the generated RTL code leverages optimized and pre-characterized components capable of meeting the specified timing requirements. Additionally, the target frequency information helps guide the HLS optimization process to achieve the desired performance.

However, Vivado HLS has limitations. One of the main drawbacks is that the efficiency of the conversion from C to Verilog may not be optimal. When designers manually write Verilog code, they have more control over the hardware resources and can optimize the design to make more efficient use of those resources. However, when using Vivado HLS, the efficiency of the generated Verilog code depends on the designer’s proficiency and quality of the HLS tool optimizations.

As Vivado HLS performs automated transformations and optimizations based on the C description, it may generate code that utilizes more hardware resources than necessary. This can lead to suboptimal resource utilization and potentially affect performance or resource constraints. Designers may need to manually optimize the generated Verilog code or fine-tune the directives of the HLS tool to achieve the desired efficiency.

Overall, the trade-off between efficiency and productivity is a consideration when using Vivado HLS, and designers need to carefully evaluate their requirements, constraints, and design goals to determine the most suitable approach for their specific project.

Loop-flattening and pipelining can be easily implemented using the Xilinx Vivado HLS tool with the provided commands. Fig. 5 illustrates the concepts of loop-flattening and pipelining. Pipelining reduces the latency by overlapping the execution of multiple operations, but at the cost of increased hardware resource utilization. In contrast, loop parallelization can save hardware resources but may result in increased latency compared to pipelining.

To optimize the hardware resource usage, an analysis of the hardware resource utilization is conducted for each layer, as shown in Fig. 6 and Table 1. Based on this analysis, optimization techniques such as loop parallelization or pipelining can be selectively applied to the layer that utilizes the most hardware resources. This allows for a more efficient utilization of the available hardware resources while considering the trade-off between latency and resource usage.

By carefully analyzing the resource requirements of each layer and applying the appropriate optimization techniques, designers can achieve an optimal balance between hardware resource utilization and performance in their design.

The first convolutional layer in the BNN design performs a preprocessing step of converting the input image into binary values (0 and 1). As a result, it tends to have a higher memory usage compared to the second convolutional layer. Additionally, convolutional layers generally account for a significant portion of the overall computations in a BNN design.

To optimize the computational and memory requirements, pipeline and loop parallelization techniques are applied to the convolutional layer with the highest computational and memory demands. In contrast, for the pooling layer (which typically involves fewer computations), only loop parallelization is applied. By selectively applying these optimization techniques, a balance can be made between the accuracy and hardware resource utilization when considering the convolutional layer with higher computational demands and pooling layer with relatively lower computational demands.

Table 2 shows the accuracy and hardware resource utilization when applying only pipeline optimization to the computation-intensive parts of the existing BNN. The table shows the utilization of hardware resources such as the Block RAM (BRAM) and flip-flops. It is observed that these resources are heavily utilized. In contrast, there is a slight reduction in the utilization of look-up tables and DSPs; the latency decreases by 12%.

This indicates that applying pipeline optimization better reduces the latency of the computations while slightly reducing the utilization of certain hardware resources. The trade-off between accuracy and resource usage is an important consideration in BNN optimization. The results from Table 2 provide insights into the impact of pipeline optimization on the overall performance and resource utilization of the BNN.

Table 3 illustrates the accuracy and hardware resource utilization when both loop parallelization and pipeline optimization are applied to the computation-intensive convolutional layers, whereas only loop parallelization is applied to the relatively less compute-intensive pooling layers. When comparing this approach to that applying only pipeline optimization, the hardware resource utilization remains similar, but an additional 16% reduction in latency appears. Overall, this results in a total reduction of 29% in latency compared to that of the original BNN design.

This indicates that by combining both loop parallelization and pipeline optimization in the convolutional layers and applying loop parallelization in the pooling layers, the latency can be further reduced while maintaining a hardware resource utilization comparable to that of the pipeline optimization-only approach.

Fig. 7 shows the block diagram of the BNN design created using Vivado HLS and implemented as a Verilog language IP in the Vivado tool. It is designed while targeting the Xilinx Zynq-7000 FPGA. The bnn_0 IP is responsible for performing all of the operations of the BNN. The BRAMs are used for storing values such as input images, weights, biases, and other necessary data.

Table 4 provides the power consumption results when performing the hardware implementation. It shows that applying pipelining significantly increases the usage of hardware resources, leading to increased power consumption. “Processing System 7” (PS7) consumes the highest amount of power; it represents the power consumption related to the FPGA. In the SoC implementation, the power consumption of the PS7 contributes significantly to the overall power consumption, resulting in a 1%, 3% increase compared to the baseline. However, the pipeline and loop-flattening optimizations result in significant decreases in latency (by 12% and 29%, respectively), highlighting their advantages in terms of the reduced delay.

3.2. Pop-Count Operation

The pop-count operation is an operation used in digital circuit design. Traditional multiply-and-accumulate operations involve the use of adders and multipliers and consume a significant amount of hardware resources. The pop-count operation counts the number of 1s in a binary representation of a value. This operation can be implemented easily and allows for faster computation with reduced hardware resources.

In BNNs, both the weights and activation values are binarized. The pop-count operation is commonly used for performing inner product operations between two binary vectors. For example, consider two vectors A = [1, 1, −1, −1] and B = [1, −1, −1, 1]. After performing an XOR operation, A⊕B = [1, 0, 1, 0] is obtained. Subsequently, applying the pop-count operation to this result yields popcount (1010) = 4 (length of the array) −2 (number of 1s) = 2. In contrast, the dot product of the two vectors A · B = (1 × 1) + (1 × −1) + (−1 × −1) + (−1 × 1) = 0. The pop-count operation is used in BNNs primarily to prioritize computation speed rather than accuracy, thereby leveraging the characteristics of BNNs.

3.3. Proposed Low-Power Accumulator

BNNs are neural networks with weights and activation values limited to +1 or −1, allowing for more efficient learning and inference compared to ANNs. Whereas ANNs mainly use multiplication and addition operations, BNNs employ bitwise operations such as XNOR, providing significant advantages in memory usage. However, as the amount of learning increases, more hardware resources are required. Correspondingly, many studies have addressed this issue. One of the proposed solutions involves using accumulators to add the results of previous operations to those of the current operation in each layer; however, this approach consumes a significant amount of hardware resources.

Fig. 8 depicts a general accumulator for accumulating and storing calculation results. In the case of two operands involved in a computation, one operand is stored in the accumulator register, whereas the other is fetched from memory or another register for the operation. In BNNs, the accumulator is responsible for accumulating the results of the binary dot product operations. When calculating the neuron output values in each layer, the accumulator accumulates the binary dot product of the previous layer’s output values and weights, then outputs them as the current layer’s output values. These output values are then fed as inputs to the next layer.

The accumulator operation method in BNNs reduces the computational complexity and memory usage compared to those in other neural networks. As a result, BNNs can be effectively utilized in low-power and resource-limited hardware environments. However, the drawback is the reduced accuracy, as the representation and computation are limited to 1 and −1 (unlike in other neural networks).

Fig. 9 shows the operation process of the BNN applied in this study. Initially, the input data and weights undergo bitwise operations, followed by pop-count operations, before entering the accumulator as inputs. The corresponding formula can be expressed as follows:

The accumulator adds the previous values and, upon completing all bit operations, ultimately outputs the final result. The pop-count operation used to identify the number of 1s; it is frequently employed in BNNs for computational optimization, as they predominantly involve bit operations.

BNNs primarily focus on the computational speed and hardware resource usage rather than the accuracy. Therefore, it is not necessary to accurately compute the cumulative operations at each layer. It is more efficient to maintain a level of accuracy sufficient for recognition while reducing the power consumption and hardware resource usage.

Not all bits have an equal impact on recognition during cumulative operations, so it is inefficient to perform cumulative operations for bits with a relatively lower impact. Consequently, the accumulator does not perform cumulative operations for bits with less impact on the accuracy, and instead directly outputs the previous value. Fig. 10 shows the accuracy when cumulative operations are not performed for each bit. The 32nd bit exhibits a high accuracy of 94.7%. In contrast, the 6th bit shows a much lower accuracy of 68.8%. This confirms that the 6th bit has a more significant influence on the final classification than that of the 32nd bit. Based on these results, a low-power accumulator is proposed herein.

Fig. 11 illustrates the proposed low-power accumulator. Conventional accumulators sequentially compute each bit, thereby consuming significant amounts of computational time and hardware resources. In contrast, the proposed accumulator performs parallel operations while adding an enable signal to each adder. If the enable signal is ‘1,’ the operation proceeds as usual; however, if the enable signal is ‘0,’ the operation does not take place. By conducting parallel operations with the same number of adders as before, the computational speed increases, and the addition of the enable signal to each adder significantly reduces the hardware resource usage.

3.4. Proposed Accumulator-Based BNN

Fig. 3 shows the overall architecture of the BNN with the proposed low-power accumulator, as designed using the MNIST dataset. The MNIST dataset consists of handwritten ten digit images, each with a size of 28 × 28 pixels. When the input data is received, the pixel values of the image are converted to 0 or 1. The preprocessed image data is then fed into the input layer. The data output from the input layer serves as the input for the convolution layer, which extracts image features using 32 filters. After the convolution operation is completed, max-pooling is performed.

Then, 64 filters are applied again, and pooling is performed once more before the data is converted into a one-dimensional format. The one-dimensional data passes through a fully connected layer to ultimately perform the classification.

The image data is converted to 0 or 1 through the preprocessing process, and other data such as weights and biases are also binarized. Therefore, most operations are performed using XOR or bitwise operators.