2.1. Image Inpainting Models

In Computer vision, inpainting has been studied by many researchers from the past to the present. The field of inpainting can be broadly divided into two parts: image inpainting and video inpainting. However, since this article is about image classification, we will focus on image inpainting rather than video inpainting. Image inpainting based on deep neural networks learns semantic priors and meaningful hidden representations in an end-to-end manner, and the network uses a convolutional fitter to fill the missing content with a fixed value. This approach relies on initial hole values, which can lead to problems such as a lack of texture in the holes, severe color contrast, artificial edges, and artificial edges. Another limitation is that they focus on rectangular-shaped holes in the center of the image, which leads to overfitting of the holes.

To address these limitations, Liu et al. ^[4] propose a partial convolution with an automatic mask update step rather than a traditional convolution, which allows the convolution to be applied only to valid pixels, and proposes a method to automatically generate an updated mask when moving from the current layer to the next layer as part of a partial convolution forward pass. The goal of the paper is thus to propose a predictive model that works on irregular hole patterns and integrates with the rest of the image without additional post-processing. However, Liu et al. ^[4] note that there are limitations: first, the classification is done empirically through validity and invalidity for all spatial locations, which means that the mask is set regardless of how many pixels are covered by the filter. Second, as you progress through the layers, there are fewer and fewer invalid pixels, and all mask values are converted to 1. This process has the limitation that it increases the probability of assigning a valid mask to an unfilled pixel.

In a limitation of Liu et al. ^[4], Yu et al. ^[5] show that partial convolution is hard-gated, which can limit flexibility because all channels in each layer share the same mask. Yu et al. ^[5] also named partial convolution ^[4] as unlearnable single-channel feature hard gating. Instead of the hard gating mask of partial convolution ^[4], gated convolution ^[5] automatically learns a soft mask from the data, paying more attention to features that are more likely to be valid features. This is called learning a channel-by-channel, spatial dynamic feature selection mechanism, and this process proposes a new convolution, the gated convolution. In addition, a spectral-normalized markovian discriminator (SN-PatchGAN) is devised to naturally fill a large number of holes with arbitrary locations and arbitrary shapes. Here, SN-PatchGAN is devised by focusing on the deep Markovian model of Li et al ^[6], which only uses the statistics of local patches, thus enabling image inpainting and user-guided inpainting. In the overall structure, the coarse network and refinement network use an encoder-decoder type network instead of the U-Net ^[7] used in partial convolution ^[4] because the skip connection of the U-Net ^[7] cannot propagate detailed color or texture information.

Recent research has attempted to enhance image inpainting. These methods ^[27-29] focus on improving the quality of image inpainting and, as a result, producing more natural-looking images.

Chen, Yuantao et al. ^[27] proposes an image inpainting method, multi-scale feature module with attention module (MFMAM), which combines a multi-scale feature module with an attention module. Existing deep learning-based image inpainting algorithms deal with the problem of information loss during the process of extracting deep-level features. The proposed network consists of a two-level structure, using a deep learning encoding-decoding network and a generative adversarial neural network (GAN) structure. The multi-scale fusion module using dilated convolution (MFDC) module reduces the information loss during the feature extraction process, and the joint attention module enhances the restoration of the semantic structure of the network. The approach of Chen, Yuantao et al. ^[27] uses a conditional semantic attention (CSA) layer and a pixel-by-pixel class activation mapping (CAM) layer to restore deep semantic features of an image and performs pixel-level restoration on shallow features. This effectively restores information in corrupted areas. Using the model trained on the Outdoor Scene ^[30] and CelebA-HQ ^[31] datasets, they performed an object removal task in real-world scenes and showed that the proposed method can well predict the structure of the corrupted image and produce more realistic and clearer restoration results. The proposed MFMAM network improves the performance of image inpainting, demonstrating the effectiveness of the attention module in different architectures. Chen, Yuantao et al. ^[27] showed that the method with the addition of style loss and perceptual loss is particularly effective.

Corneanu, C et al. ^[28] proposes a new information propagation mechanism called LatentPaint, which conditions existing diffusion models to perform image inpainting tasks. Existing inpainting methods primarily propagate information based on patch similarity or use generative adversarial networks (GAN) to generate images, often suffering from high computational cost, complex training, and inconsistent quality of results. LatentPaint, the approach of Corneanu, C et al. ^[28] works by training a neural network to predict the original image from a noisy image. In doing so, they introduce an Explicit Propagation (EP) module that propagates information from the conditional pixels to the pixels to be inferred, which they combine with a traditional diffusion model (DM) to perform inpainting. Corneanu, C et al. ^[28] evaluated LatentPaint’s performance on several visual domains, including the CelebA-HQ ^[31] dataset, and found that it outperforms existing state-of-the-art techniques, especially in producing images with fast runtimes and high quality. LatentPaint opens up new possibilities in the field of image inpainting, demonstrating an innovative way to deliver high-quality results with an efficient information propagation mechanism.

Xie, S et al. ^[29] proposes SmartBrush, a diffusion-based model that fills damaged areas with objects using text and shape guides. They address the problem that the existing DALLE-2, Stable Duffusion, only supports text-based inpainting but not shape guides, and as a result tends to modify the background texture around the object. The approach proposed in this paper is to perform multi-modal object inpainting through a new diffusion-based model called SmartBrush, which focuses on improving the quality and control of inpainted objects by simultaneously utilizing text describing the properties of the object that the user wants and a mask defining the shape of the area to be inpainted. SmartBrush is based on a diffusion model, which produces high-quality images by gradually removing noise in the image. During this process, the model performs inpainting by considering the input text and shape mask, and it also predicts the foreground object mask to help preserve background pixels around the inpainted object in order to preserve the background of the original image. In addition, SmartBrush adopts a multi-task training strategy that simultaneously trains on object inpainting and text-to-image generation tasks to improve its ability to handle different text descriptions and image content. Training data is taken from large datasets such as LAION-Aesthetics v2 ^[32], and the model’s performance has been evaluated through user studies on Amazon Mechanical Turk. This approach enables SmartBrush to produce higher quality results than traditional inpainting models, and to more accurately reflect the shapes and attributes desired by users.

2.2. Text-to-image Models

Hyper-scale artificial intelligence technologies are gaining worldwide attention, with text-to-image models like DALL-E 2 ^[1], Imagen ^[2], and Stable Diffusion ^[3], which generate images from input text, are under continuous improvement and research. For example, the contrastive models such as contrastive language-image pretraining (CLIP) ^[8] are known to learn various representations of images well. Ramesh et al. ^[1] propose a two-stage model to leverage representations for image generation. The two-stage model consists of a prior and a decoder. The prior generates a CLIP image embedding from the text, and the decoder takes the image embedding from the prior as a condition and generates an image with a diffusion model ^[9].

In addition, the diffusion model ^[9] can be used as a decoder to learn to restore the CLIP image encoder, and so on, to output semantically similar images, such as generative adversarial networks inversion (GAN inversion) ^[10,11], and to enable image interpolation through image embedding. Therefore, Ramesh et al. propose the unCLIP model ^[1] which finds a text embedding, given a text, that matches the image embedding present in CLIP ^[8] and converts the text embedding into a CLIP ^[8] image using a prior model. Given an image embedding, the next step is to decode it with a diffusion model ^[9] to generate a range of images that match the image embedding.

Another text-to-image model, Imagen ^[2], which focuses on improving the match between text and images, uses a large transformer language model. This model, T5-XXL ^[12], is a pre-trained language model with frozen weights that is known for its ability to understand and generate human language. To map text to a sequence of embeddings, Imagen combined this large transformer language model with an encoder and a diffusion model ^[9,13-15]. The diffusion model generates 64x64 images, 256x256, and 1024x1024 super-resolution images. Saharia et al. ^[2] emphasize that, unlike previous works that use only image-text data for model training, the use of this large transformer language model ^[12] pre-trained on text only is very effective for image-text matching, and the problem of classifier-free guidance, a new diffusion sampling technique, is solved by dynamic thresholding to improve the quality of images.

Stable Diffusion ^[3] models represent a significant advancement in text-to-image generation. Unlike its predecessors, Stable Diffusion ^[3] utilizes diffusion methodologies and latent space representation to generate lifelike images from textual and visual prompts while substantially reducing computational requirements. The training process involves encoding text inputs into latent vectors using pre-trained language models like CLIP. This encoding facilitates the generation of compressed representations of images, thus alleviating the challenges associated with high-resolution image generation. The model comprises three main components: a variational autoencoder (VAE) for transforming images into latent representations, a U-Net ^[7] for denoising noisy latents, and a text-encoder for converting input prompts into embeddings that guide the denoising process. Stable Diffusion ^[3] enables various creative applications, including text-to-image generation, image upscaling, and inpainting, while also democratizing high-resolution image synthesis by reducing the cost of training and inference.

2.3. Detecting Generated Images

Much research is being done on the classification of generated and real images, so it is becoming essential to study the classification of generated and real images. In a study of image classification of real and generated images, Bird, Jordan J. et al. ^[22] used CIFAR-10 ^[23] as the real dataset and Stable Diffusion 1.4 ^[3] as the generated dataset to generate images based on 10 CIFAR-10 ^[23] classes. Then, they trained an image classification model using this image data. The trained image classification model was trained with a simple CNN model. The trained image classification model experimented with different hyperparameter tuning, such as the number of layers, filters, neurons, etc., and found the best model experimentally by measuring precision, recall, F1-score, etc. The best model achieved an accuracy of 92.98%.

Other research in generated and real image classification combines transfer learning with specific algorithms to improve classification accuracy despite the small number of generated datasets. Mittal, Himanshu et al. ^[24] use the real and fake face detection by Yonsei University ^[25] to train a model for the classification of real and generated face images. Due to the small dataset, they trained the model based on pre-trained AlexNet ^[26]. Finally, they propose a model called improved quantum-inspired evolutionary-based feature selection (IQIEA-FS). IQIEA-FS is a feature selection method developed based on a quantum-inspired evolutionary algorithm’s basic concepts and ideas (QIEA). QIEA leverages concepts from quantum computing to solve optimization problems. IQIEA-FS is based on this QIEA algorithm but presents an improved method for more efficient and accurate feature selection to improve its performance. IQIEA-FS is an evolution of QIEA, which provides better feature selection and classification performance while maintaining its basic principles. Furthermore, it is used with a k-nearest neighbor (KNN) classifier to classify images into real and fake faces. Thus, the IQIEA-FS method consists of feature extraction based on AlexNet, feature selection using the IQIEA algorithm, and image classification using the KNN classifier, and achieves a mean normalized accuracy (MNA) of 58.3%.

5.1. Dataset and Augmentation

We used MS-COCO ^[16] 2017 dataset to collect real images. For training, 10,000 images were sampled for the training dataset and 1,900 images for the validation dataset, both drawn from MS-COCO ^[16] train. Also, 1,500 test images for evaluation were collected from MS-COCO ^[16] valid. Stable Diffusion is recently used to generate high quality images with text descriptions ^[3]. We generated 768 * 768 images via the Stable Diffusion ^[3] 2.0 model, matching the number and ratio of the natural dataset, excluding its training dataset. We used the captions from MS-COCO ^[16] annotations as text descriptions to synthesize images.

We implemented the algorithm for data augmentation in our generated dataset. The pseudo-code in Algorithm 1 shows the specific generated image augmentation process. Specifically, on the 7,000 images generated dataset for training, we randomly applied augmentation techniques to produce a total of 3,000 images.

Algorithm 1: Pseudo-code for data augmentation.

5.2. Real and Generated Images: Comparative and Quantitative Analysis

We additionally applied inpainting to each real and generated dataset This algorithm was chosen for its everyday use in the field and its ability to provide a baseline for comparison. We then and conducted a quantitative evaluation to increase the credibility of the poor inpainting performance in the generated image among the real and generated images. We applied 10 images for each dataset, for a total of 80 images inpainted. In Fig. 6, the real image dataset, MS-COCO [16], shows that the masked objects are well erased, while the generated image datasets, DALLE-2 [1], Imagen [2], and Stabe Diffusion [3], show unnatural results where the masked objects are not well erased.

Fig. 6. Examples after applying inpainting to real and generated image datasets.

We performed a quantitative evaluation to provide an objective assessment of the results. The metrics we used for the quantitative evaluation are the frechet inception distance (FID) score and naturalness image quality evaluator (NIQE). The FID score is used to measure the quality difference between the original image and the generated image, while NIQE is a quantitative measure of the naturalness of the image. In other words, we used FID to measure the quality difference between the original image and the image with objects removed by inpainting and NIQE to measure the naturalness of the image with objects removed by inpainting. In Table 2, which shows the quantitative figures, we can objectively see that the generated images have higher values on average than the real images, so we can say that the generated images are not inpainting better than the real images. In particular, the average FID score shows a significant difference. Therefore, we have shown the difference between the real and generated images through experiments and proved that the generated images are not inpainting better through quantitative evaluation.

5.3. Result

The image size was set to 384 to match the input size of ConvNeXt-XL ^[17]. We used a batch size of 4 and 200 epochs, with a training time of 60 minutes per epoch. To minimize unnecessary training time and prevent overfitting, we applied an early stopping technique to identify the best performance networks quickly. In conclusion, a 99.87% classification accuracy was shown by the trained proposed model on the test dataset. Fig. 6 shows a visualization of the predictions made by the trained model on the test dataset. It can be verified that the model correctly classifies the generated and real images for the test dataset.

7.1. Limitation and Future Research

The model trained on the Stable Diffusion ^[3] dataset demonstrated good performance; however, it struggled to accurately classify artworks, illustrations, photographs, and other content generated by DALLE-2 ^[1], indicating the need for more diverse generated images to be included in the training dataset to improve classification of generated images. In addition, although we obtained objective performance results through quantitative evaluation, we also needed to conduct subjective evaluation to understand the subtle differences and context that are difficult to describe in numbers. However, due to the time constraints of the research schedule and experimental period, we were unable to recruit enough subjects, which limited the subjective evaluation. In addition, the model we learned was limited by high computational cost and processing time, which makes it difficult to apply to real-world applications.

In our future work, we will focus on lightweighting and optimizing the model to improve computational cost and inference speed to compensate for these limitations. We also plan to obtain more generated data for training and recruit enough subjects to make the subjective evaluation more effective. Finally, we will continue our research by validating the performance of the model in different environments to increase its real-world applicability.