2.1 Instance Segmentation

Recently, networks that produce good instance segmentation performance are mainly object detection-based segmentation [8, 9, 20]. Many of these networks have developed a segmentation technique based on object detection of Faster R-CNN ^[6], which is a method that adds Region Proposal Network (RPN) to R-CNN ^[7] that performs object detection using a bounding box. In particular, Mask R-CNN ^[8] is a model that has been widely used as a comparison method and shows high performance among segmentation networks developed based on Faster R-CNN. Faster R-CNN has a class label branch and a bounding box offset branch, where Mask R-CNN is a network that adds an object mask branch. Therefore, segmentation networks that reduce the size of the model or speed up the computation using the Mask R-CNN backbone have been actively studied until now.

As another method of segmentation, there are networks of performing segmentation based on an embedding loss function ^[10,27]. These networks allow feature vectors of pixels belonging to the same instance to be similar to each other, and feature vectors of pixels belonging to different instances to be different. The fully convolution network is not suitable for the embedding method, so it shows worse performance than the detection-based network ^[11-15]. To solve the problem, methods have been proposed to increase segmentation performance by allocating pixels to point to the center of an object ^[16-19]. In addition, a method for adding clustering to the loss function and optimizing intersection over union has been proposed ^[27].

The proposed method is the same concept as ^[27] which is an embedding-based segmentation, and we have introduced additional knowledge distillation. Plants have a small instance size and many overlapping parts, so it does not fit to the detection-based segmentation, which necessarily has bounding box information, so we adopted embedding-based segmentation.

2.2 Knowledge Distillation

Knowledge distillation is a method of transferring knowledge from a large and heavy network, teacher network, to a light and small network, student network. In general, a teacher network extracts useful information such as feature information from input images and transfers it to a student network. Knowledge distillation has been widely used to improve performance in classification, segmentation and object detection ^[21-23].

Our network uses knowledge distillation to achieve good performance with even a small set of data, instead of lightening the model. The structure of the teacher network and the student network are the same, but the input image of each network is different. For training a teacher network, a large amount of plant dataset is used, and for training a student network, a small amount of plant dataset is used. In addition, two types of distillation methods are used: attention distillation ^[24] and region affinity distillation ^[21].

3.1 Dataset Acquisition

We grew butterheads in a plant grower chamber to directly acquire a plant dataset. 11 butterheads were grown for 18 days, and an IP camera was installed in the chamber to record butterhead growth. After cultivation was completed, we cropped the acquired images to generate a total of 198 RGB images. 175 images were used for train, and 23 images were used for test.

3.2 Dataset Processing

Instance labels are required for plant leaf instance segmentation. We labelled to distinguish between leaf instances. Using the annotation tool ^[25], we generated the labeled ground truth of the Butterhead dataset we processed. Each leaf instance was labeled in a different color, and rules were set so that the leaves that grew later would be on top if overlapping parts occurred.

Fig. 3. Butterheads taken with an IP camera.

Fig. 4. RGB Original Image and Labeling Results.

4.1 Leaf Instance Segmentation

We performed a spatial embedding-based instance segmentation, considering the properties of small and overlapping plant leaves. The backbone of our network is ERFNet ^[26], which is often used for segmentation. ERFNet consists of encoder and decoder structures, and factorized convolutions and residual connections are applied to help perform efficient segmentation. The decoder is divided into two branches: seed branch and instance branch. Seed branch generates a seed map, which has a value close to 1 as it is closer to the center of the instance to which each pixel belongs, and a value close to 0 as it moves away from the center. It is calculated using Gaussian functions.

The instance branch generates sigma map and pixel offset vector (x, y). Sigma map is related to the margin of each instance, and offset vector (x, y) represents the distance away from the center in the x and y directions. After that, with seed map and sigma map, each pixel is subjected to instance segmentation through clustering. Spatial embedding which points to the centroid of the instance is performed with each pixel and the corresponding offset vector. Using Gaussian functions, we compute the distance between spatial pixel embedding and instance center to obtain the probability of belonging to that instance. The spatial embedding loss is used as a segmentation loss.

4.2 Knowledge Distillation

A large amount of plant dataset was used to train the teacher network, and a small amount of plant dataset was used to train the student network. Unlike other general datasets, it is difficult to obtain plant datasets because they can only be obtained by growing plants in facilities with the same environment for a certain period of time. To solve this dataset shortage problem, we used knowledge distillation to achieve good segmentation performance even with a small amount of dataset. Since the purpose is not to make the model lighter, the structures of the teacher network and the student network are the same. Two types of distillation are used in the encoder and decoder of our backbone network ERFNet: attention distillation ^[24] and region affinity distillation ^[21]. In the encoder, attention distillation is used to transfer attention containing important information from the teacher network to the student network. We obtained the attention using the squeeze-and-excitation block of ^[24]. In the decoder, region affinity distillation ^[21] is used to transfer region contrast information from the teacher network to the student network.

The region contrast is obtained by calculating the cosine similarity between two classes. The two classes are multiplied by foreground and background of the plant binary mask, respectively. One class is multiplied by the feature map and the background of the plant binary mask. The other class is multiplied by the feature map and the foreground of the plant binary mask. The region contrast information of the teacher network is transferred to the student network. Two methods, attention distillation and region affinity distillation, are used for the distillation loss.

4.3 The Proposed Network Architecture

Fig. 5 is the overall structure of the proposed network. The top is the student network and the bottom is the teacher network. The encoder consists of 16 layers, the decoder consists of 8 layers each of the seed branch and the instance branch, and the decoder consists of 16 layers in total. The teacher network is trained with a large plant dataset with 750 images.

The sequence of network operations is as follows. First, we generate seed map, sigma map, and offset vector via ERFNet with a large dataset from the teacher network. Then, with the generated map, the leaf instance segmentation is performed using the spatial embedding. The teacher network is frozen after learning 1000 epochs. After the training of the teacher network is completed, the student network is trained. The student network works identically to the teacher network, but it can perform well with a small amount of plant dataset through distillation transferred from the teacher network. Unlike the teacher network, the student network learns with a total of three losses, two distillation losses as well as segmentation loss. An attention distillation is performed at the encoder portion, and a region affinity distillation is done at the decoder portion.

Fig. 5. The Proposed Network Architecture.

5.1 Experimental Setting

The proposed network is trained using several kinds of plant datasets. The image size of all datasets with different types is unified to 128 ${\times}$ 128. The network is implemented using the PyTorch framework on a PC with NVIDIA RTX 3090 GPU ^[28]. We adopted the Adam optimizer for loss optimization ^[29]. The initial learning rate is 0.00001.

We used four kinds of plant datasets in the experiment. Komatsuna ^[30] dataset is used to experiment with the teacher network. There are a total of 900 images in the Komatsuna dataset, 720 for train and 180 for test. For the student network experiment, three types of datasets are used: CVPPP 2017 A1, A4 ^[31], and the Butterhead dataset that we made. The CVPPP 2017 A1 dataset has a total of 128 images, 100 for train and 28 for test. The CVPPP 2017 A4 dataset has a total of 624 images: 459 for train and 165 for test. Finally, the Butterhead dataset has a total of 198 images, 175 for train and 23 for test. The dataset experimenting with the student network has fewer images than the dataset experimenting with the teacher network. We conducted many experiments by placing distillations at different locations in the network four types of distillations: feature distillation ^[32], attention distillation ^[24], prediction map distillation ^[21] and region affinity distillation ^[21].

5.2 Result on the CVPPP 2017 A1 Dataset

We evaluated the knowledge distillation and instance segmentation performance of the proposed student network using the A1 dataset of CVPPP 2017 ^[31], a leaf counting challenge dataset. Before training the student network, the teacher network was pre-trained after 1500 epochs learning with the Komatsuna dataset. In order to find out the effect of distillation, we conducted four experiments using four types of distillation. The distillation is performed on the encoder layer and decoder layer of the ERFNet. Table 1 shows the results of evaluation with mIoU, a quantitative evaluation metric. Fig. 10 shows the results of the experiment. Fig. 10(a) is a ground truth image. Fig. 10(b) is a result image when knowledge distillation is not used. Fig. 10(c) is a result image when using attention distillation in the encoder and region affinity distillation in the decoder and this is the result of the proposed method. Fig. 10(d) is a result image when using feature distillation in the encoder and region affinity distillation in the decoder. Fig. 10(e) is a result image that adds a prediction map distillation to Fig. 10(d). Fig. 10(f) is a result image that uses region affinity distillation in the encoder and attention distillation in the decoder, as opposed to the proposed method.

First, in the case of feature distillation in the encoder and region affinity distillation in the decoder, the mIoU increases compared to no knowledge distillation. But looking at Fig. 10(d), a small leaf instance cannot be segmented in the middle part. When precision map distillation is added in the previous case, the mIoU increases, and the leaf stems become a little clearer, as Fig. 10(e) shows. In the case of region affinity distillation in the encoder and attention distillation in the decoder, it can be seen that it has been segmented to the small leaf instance located in the center due to the influence of attention distillation as shown in Fig. 10(f). The proposed method is to change the position of distillation in the previous case. All the leaves are well segmented, and stems and borders are more distinct as shown in Fig. 10(c).

Fig. 6. Komatsuna Dataset[30].

Fig. 7. CVPPP 2017 A1 Dataset[31].

Fig. 8. CVPPP 2017 A4 Dataset[31].

Fig. 9. Butterhead Dataset.

5.3 Result on the CVPPP 2017 A4 Dataset

Also, we evaluated the knowledge distillation and instance segmentation performance of the proposed student network using the A4 dataset of CVPPP 2017 ^[31], a leaf counting challenge dataset. The experimental evaluation method is the same as in 5.2. Table 2 shows the results of evaluation with mIoU, a quantitative evaluation metric. Fig. 11 shows the results of the experiment. The resulting image sequence is the same as in section 5.2. The CVPPP 2017 A4 dataset is very similar except for the background of the CVPPP 2017 A1 dataset, showing similar results.

Fig. 10. Segmentation result comparison on CVPPP 2017 A1 dataset[31]: (a) is the ground truth; (b) is without knowledge distillation; (c) is attention distillation in the encoder and region affinity distillation in the decoder (proposed); (d) is feature distillation in the encoder and region affinity distillation in the decoder; (e) is prediction map distillation and feature distillation in the encoder and region affinity distillation in the decoder; (f) is region affinity distillation in the encoder and attention distillation in the decoder.

Fig. 11. Segmentation result comparison on CVPPP 2017 A4 dataset[31]: (a)-(f) are the same as the detailed description of Fig. 10.

Fig. 12. Segmentation result comparison on Butterhead dataset: (a)-(f) are the same as the detailed description of Fig. 10.

5.4 Result on the Proposed Butterhead Dataset

Finally, we evaluated the knowledge distillation and instance segmentation performance of the proposed student network using the Butterhead dataset. The experimental evaluation method is the same as in 5.2. Table 3 shows the results of evaluation with mIoU, a quantitative evaluation metric. Fig. 12 shows the results of the experiment. The resulting image sequence is the same as in 5.2.

Unlike the CVPPP 2017 dataset, the Butterhead dataset is one produced by labeling ourselves, so the experimental results are slightly different. Butterhead dataset has no stems, and there are many overlapping leaves, so their performance is not good in these overlapping areas. But still, the performance in the proposed method with attention distillation in the encoder and region affinity distillation in the decoder is the best. Our next research goal is to design a network that can perform better for overlapping leaves.