2.1. Visual Backbones for Semantic Segmentation

Semantic segmentation has evolved significantly with the development of high-capacity visual backbones. Transformer-based models such as ViT-Adapter ^[1], SegFormer ^[16], and Segment Anything ^[5] leverage self-attention to capture global dependencies, resulting in strong performance on dense prediction tasks. However, their high computational demands often limit their applicability in resource-constrained environments.

To address this, recent works have explored efficient CNN-based architectures that retain strong spatial representation capabilities ^{[3, 11]}. ConvNeXt ^[9] revisits ResNet-style backbones using large kernel convolutions and simplified normalization strategies. InternImage ^[14] adopts deformable convolutions (DCNv3) throughout the network, enabling adaptive receptive fields and fine-grained spatial modeling. InceptionNeXt ^[17] integrates multi-branch convolutions that split input channels into parallel paths with different kernel sizes, enabling efficient multi-scale feature extraction while reducing computational cost. While these backbones offer a good balance between accuracy and efficiency, their reliance on purely visual features makes them vulnerable to errors in semantically ambiguous regions, such as areas that appear similar but differ in meaning. Since the CLIP image encoder focuses on global semantics, we adopt InceptionNeXt as a visual backbone and incorporate the CLIP text encoder to provide complementary context.

2.2. Prompt-Based Vision-Language Segmentation

CLIP ^[12] demonstrates the effectiveness of language supervision in guiding visual representation. However, CLIP relies on static and class-level prompts, which limits its adaptability to diverse visual contexts. To address this, CoOp ^[19] introduces learnable prompts that adapt to downstream tasks, and CoCoOp ^[18] further conditions prompts on image-level features to enhance generalization.

For dense prediction, several methods such as ClipSeg ^[10], LSeg ^[6], and DenseCLIP ^[13] extend CLIP by computing similarity maps between image and text features. ClipSeg directly compares pixel-wise embeddings, and LSeg further refines this process by aligning pixel-level features using contrastive learning. DenseCLIP ^[13] builds upon CLIP by enabling end-to-end training for semantic segmentation. It introduces a learnable prompt mechanism in the text encoder, aligning it with segmentation objectives rather than relying on fixed textual inputs. Additionally, a segmentation head is integrated into the visual encoder to produce dense predictions. This design allows DenseCLIP to better adapt textual representations to the segmentation task, but it still employs class-level prompts, which limit its ability to capture instance-specific details such as fine boundaries.

2.3. Multi-modal Feature Fusion

Multi-modal fusion is a central component in vision-language models, and is commonly implemented via transformer-based cross-attention. ViLT ^[4] removes modality-specific encoders and directly feeds image patch embeddings and tokenized text into a shared transformer, enabling end-to-end multi-modal learning with reduced architectural complexity. ALBEF ^[8] employs a dual-stream architecture that separately encodes visual and textual inputs before aligning them using a cross-modal attention mechanism. It also introduces contrastive learning with a momentum encoder to stabilize representation alignment between modalities. BLIP ^[7] extends this idea by combining vision-language pretraining with image-grounded generation tasks, further enhancing semantic grounding through both discriminative and generative objectives. However, positional encoding alone is insufficient to preserve local visual structures, and it limits transformers in spatially detailed tasks.

3.1. Wavelet-Aware Context Construction

Previous prompt-based segmentation approaches ^{[13] [10]} have relied on static, class-specific prompts shared across all input images. Although using fixed prompts reduces implementation complexity, it lacks the adaptability to reflect diverse visual contexts across scenes. To overcome this limitation, we propose a dynamic context construction mechanism that constructs textual contexts for each class based on the input image, thereby enabling class representations to adapt to the visual variations present in different scenes.

To construct textual contexts that are both semantically rich and sensitive to object boundaries, we introduce the Wavelet-Aware Context Construction (WACC) module. This module applies the Discrete Wavelet Transform (DWT) to the input visual features, producing four frequency subbands: LL, LH, HL, and HH. These components are grouped and processed along three specialized paths to capture different aspects of visual information. The LL subband captures low-frequency information that reflects the global structure and coarse semantic layout, providing essential context for understanding overall image composition. The LH and HL subbands capture horizontal and vertical high-frequency components, respectively, highlighting directional edges and structural boundaries essential for representing object shapes. The HH subband contains high frequency components in both horizontal and vertical directions, capturing fine-grained textures and subtle local variations that contribute to precise boundary localization and extraction of narrow structures and fine structural details.

To ensure balanced contribution from each frequency band, all subbands are independently normalized, compensating for the typically lower amplitude of high-frequency signals. The encoder features are decomposed using DWT and then processed through subband-specific paths, which incorporate inverse DWT (IDWT) which is followed by normalization and projection. To reconstruct frequency-specific spatial features, we apply IDWT with selective subband activation. This operation allows each path to focus on its target frequency components while maintaining spatial consistency for subsequent processing.

where $\hat{F}_{low}$, $\hat{F}_{mid}$, and $\hat{F}_{high}$ denote the reconstructed spatial features corresponding to low-frequency, mid-frequency, and high-frequency information, respectively. $F_{LL}$, $F_{LH}$, $F_{HL}$, and $F_{HH}$ are the subband features obtained from the Discrete Wavelet Transform (DWT), and 0 denotes a zero tensor of the same shape used to mask out irrelevant frequency components during the inverse transformation. We use the InceptionNext block as the projection block to capture rich representations from each frequency band while maintaining computational efficiency. These representations are aggregated into textual contexts that reflect image-specific meanings and detailed visual patterns from different frequency bands. The aggregation process corresponds to the merging of the three projection outputs at the rightmost part of Fig. 2, where low-, mid-, and high-frequency features are combined to construct a unified text context representation.

Fig. 2. Structure of Wavelet-aware Context Construction (WACC) module

3.2. Cross Deformable Fusion

Aligning textual and visual information effectively remains a key challenge in vision-language models. Traditional fusion methods often rely on transformer-based cross-attention to model global interactions between modalities via dense pairwise comparisons. However, such representations are limited in capturing local continuity and structural patterns, which are essential in visual understanding. Visual scenes inherently exhibit coherent structures such as edges, textures, and object boundaries that are spatially localized. Convolutional architectures naturally encode these patterns through their inductive biases, offering advantages in dense prediction tasks where spatial accuracy is crucial. In contrast, transformers lack such spatial priors, often resulting in less consistent structural representations for segmentation.

Inspired by the design of InternImage ^[14], we propose Cross Deformable Fusion (CDF), a content-adaptive fusion module that leverages deformable convolution to align textual and visual features more precisely. As illustrated in Fig. 3, the fusion process consists of two stages. In the first stage, visual features are used to generate offsets and modulation masks, which are applied to the text embeddings via deformable convolution. To enable this interaction, the visual features are spatially aggregated to produce class-level representations, allowing each class token to focus on semantically corresponding regions in the image. Since the text embeddings are organized as a class-wise token set, we adopt a deformable convolution with a kernel size of $s \times 1$, which operates along the class axis. The deformable convolution is defined as:

where $\mathbf{T}'_n$ is the refined text embedding at class $n$, and $w_k$ denotes the convolution weight. $\Delta p_k$ is the learnable offset and $m^t_k \in [0,1]$ is the modulation mask. This formulation enables the model to adaptively refine the text features based on spatial cues derived from the image, effectively aligning them with semantically relevant visual regions and improving cross-modal consistency.

Fig. 3. Structure of Cross Deformable Fusion (CDF) module

In the second stage, we use the refined text embeddings from the previous step to guide spatial interaction within the visual domain. Specifically, we perform an element-wise multiplication between the refined text embeddings and the original visual features to produce a fused representation that captures semantic relevance between the two modalities. This operation is implemented as a broadcasted element-wise multiplication between the refined text embeddings $\mathbf{T}' \in \mathbb{R}^{B \times N \times C}$ and the visual feature map $\mathbf{V} \in \mathbb{R}^{B \times C \times H \times W}$, denoted as $\mathbf{T}' \odot \mathbf{V}$, where $B$ is the batch size, $C$ the channel dimension, and $H, W$ denote height and width. The refined text embeddings act as semantic guidance, modulating the spatial focus of the visual features. From the resulting fused representation, we generate spatial offsets $\Delta q_k$ and modulation masks $m^v_k \in [0,1]$, which are then applied to $\mathbf{V}$ through deformable convolution:

where $\mathbf{V}'_i$ is the updated visual representation at location $i$, $w_k$ is the convolution weight. As a result, the model can emphasize regions that are closely related to class-specific concepts, enhancing the alignment between semantic intent and visual structure.

Unlike conventional fusion methods that treat all spatial locations uniformly, our module allows the model to selectively focus on class-relevant regions by using textual cues to guide the refinement of visual features. By structuring this process in two stages, Cross Deformable Fusion first leverages visual context to refine textual representations, and then uses the updated text to enhance spatially meaningful regions in the visual domain, achieving more precise cross-modal alignment.

3.3. Loss Function

Our training objective consists of two complementary loss functions. The first is a segmentation loss that provides supervision for pixel-wise classification. The second is a text alignment loss that enhances semantic consistency between visual and textual representations. Together, these objectives guide the model to produce accurate and semantically coherent predictions.

We adopt the standard cross-entropy loss for semantic segmentation. This loss encourages the model to predict class distributions at each pixel to match the ground-truth label. The segmentation loss is defined as follows.

$K$ is the number of classes, $y^k$ is the one-hot ground truth label, and $\hat{y}^k$ is the predicted probability for class $k$ obtained via softmax.

To enhance semantic alignment between modalities, we employ an auxiliary loss that compares visual features with class-level text embeddings. For each pixel, we compute the cosine similarity between its visual feature vector $f$ and each text embedding $t_k$:

The similarity scores are normalized via softmax to obtain a probability distribution, which is then compared with the ground-truth label using cross-entropy loss.

This formulation follows the similarity-based supervision strategy adopted in ClipSeg ^[10], where pixel-wise similarity between visual and textual embeddings is directly used for segmentation. This loss encourages the visual features to remain close to the corresponding text embeddings, improving semantic consistency across modalities. The total loss is computed by combining the segmentation loss and the text alignment loss with a balancing factor $\lambda$.

4.1. Settings

To evaluate the effectiveness of the proposed method, we conducted experiments on the Cityscapes ^[2] dataset. We utilized 1,000 images for training and 50 for evaluation. During training, input images were randomly cropped to a resolution of $512 \times 1024$. In our experiments, we used a total of 19 semantic segmentation classes, including road, sidewalk, building, wall, fence, pole, traffic light, traffic sign, vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle, and bicycle. Each class name was combined with a constructed text context embedding and passed through a frozen CLIP model to generate class-level text features. We set the balancing factor $\lambda = 0.5$ and the kernel size $s = 7$. The performance was evaluated using the Intersection over Union (mIoU) metric.

4.2. Results on the Cityscapes Dataset

For comparative evaluation, we selected state-of-the-art segmentation backbones including ConvNeXt ^[9], InternImage ^[14], and InceptionNeXt ^[17], all in their Base configurations. Each backbone was initialized with weights pretrained on the ImageNet-1K dataset, and the UPerNet ^[15] decoder was used for all models. The proposed method is built upon InceptionNeXt-B and is further enhanced by integrating the Wavelet-Aware Context Construction (WACC) and Cross Deformable Fusion (CDF) modules. Quantitative results, measured by class-wise and mean IoU, are summarized in Table 1. The proposed method achieved the highest overall performance among all compared models. Improvements were particularly notable in categories involving fine structures and complex boundaries, such as fence, truck, bus, motorcycle, and bicycle. Qualitative comparisons are illustrated in Fig. 4. The proposed method produces more accurate segmentation results, particularly in complex scenes involving small or fine-grained objects. The proposed method shows improved segmentation of traffic signs, poles, and riders, particularly in regions with fine structures and ambiguous boundaries. Additionally, the proposed method preserves object consistency and reduces miss classification around class boundaries, leading to more coherent scene understanding. These visual improvements reflect the benefit of integrating contextual text features, which provide complementary semantic cues during decoding.

Fig. 4. Qualitative comparison of segmentation results

4.3. Ablation studies

To further investigate the effectiveness of the proposed components, we conducted ablation studies on the Cityscapes dataset. The first set of experiments examines different textual context construction strategies, as shown in Table 2. The baseline model without any textual guidance yields the lowest mIoU score, suggesting limited capability in capturing semantic distinctions. Introducing learnable prompt parameters leads to noticeable improvement, demonstrating that even static textual priors can enhance semantic discrimination. Further gains are achieved by incorporating image-based context. In this setting, visual features are used without applying DWT to isolate the effect of visual guidance. Our proposed method enhances textual context construction by applying the DWT to visual features, enabling frequency-aware decomposition. This enables the encoding of both global semantics and fine-grained boundary details into the textual context, resulting in richer and more discriminative textual representations.

The second group of experiments compares different cross-modal fusion mechanisms, as summarized in Table 3. The cross attention approach uses image features as queries and text features as keys and values in a single transformer-based operation. While this enables basic modality interaction, attending in a single direction limits the model’s ability to capture fully interactive cross-modal relationships. To address this, Dual Cross Attention performs attention in both directions, from text to image and from image to text, enhancing semantic consistency and spatial precision. In contrast, the proposed CDF module adopts a sequential interaction scheme that leverages lightweight, spatially-adaptive convolution guided by cross-modal feature interaction. To assess computational efficiency, we compare dual cross attention with our CDF module. While dual cross attention requires 35.65 GFLOPs, CDF reduces the computational cost to 26.34 GFLOPs. Despite this reduction, CDF achieves better segmentation performance, demonstrating the benefits of its convolutional design that efficiently captures spatially local dependencies. This design enables effective bidirectional fusion without relying on transformer-based attention mechanisms.