Real-Time Image Segmentation via Hybrid Convolutional-Transformer Architecture Search

Image segmentation is one of the most fundamental problems in computer vision and has drawn a lot of attention due to its vast applications in image understanding and autonomous driving. However, designing effective and efficient segmentation neural architectures is a labor-intensive process that may require numerous trials by human experts. In this paper, we address the challenge of integrating multi-head self-attention into high-resolution representation CNNs efficiently by leveraging architecture search. Manually replacing convolution layers with multi-head self-attention is non-trivial due to the costly overhead in memory to maintain high resolution. By contrast, we develop a multi-target multi-branch supernet method, which not only fully utilizes the advantages of high-resolution features but also finds the proper location for placing the multi-head self-attention module. Our search algorithm is optimized towards multiple objectives (e.g., latency and mIoU) and is capable of finding architectures on the Pareto frontier with an arbitrary number of branches in a single search. We further present a series of models via the Hybrid Convolutional-Transformer Architecture Search (HyCTAS) method that searches for the best hybrid combination of light-weight convolution layers and memory-efficient self-attention layers between branches from different resolutions and fuses them to high resolution for both efficiency and effectiveness. Extensive experiments demonstrate that HyCTAS outperforms previous methods in both semantic segmentation and panoptic segmentation tasks. Code and models are available atthis https URL.

@article{yu2025_2403.10413,
  title={ Real-Time Image Segmentation via Hybrid Convolutional-Transformer Architecture Search },
  author={ Hongyuan Yu and Cheng Wan and Xiyang Dai and Mengchen Liu and Dongdong Chen and Bin Xiao and Yan Huang and Yuan Lu and Liang Wang },
  journal={arXiv preprint arXiv:2403.10413},
  year={ 2025 }
}