TY - JOUR

T1 - Cross-Modal Comprehensive Multi-Level Granularity Alignment for Vision-Language Pre-Training

AU - Jiang, J

AU - Вукович, Дарко

AU - Cao, J.

AU - Haoran, Xie

AU - Wang, Youquan

N1 - Jiang, J., Vukovic, D. B., Cao, J., Xie, H., Wang, Y., & Wu, J. (2026). Cross-Modal Comprehensive Multi-Level Granularity Alignment for Vision-Language Pre-Training. ACM Transactions on Multimedia Computing, Communications and Applications, 284, 1-23, doi: https://doi.org/10.1145/380254

PY - 2026/10/9

Y1 - 2026/10/9

N2 - Vision-Language Pre-Training (VLP) aims to convert vision-language tasks into image-text matching challenges, necessitating robust interaction between visual and textual modalities. However, current research often faces a tradeoff: methods pursuing fine-grained alignment typically rely on computationally expensive object detectors and pre-defined annotations, while coarse-grained alignment methods overlook subtle visual details. To alleviate these issues, we present OMEGA, a COmprehensive Multi-lEvel Granularity Alignment framework. OMEGA achieves sufficient cross-modal alignment without relying on external object detectors or annotations. Specifically, we first design a Text-Aware Patch Selection (TAPS) module, which dynamically constructs patch-level regions for fine-grained excavation based on informative token selection, effectively bypassing the need for bounding boxes. Second, to facilitate multi-grained information fusion, we introduce a Cross-Grained Alignment (CGA) module to learn modality-shared features across global and object levels. Experimental results demonstrate that OMEGA surpasses existing approaches in five pivotal vision-language tasks—image-text retrieval, visual question answering, visual reasoning, visual entailment, and image captioning, while maintaining superior inference efficiency compared to detector-based methods. We believe this study offers a promising object-free paradigm for scalable vision-language pre-training.

AB - Vision-Language Pre-Training (VLP) aims to convert vision-language tasks into image-text matching challenges, necessitating robust interaction between visual and textual modalities. However, current research often faces a tradeoff: methods pursuing fine-grained alignment typically rely on computationally expensive object detectors and pre-defined annotations, while coarse-grained alignment methods overlook subtle visual details. To alleviate these issues, we present OMEGA, a COmprehensive Multi-lEvel Granularity Alignment framework. OMEGA achieves sufficient cross-modal alignment without relying on external object detectors or annotations. Specifically, we first design a Text-Aware Patch Selection (TAPS) module, which dynamically constructs patch-level regions for fine-grained excavation based on informative token selection, effectively bypassing the need for bounding boxes. Second, to facilitate multi-grained information fusion, we introduce a Cross-Grained Alignment (CGA) module to learn modality-shared features across global and object levels. Experimental results demonstrate that OMEGA surpasses existing approaches in five pivotal vision-language tasks—image-text retrieval, visual question answering, visual reasoning, visual entailment, and image captioning, while maintaining superior inference efficiency compared to detector-based methods. We believe this study offers a promising object-free paradigm for scalable vision-language pre-training.

U2 - 10.1145/3802546

DO - 10.1145/3802546

M3 - Article

VL - 22

SP - 1

JO - ACM Transactions on Multimedia Computing, Communications and Applications

JF - ACM Transactions on Multimedia Computing, Communications and Applications

SN - 1551-6857

IS - 10

M1 - 284

ER -